H0, null hypothesis – challenge your preconceived notions

After listening to Göte Nyman at TEDxHelsinki about Null hypothesis (H0), I just had to write down some thoughts that arose from it. First a recap:
Göte presented examples from popular news media, such as “coke and pepsi activate different areas of the brain”. Well duh, if they are experienced differently, someone might assume that the experience manifests itself as brain activity. So the null hypothesis behind the news item was that coke and pepsi affect the brain in the same way. So the point was that looking at the assumptions behind “revelations” tells you something about the naïve assumptions people generally have about the topic. I mean, it’s not news unless it challenges current thinking. So each scandalous headline reveals something about the assumptions of the editor about what the audience thinks about the topic.
OK. The null hypothesis of course is named because in statistical testing of quantitative experiment results, the measured effect between the experiment and the control doesn’t differ from zero (or null) unless there’s a difference between the two.
Names are important. Because the null hypothesis is called the null hypothesis, it doesn’t sound very valuable, now does it? And this actually is something that the scientific community handles pretty badly. If a researcher conducts an experiment, his H0 is the “current knowledge”, or the currently valid theory, which from his point of view is quite uninteresting. He’s trying to get results that prove that theory to be wrong, or insufficient, thereby giving grounds to forming a new, better theory. So if the results are that the H0 is valid (there was no measurable effect), that results almost never is published. Because it is not very interesting, or worth anything.
Now, let’s remember that no theory can never be proven to be correct. That’s why they’re called theories. You can only disprove a theory by showing empirical data that contradicts it. And if a theory talks about general trends (like most of social sciences do), then a single contradiction isn’t enough – you need to get several large samples that consistently contradict a theory. If a theory cannot be contradicted despite repeated attempts, it will start to look reliable. Now, the problem is this: since scientists (and journals) tend to not publish results that say “yeah, the old theory seems to still hold”, but publish ones that say “aha, we have a contradiction”, the published literature is biased towards results that contradict H0 assumptions. And we jump to conclusions, because we do not see all the other “failed” studies that didn’t show a measurable effect. So we can build cloud castles of experimental results that all point to something, which in reality is only in the 5% of the 95% margin of error of these studies – because the other 95% of study results are never published. Darn.
So whenever you think you’ve learned something new, take a step back and look at your H0 – what did you believe just a few seconds ago? And why?
Talking about learning, let’s talk about Progressive inquiry since that learning model emulates the way scientists and knowledge workers operate.

In this model, the null hypothesis has its place in the “Create working theories” phase. When learners present their current knowledge and their naïve theories about the subject matter, they are in fact presenting their H0, the null hypothesis. The iterative process that follows then tries to build upon the learners’ existing knowledge, arriving at a better state of understanding. Hopefully at the end of the process, the learner can see that his H0 has been disproven. If this does not happen, then he has basically not learned anything new. So, just like in science, in learning the value is in challenging the existing ideas.
But is it really worthless if someone goes through a learning cycle, and comes out with the same ideas? That should give some value to those ideas – they stand the test of time, or the test of learning. But was it still a waste of time?
PS. If reading this gave you new ideas, what was your H0 before reading this?

Tags: nullhypothesis, h0, tedxhelsinki, ted, learning, progressiveinquiry, science, bias