The fundamental idea of inferential statistics is determining the probability of obtaining the observed data when we assume the null hypothesis is true. For example, if we roll a die 10 times and got 10 sixes, what is the probability of observing this result if we assume the null hypothesis that the die was fair? If the die is fair, the probability of getting 10 sixes in 10 rolls is , which is a very low probability. Since it's extremely unlikely that we observe 10 sixes on 10 rolls of a fair die by chance, we should reject the null hypothesis. This probability is the p-value.

Let's consider a less extreme case than the previous example. Here I will use the example from the first lecture of the Statistics for Neuroscience (9506) course, whereby a person (the lecturer) claimed that he had to ability to distinguish two different brands of espresso. Our null hypothesis in this case, is that the lecturer doesn't have the ability to distinguish the brands and is simply guessing. We come up with an experiment to test his ability by giving him 8 cups of espresso, where 4 are from brand A and the other 4 are from brand B, and ask him to separate them into two groups. If he managed to correctly group the 8 cups into their respective brands, what is the probability of getting this result if we assume the null hypothesis is true, i.e. what's the probability of getting this result just by chance?

Continue reading →