Probability

The fundamental idea of inferential statistics is determining the probability of obtaining the observed data when we assume the null hypothesis is true. For example, if we roll a die 10 times and got 10 sixes, what is the probability of observing this result if we assume the null hypothesis that the die was fair? If the die is fair, the probability of getting 10 sixes in 10 rolls is $$\frac{1}{6}^{10} = 1.653817e-08$$, which is a very low probability. Since it's extremely unlikely that we observe 10 sixes on 10 rolls of a fair die by chance, we should reject the null hypothesis. This probability is the p-value.

Let's consider a less extreme case than the previous example. Here I will use the example from the first lecture of the Statistics for Neuroscience (9506) course, whereby a person (the lecturer) claimed that he had to ability to distinguish two different brands of espresso. Our null hypothesis in this case, is that the lecturer doesn't have the ability to distinguish the brands and is simply guessing. We come up with an experiment to test his ability by giving him 8 cups of espresso, where 4 are from brand A and the other 4 are from brand B, and ask him to separate them into two groups. If he managed to correctly group the 8 cups into their respective brands, what is the probability of getting this result if we assume the null hypothesis is true, i.e. what's the probability of getting this result just by chance?

The easiest way to calculate the probability of correctly identifying each brand is to simply multiply the probabilities of correctly identifying one particular brand among the 8 cups; this works because once you correctly identify all of one brand, you're left with the other four that belong to the other brand:

$$!\frac{4}{8} \times \frac{3}{7} \times \frac{2}{6} \times \frac{1}{5} = 0.01428571.$$

Thus the p-value is rounded to 0.01; there is roughly a 1 percent chance he was simply guessing if he managed to correctly separate the 8 cups of espresso into their two brands.

Another way of calculating this probability, as demonstrated in the lecture, is to first work out the total number of ways of grouping 8 cups into two groups. Of the total possible ways, there is only one grouping that correctly separates the two brands. Therefore we can work out the probability by:

$$!\frac{1}{total\ possible\ groups}$$

Once again, note that once 4 cups are placed into one group, two groups have formed. Therefore to work out the total possible groups, we can calculate the total number of combinations of choosing 4 cups from 8 (without replacement):

$$!\frac{n!}{r!(n - r)!} = \frac{8!}{4! \times 4!} = \frac{8 \times 7 \times 6 \times 5}{4 \times 3 \times 2 \times 1} = 70$$

Therefore the probability of getting the correct grouping, i.e. correctly identified the two brands, of the 8 cups by guessing is:

$$!\frac{1}{70} = 0.01428571.$$

If you want to see all the groupings/combinations, we can use the gtools package in R:

#install if necessary
install.packages('gtools')
#load package
library(gtools)
#name each espresso cup
x <- c('a1','a2','a3','a4','b1','b2','b3','b4')
head(combinations(n=8, r=4, v=x,repeats.allowed=F))
     [,1] [,2] [,3] [,4]
[1,] "a1" "a2" "a3" "a4"
[2,] "a1" "a2" "a3" "b1"
[3,] "a1" "a2" "a3" "b2"
[4,] "a1" "a2" "a3" "b3"
[5,] "a1" "a2" "a3" "b4"
[6,] "a1" "a2" "a4" "b1"
tail(combinations(n=8, r=4, v=x,repeats.allowed=F))
      [,1] [,2] [,3] [,4]
[65,] "a3" "b2" "b3" "b4"
[66,] "a4" "b1" "b2" "b3"
[67,] "a4" "b1" "b2" "b4"
[68,] "a4" "b1" "b3" "b4"
[69,] "a4" "b2" "b3" "b4"
[70,] "b1" "b2" "b3" "b4"

Note that the 70th combination (above) is getting all 4 guesses wrong, which is equally likely as getting all 4 guesses correct.

Now consider this: what if the person correctly grouped 3 cups into the right brand (out of the 8)? Should we believe that he has the ability to taste different brands? What is the probability of correctly grouping 3 cups by chance? The easiest way of calculating that is by counting the total number of combinations where the person got 3 correct and 1 incorrect (e.g. {a1, a2, a3, b1}, {a1, a2, a3, b2}, and so on) and dividing this by the total number of combinations. There are 16 combinations of getting 3 correct and 1 incorrect out of the 70 total; therefore getting 3 right just by chance is 0.2285714, i.e. the p-value is 0.23. There is roughly a 1/4 chance that he was simply guessing despite getting 3 cups of espresso correct!

Now is there a better way of calculating this instead of counting the combinations where he got 3 correct? We can use the hypergeometric distribution, which is a discrete probability distribution that describes the probability of k successes in n draws without replacement from a finite population of size N containing exactly K successes.

A random variable X follows the hypergeometric distribution if its probability mass function (pmf) is given by:

$$!P(X = k) = \frac{\binom{K}{k} \binom{N-K}{n-k}}{\binom{N}{n}}$$

where N is the population size, K is the number of success states in the population, n is the number of draws and k is the number of successes. If we define a success as correctly classifying a brand, then the probability of getting three successes is:

$$!P(X = 3) = \frac{\binom{4}{3} \binom{4}{1}}{\binom{8}{4}} = \frac{4 \times 4}{70} = 0.2285714$$

So it seems it's not so impressive that he got 3/4 correct. Let's overdose him with espresso by giving him 10 cups to group. What's the probability that he gets them all correct by chance?

$$!P(X = k) = \frac{\binom{5}{5} \binom{5}{0}}{\binom{10}{5}} = \frac{1}{252} = 0.003968254$$

And 4/5?

$$!P(X = 4) = \frac{\binom{5}{4} \binom{5}{1}}{\binom{10}{5}} = \frac{5 \times 5}{252} = 0.09920635$$

If he manages to get 4/5 correct, there's a ~10% chance he was guessing. Should we believe him? In scientific studies, the p-value is not statistically significant, i.e. p-value > 0.05, so we can't reject the null hypothesis that he is simply guessing.

Conclusions

There was a point in my life, when I didn't know how to explain the p-value. I would use some statistical test seemingly appropriate for my dataset and would hope for a significant p-value, i.e. p-value < 0.05. I never really thought about what the p-value represents in a simplistic manner. As I was going through the first lecture of the Statistics for Neuroscience course, I thought their concept of explaining the p-value was really intuitive, so I went with it and expanded on it a little resulting in this post. This leads to probability distributions, which are lists or functions that shows all the possible values of a random variable and the corresponding probabilities of the possible outcomes. The probability distribution of a single fair die, is just a histogram with 6 bars, one for each number and each with a 1/6 probability of occurring. From a probability distribution, we can easily work out the probability of a certain event, and therefore the p-value.