This article was published as part of the Data Science Blogathon.
Introduction
How much data is sufficient to establish statistical significance? In other words, What should be the optimal sample size? Often, it is not entirely feasible to run the statistical experiment several times to ensure sufficient power. At the same time, our machine learning model might not be statistically conclusive if we don't have a suitable sample size.
Let's first analyze statistical power in detail.
Statistical power of a hypothesis test implies the probability of detecting an effect, since there is a true effect. It is the confidence that is derived from the results of a study and is inversely related to the type error 2.
Note that the type error 2 is false negative when we do not reject the false null hypotheses.
In this point, it is important to understand what is the null hypothesis? It is a statistical hypothesis test that assumes a result. For instance, the null hypothesis in the KS statistic implies that the two populations belong to a similar distribution.
Source: Wiki with additions by the author
It is important to note in the illustration above that the higher the power of a test, smaller is the β, namely, the type error 2
How the lowest statistical power of an experiment leads to invalid conclusions about the result, you want your experiments to have a minimum power threshold. Generally, it is expected to be from 80% or more. A powerhouse of 80% means there is a 80% probability of detecting an effect that exists (Y, at the same time, a 20% probability of observing a type error 2).
Power analysis:
Now is the time to see the bigger picture, namely, the power analysis that depends on four related variables, as mentioned below:
1) Effect size: the more prominent the data has, smaller is the random error.
2) Sample size: a larger sample size helps detect smaller effects
3) Significance level: a
4) Statistical power
The four variables are linked to each other and changing one variable affects the rest of the variables.
Power analysis is the process of estimating one of the 4 variables given values for 3 variables. Often used to estimate the minimum sample size to perform an experiment.
As we increase the sample size, we can also detect the small effects, although at the cost of conducting statistical experiments several times. Even then, there comes a point where adding more data does not increase power any more.
Note that the sample we are working with may not capture the effect even if it exists in the population. This is largely attributed to sampling error when the sample is not representative of the population..
Power analysis is also used to verify and validate the results and findings of the experiment.. For instance, if we specify the effect size, sample size and significance level, we can calculate the power of an experiment to check whether the probability of type error 2 is within an acceptable range.
According to documentation, we can solve any of the 4 parameters in an independent T test of 2 samples:
Source: Author
We can also use plot power curves to check how varying the effect size and different sample size change the power of the experiment at a given significance level.
Source: Author
References:
https://statisticsbyjim.com/hypothesis-testing/sample-size-power-analysis/
https://machinelearningmastery.com/statistical-power-and-power-analysis-in-python/
