What is the difference between effect size and statistical significance?

Hello, I'm a statistician with a keen interest in research methodology. Let's dive into the nuances between statistical significance and effect size, two pivotal concepts in data analysis and interpretation. Statistical Significance is a measure that tells us whether the results of a study are likely to be due to chance or not. It's based on the P-value, which is the probability of observing a result as extreme as the one obtained, assuming that the null hypothesis is true. If the P-value is less than a predetermined threshold, often set at 0.05, we say that the result is statistically significant. This means that there's less than a 5% chance that the observed difference is due to random variation alone. It's important to note that statistical significance does not imply practical significance; a statistically significant result might be very small and not meaningful in real-world terms. On the other hand, Effect Size is a measure of the magnitude of a result. It's a way to quantify the strength or importance of the relationship between variables. Unlike statistical significance, effect size is not dependent on the sample size. It provides an estimate of the actual difference or strength of the relationship that is not influenced by the size of the sample. Common measures of effect size include Cohen's d for continuous data, odds ratios for categorical data, and eta squared for ANOVA. Here are some key differences: 1. Nature of the Measure: Statistical significance is a binary outcome—it tells us whether the result is likely due to chance (P < alpha) or not. Effect size, however, is continuous and provides information about the magnitude of the effect. 2. Sample Size Dependency: Statistical significance can be influenced by the sample size. With a large enough sample, even a very small effect can become statistically significant. Effect size is not affected by sample size and remains a constant measure of the strength of the effect. 3. Practical vs. Statistical: While statistical significance addresses the question of whether an effect exists, effect size answers the question of how large the effect is. A result can be statistically significant but have a tiny effect size, which might not be meaningful in a practical context. 4. Research Questions: Statistical significance is often used to test specific hypotheses. Effect size is more relevant when considering the practical implications of the findings or when comparing the strength of effects across different studies. 5. Reporting Standards: In academic research, it's becoming increasingly important to report both statistical significance and effect size. This provides a more comprehensive understanding of the study's findings. 6. Decision Making: In fields like clinical trials or policy-making, knowing the effect size can be crucial for making informed decisions. A statistically significant result with a small effect size might not lead to a change in practice if the practical benefits are negligible. 7. Type of Data: Statistical significance is often associated with null hypothesis significance testing (NHST), which is a specific framework for hypothesis testing. Effect size can be calculated for a wide range of data types and statistical methods, not just those that rely on NHST. In conclusion, while statistical significance tells us if our findings are likely to be real, effect size tells us how meaningful those findings are. Both are important for a thorough understanding of research results, but they serve different purposes and should be interpreted in conjunction with one another.

Statistical significance is the probability that the observed difference between two groups is due to chance. If the P value is larger than the alpha level chosen (eg, .05), any observed difference is assumed to be explained by sampling variability. ... Unlike significance tests, effect size is independent of sample size.