P-values, demystified: what "statistically significant" actually means

# P-values, demystified: what "statistically significant" actually means If you've ever sat in a stats class, a research-methods seminar, or a data science interview, you've heard some version of this: > "If p < 0.05, the result is statistically significant." Almost everyone who says this — including people teaching it — is fuzzy on what it actually means. After 10 years of tutoring Python, ML, and data science to working engineers and career-changers, I can tell you the same three misunderstandings come up in nearly every session that touches statistics. This post fixes them. No formulas you don't need. No shame for not knowing — this stuff is genuinely confusing and the textbooks are partly to blame. ## The one-sentence definition A p-value is the probability that, **assuming there is no real effect**, you would see results at least as extreme as the ones you got, just from random chance. That's it. Now read it again, because every single word is doing work. - **"Assuming there is no real effect"** — this is the null hypothesis. The whole calculation starts from "what if my treatment did nothing?" - **"At least as extreme"** — not "exactly as extreme." We include everything as weird-or-weirder. - **"Just from random chance"** — sampling variability, not measurement error, not bias, not selection. If your p-value is 0.03, that means: *if my drug actually does nothing, there's a 3% chance I'd see a result this strong (or stronger) just because I happened to sample lucky people.* ## The three things p < 0.05 does NOT mean This is where almost everyone trips up. None of these is what a p-value tells you: ### 1. It is **not** "the probability that my hypothesis is true" If p = 0.03, that does **not** mean "there is a 97% chance my drug works." That confuses two different probabilities: - **What the p-value tells you:** P(seeing this data | drug does nothing) - **What you wish it told you:** P(drug works | seeing this data) These are the same kind of error as confusing P(rain | cloudy) with P(cloudy | rain). To get the second one (which is what you actually want), you need Bayes' theorem and a prior probability that your hypothesis is true. The p-value alone cannot give it to you. This isn't a pedantic distinction. It changes the strength of your evidence dramatically: - Test 100 drugs that mostly don't work. About 5 will hit p < 0.05 by chance alone. Of those 5, none of them are actually effective. **But each one looks "statistically significant."** This is why "5% of psychology papers might be wrong" is a known concern. It's also why "p < 0.05" alone is no longer accepted as proof in most rigorous fields without replication or pre-registration. ### 2. It is **not** a measure of effect size A drug that lowers blood pressure by 0.1 mmHg can have p < 0.001 if you test 100,000 people. A drug that lowers it by 30 mmHg can have p > 0.05 if you test 8 people. The p-value tells you how confident you are that *some* effect exists. It says nothing about whether the effect is *big enough to matter*. Always report both: ``` "Mean reduction: 12 mmHg (95% CI: 8–16 mmHg, p = 0.002)" ``` The 12 with the confidence interval is the *result*. The p-value is just confirming the result isn't noise. A study that reports only "p < 0.001" and not the effect size is hiding something. ### 3. It is **not** binary The line between p = 0.049 and p = 0.051 is not the line between "real" and "not real." It is the line between "we used the conventional cutoff one way" and "the other way." Nature does not care about 0.05. If your study comes in at p = 0.06, the right thing to say is "we observed an effect of magnitude X with marginal evidence (p = 0.06); replication in a larger sample would clarify." The wrong thing is "the result was not significant, so the drug doesn't work." The 0.05 threshold is a Fisher convention from agricultural experiments in the 1920s. It was never meant to be a divine line. The American Statistical Association published a [statement](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf) in 2016 explicitly cautioning against treating it as one. :::contextual-cta topic="ml" ::: ## A worked example: the "miracle pill" trap You're testing whether a new study technique improves test scores. You give the technique to 30 students and a placebo (regular study guide) to 30 others. After two weeks, you collect their scores. ```python import numpy as np from scipy import stats rng = np.random.default_rng(42) # Imagine the truth: the technique gives a tiny boost (+1.5 points on a 100 scale) treatment = rng.normal(73.5, 12, size=30) # mean 73.5, sd 12 control = rng.normal(72.0, 12, size=30) # mean 72.0, sd 12 t_stat, p_value = stats.ttest_ind(treatment, control) print(f"Treatment mean: {treatment.mean():.1f}") print(f"Control mean: {control.mean():.1f}") print(f"Difference: {treatment.mean() - control.mean():.1f}") print(f"p-value: {p_value:.3f}") # Treatment mean: 73.8 # Control mean: 71.4 # Difference: 2.4 # p-value: 0.444 ``` A 2.4-point boost in your sample, but p = 0.444 because 30 students per group is not enough to distinguish a tiny real effect from sampling noise. Headline: "Study technique not statistically significant." Reality: the effect was there, the test just couldn't see it. Now run the same experiment with 3,000 students per group: ```python treatment = rng.normal(73.5, 12, size=3_000) control = rng.normal(72.0, 12, size=3_000) t_stat, p_value = stats.ttest_ind(treatment, control) print(f"Difference: {treatment.mean() - control.mean():.2f}") print(f"p-value: {p_value:.4f}") # Difference: 1.47 # p-value: 0.0000 (i.e., highly significant) ``` Same effect, **same drug**, but now p < 0.0001 because we have the power to detect a real-but-tiny effect. Headline: "Study technique highly significant!" Both headlines are misleading on their own. The honest summary is: *the technique appears to give a ~1.5-point improvement on a 100-scale test*. Whether that's worth doing depends on the cost and on whether 1.5 points matters for the decision you're making — not on whether p crossed an arbitrary threshold. ## The "p-hacking" trap (and how interview questions test for it) Suppose your study is going badly and p = 0.12. You decide to: - Drop the three weirdest data points ("they're outliers") - Re-segment by gender ("maybe it works for women") - Try a different test ("Mann-Whitney instead of t-test") - Add 5 more participants ("just to be safe") Eventually one of these gives you p = 0.04. You publish that one. This is **p-hacking**. The math says: if you try 20 things, one of them will hit p < 0.05 by chance alone, even with no real effect. You've turned the 5% false-positive rate into something closer to 60%. The fix is **pre-registration**: you commit, in writing, before you collect data, to exactly which test you'll run and how. If your pre-registered test gives p = 0.12, you report p = 0.12. If you want to explore other angles, you label them clearly as exploratory and run a fresh study to confirm them. In an interview, if someone tells you "we ran a few different analyses and one of them was significant," the right response is: "what was the pre-registered analysis?" If they don't have one, the result is exploratory at best. ## What to use instead (or alongside) Most modern data science practice supplements or replaces raw p-values with: 1. **Confidence intervals.** "12 mmHg reduction (95% CI: 8–16)" tells you the magnitude *and* the uncertainty in one number. It contains everything a p-value contains and more. 2. **Effect sizes.** Cohen's d, R², odds ratios — these answer "how big?" not just "is it there?" 3. **Bayesian credible intervals.** "There is a 95% probability the true effect is between 8 and 16 mmHg" — which is what most people *thought* a confidence interval already meant. (It doesn't. That's a different post.) 4. **Pre-registration.** Decide your analysis plan before seeing the data. 5. **Replication.** A real effect should show up again in a fresh sample. ## The interview / take-home version Three things that will signal you actually understand p-values, not just that you've heard of them: 1. **A p-value answers "how surprising is this data, assuming no effect?"** It does not answer "what's the probability my hypothesis is true?" 2. **Statistical significance is not effect size.** Always report both. 3. **The 0.05 threshold is a convention, not a law of nature.** Treating it as binary is what got reproducibility into trouble. If you can articulate these in an interview when asked "tell me about p-values," you'll be ahead of most candidates with quantitative degrees. ## Closing Statistics is hard because it asks you to reason carefully about uncertainty in the presence of strong intuitions that almost always pull you wrong. The p-value is one of the most-misunderstood numbers in science not because anyone is dumb, but because the formal definition is genuinely awkward and most teaching skips the careful version. If you want help building a stronger statistical intuition — for ML model evaluation, A/B tests, or a stats class you're stuck in — I do 1-on-1 sessions in data science fundamentals. The first 30-minute session is free, no card required. [Book one here](/book-session?service=data-science-fundamentals). — Ali *M.Sc. Mechanical Engineering · Licensed P.Eng. · 10+ years of teaching engineering statistics, machine learning evaluation, and applied data science.*