p-values appear in research reports, A/B test results and papers everywhere, and they are also the most frequently misread statistic. The American Statistical Association (ASA) even issued a formal statement on their proper use in 2016. This guide starts from the exact definition, then walks through common misconceptions and good reporting practice with an example.
Definition of a p-value
A p-value is defined as:
The probability, assuming the null hypothesis is true, of getting a result at least as extreme as the one observed
The key is the condition at the start. A p-value is computed under the assumption "if the null hypothesis is true", so it cannot tell you the probability that this assumption is itself true. A small p-value means "if the null hypothesis were true, data like these would be rare", in other words the data fit the null hypothesis poorly.
Example: comparing two group means
Group A used a new study method and group B the usual one. Their scores (8 people each) are:
- A:
72, 75, 78, 80, 74, 77, 79, 81 - B:
70, 71, 74, 73, 69, 72, 75, 68
The null hypothesis is "the two population means are equal" and the alternative is "they differ". We use Welch's t-test, which does not assume equal variances.
- Means:
x̄_A = 77.0,x̄_B = 71.5, difference5.5 - Sample standard deviations:
s_A ≈ 3.1168,s_B ≈ 2.4495 - Standard error:
SE = √(s_A²/n_A + s_B²/n_B) = √(9.7143/8 + 6.0000/8) = √1.9643 ≈ 1.4015 - Test statistic:
t = 5.5 / 1.4015 ≈ 3.924 - Welch degrees of freedom: about
13.26 - Two-sided p-value: about
0.0017
The interpretation: "If the two methods truly had equal population means, a difference in sample means of 5.5 points or more (in either direction) would occur about 0.17% of the time." That is well below 0.05, so we reject the null hypothesis.
Significance level α
The significance level α is a threshold fixed before looking at the data. If p ≤ α, the null hypothesis is rejected. α is the maximum rate at which you accept rejecting a true null hypothesis (a Type I error). 0.05 is conventional, but 0.01 or 0.10 are also used depending on the field and situation.
α and the p-value are different things. α is the long-run error rate of the testing procedure; the p-value measures how far this particular data set departs from the null hypothesis.
Five common misconceptions
1. "The p-value is the probability that the null hypothesis is true"
It is not. p = 0.0017 does not mean "there is a 0.17% chance the null hypothesis is true". A p-value is the probability of the data given the null hypothesis, while the probability of the null hypothesis is the probability of a hypothesis given the data. The two conditional probabilities point in opposite directions, and the latter requires Bayesian inference with prior probabilities.
2. "1 − p is the probability the alternative is true" or "the chance of replication"
This just flips the first misconception, so it is wrong too. p = 0.03 does not mean a 97% chance that the effect is real, nor a 97% chance of a significant result if the experiment is repeated.
3. "p > 0.05 means there is no effect"
A non-significant result is not "evidence of no effect". It means "not enough evidence to claim an effect". With a small sample, p can easily be large even when a real effect exists. For example, 60 heads in 100 coin flips gives an exact two-sided binomial p-value of about 0.0569 against a fair coin. That is just above 0.05, but it certainly does not show the coin is fair.
4. "A smaller p-value means a bigger effect"
A p-value is a joint product of effect size and sample size. With a large enough sample, even a tiny difference becomes significant. A difference of 0.02 standard deviations between two groups, practically meaningless, gives z ≈ 4.47 and p ≈ 0.0000077 with 100,000 people per group. That is why an effect size should be reported with every p-value. In the study method example, the effect size (Cohen's d, using the average of the two sample variances) is about 1.96, conventionally a "large" effect.
5. "p = 0.049 and p = 0.051 lead to opposite conclusions"
0.05 is a convenience, not a law of nature. The two values carry almost the same strength of evidence. Also, running many tests and reporting only the significant ones (multiple comparisons), or collecting data until the result becomes significant (p-hacking), pushes the real Type I error rate far above α.
Power and sample size
When reading a non-significant result, also think about power, the probability of detecting a real effect as significant. Suppose the true difference is an effect size of d = 0.5 (medium) and you run a two-sided test at α = 0.05. With 20 people per group the power is only about 34%. With 64 per group it is about 80%. If a study with 20 per group finds nothing significant, it more likely lacked the power to find an effect than showed there is none.
Key points of the ASA statement
The 2016 ASA statement's principles can be summarized as:
- p-values can indicate how incompatible the data are with a specified statistical model.
- p-values do not measure the probability that a hypothesis is true, or the probability that the data were produced by chance alone.
- Scientific conclusions and decisions should not be based only on whether a p-value passes a specific threshold.
- Proper inference requires full and transparent reporting.
- A p-value does not measure the size of an effect or the importance of a result.
- By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
Two-sided and one-sided tests
- Two-sided test: tests "there is a difference (in either direction)". Extreme values in both tails count.
- One-sided test: tests a stated direction, such as "A is greater than B". Only one tail counts.
For a symmetric distribution such as t, when the observed direction matches the alternative, the one-sided p-value is half the two-sided one. In the example, the one-sided p-value for "A is higher" is about 0.00084. Use a one-sided test only when the direction was fixed on theoretical grounds before seeing the data. Switching to one-sided after the fact to halve the p-value is a misuse.
How to report well
- Give the exact p-value (
p = 0.0017rather thanp < 0.05). - Report the effect size and a confidence interval.
- State the test used, whether it was one- or two-sided, the sample sizes and how many tests were run.
This site's t-test calculator shows the t value, degrees of freedom, one- and two-sided p-values and the effect size together.
Key takeaways
- A p-value is the probability of a result at least as extreme as the observed one, assuming the null hypothesis is true.
- It is not the probability that the null is true, that the effect is real, or that the result will replicate.
- A non-significant result is not evidence of no effect, and a small p-value does not mean a large effect.
- The two-group example (Welch t) gives t ≈ 3.924, about 13.26 degrees of freedom, two-sided p ≈ 0.0017 and one-sided p ≈ 0.00084.
- Report exact p-values with effect sizes and confidence intervals, and fix the direction of a one-sided test in advance.