Which Statistical Test Should You Use? A Practical Selection Guide

You have collected data but are unsure which test to run. Fortunately, for most basic analyses the right test follows from three questions. This guide lays out those questions, a decision table, and each test's assumptions and how to check them.

Three questions to answer first

1. What type is the outcome variable?

• Continuous (numeric): values you can average, such as height, score, revenue or reaction time
• Categorical: values that fall into categories, such as gender, purchased or not, or preferred brand. The data are usually summarized as counts.

Ordered categories such as a 5-point Likert item (ordinal data) are usually analyzed with nonparametric tests for a single item, while sums of several items are often treated as continuous.

2. How many groups are you comparing?

Check whether you compare one group against a reference value, two groups, or three or more. If the goal is the relationship between two variables rather than a group comparison, go to correlation and regression.

3. Are the measurements paired?

• Independent: two groups made up of different people (class A and class B)
• Paired: the same subjects measured twice (before and after), or natural pairs such as twins or spouses

Analyzing paired data with an independent test mixes individual differences into the error and sharply reduces power. Analyzing independent data with a paired test is simply wrong.

Decision table

A few notes:

• Use Welch's version as the default independent t-test. Student's t-test, which assumes equal variances, gets its error rate wrong when group variances or sizes differ, while Welch's test loses almost nothing when variances are equal.
• Do not run many t-tests for three or more groups. Testing every pair at α = 0.05 inflates the overall Type I error rate. Use ANOVA for the overall difference, then post hoc tests.
• Comparing two proportions (conversion rate A vs B) can be done with a chi-square test of independence on a 2×2 table; it gives the same result as a two-proportion z-test.

Example: 2×2 chi-square test of independence

Two ad headlines were each shown to 50 people, and clicks were recorded.

The outcome (clicked or not) is categorical and so is the headline, so we use the chi-square test of independence.

1. Expected counts: row total × column total / grand total. A-clicked is 50 × 48 / 100 = 24, A-not clicked is 50 × 52 / 100 = 26, and B is likewise 24 and 26.
2. (observed − expected)² / expected for each cell: (30−24)²/24 = 1.5, (20−26)²/26 ≈ 1.3846, (18−24)²/24 = 1.5, (32−26)²/26 ≈ 1.3846
3. Chi-square statistic: χ² ≈ 5.769
4. Degrees of freedom: (rows − 1) × (columns − 1) = 1
5. p-value: about 0.0163 (the critical value at α = 0.05 is about 3.841)

Since p < 0.05, we conclude that headline and clicking are related. The click rate is 60% for A and 36% for B. With Yates' continuity correction, the same data give χ² ≈ 4.848 and p ≈ 0.0277, slightly more conservative. State which version you used.

Checking assumptions

Every test gives accurate p-values only when its conditions hold. The main assumptions and how to check them:

Independence

Common to all tests. It is violated when one person contributes several responses, or when data are clustered, such as students in the same class. Judge it from how the data were collected rather than with a statistical test.

Normality (t-tests, ANOVA, the test of Pearson's correlation)

• Strictly, what needs to be near normal is the distribution of the sample mean, not the raw data. Thanks to the central limit theorem, with about 30 or more per group the t-test is fairly robust unless the distribution is very skewed.
• For small samples, use histograms, box plots and Q-Q plots to look for severe skew or outliers.
• Normality tests such as Shapiro-Wilk miss violations in small samples and flag trivial ones in large samples, so combine them with plots.

Equal variances

Welch's t-test removes this concern. For ANOVA, check whether the ratio of group standard deviations exceeds 2, and consider Welch's ANOVA if they differ a lot.

Expected counts (chi-square tests)

Chi-square tests are approximations and become inaccurate with small expected counts. A common rule is every expected count at least 5 (a looser version: no more than 20% of cells below 5 and none below 1). For a 2×2 table that breaks the rule, use Fisher's exact test. In the example all expected counts are 24 or more, so there is no problem.

Linearity (Pearson correlation, regression)

Draw a scatter plot first. If the relationship is curved, Pearson's r understates it. For a monotonic relationship, Spearman's correlation is the alternative.

When to use nonparametric tests

• When the sample is small and the distribution is badly skewed or has outliers
• When the data are ranks or ordinal
• When you care about medians or the location of the distribution

Nonparametric tests assume less but have somewhat lower power when normality actually holds. Also note that the Mann-Whitney U test does not strictly test "equal means", so do not interpret its result as a difference in means.

This site offers t-test (one-sample, independent Welch and paired), chi-square (goodness-of-fit and independence) and correlation and regression calculators. Pick a method with the table above and compute it right away in the matching calculator.

Key takeaways

• Three things determine the test: outcome type (continuous or categorical), number of groups and pairing.
• Use Welch's t-test for two independent means, the paired t-test for before and after, and ANOVA for three or more groups.
• The association of two categorical variables is tested with the chi-square test of independence; the example 2×2 table gives χ² ≈ 5.769, 1 degree of freedom, p ≈ 0.0163.
• For chi-square, check that expected counts are at least 5, and use Fisher's exact test otherwise.
• When assumptions fail badly, consider nonparametric alternatives such as Wilcoxon, Mann-Whitney, Kruskal-Wallis and Spearman.

→ Calculate it now: t-test

Updated 2026-09-23