Skip to content
Ahmad Ammar

6. Many tests, and how to report them

6.1 Many tests at once: false discoveries

Thinking it through. What do we want to measure? Which pixels of a map really changed. What kind of data? One p-value per pixel, thousands of them. How do such data behave? Where nothing changed, p-values are spread evenly from 0 to 1, so 5 % of them fall below 0.05 by chance. Which method, and why? Benjamini–Hochberg, which controls the share of false findings among those reported: the right target for a map, where a few false pixels are tolerable but a map made mostly of them is not.

When a map is made by testing every pixel, some pixels will be "significant" by chance. With α = 0.05, about 5% of pixels with no real change pass the test. The Benjamini–Hochberg procedure controls the false discovery rate (FDR): among the results called significant, the expected share that are false is kept below a chosen q (say 5%). Simulation: 1,000 pixels, of which 900 have no trend and 100 a real one.

In [12]
n_null, n_real, years = 900, 100, np.arange(30)
pvals, truth = [], []
for i in range(n_null + n_real):
    slope = 0.0 if i < n_null else 0.04
    y = slope * years + rng.normal(0, 0.3, len(years))
    pvals.append(stats.linregress(years, y).pvalue); truth.append(i >= n_null)
pvals, truth = np.array(pvals), np.array(truth)
raw = pvals < 0.05
bh = stats.false_discovery_control(pvals, method="bh") < 0.05
fig, ax = plt.subplots(figsize=(6.5, 2.6))
ax.hist([pvals[~truth], pvals[truth]], bins=20, range=(0, 1), stacked=True, color=["#cbd5e1", "#b91c1c"], label=["no real trend", "real trend"])
ax.axvline(0.05, color="k", lw=1)
ax.set(xlabel="p-value of each simulated pixel", ylabel="pixels", title="p-values from 900 pixels with no trend and 100 with one")
ax.legend(frameon=False, fontsize=8); plt.tight_layout(); plt.show()
for name, call in [("p < 0.05 for each pixel", raw), ("Benjamini-Hochberg, q = 0.05", bh)]:
    tp, fp = (call & truth).sum(), (call & ~truth).sum()
    print(f"{name:30s} called significant {call.sum():4d}: real {tp:3d}, false {fp:3d}, share false {fp / max(call.sum(), 1):.0%}")
Out [12]

Figure

Out [12]
p < 0.05 for each pixel        called significant  128: real 100, false  28, share false 22%
Benjamini-Hochberg, q = 0.05   called significant  105: real 100, false   5, share false 5%

How to read the figure. Each bar counts pixels whose p-value fell in that range. Pixels with no real trend (grey) spread evenly from 0 to 1: under H₀ every p-value is equally likely, so about 5% of them land below 0.05 by chance. Pixels with a real trend (red) pile up near 0. The black line is the usual 0.05 cut-off: everything to its left is called significant, including the grey sliver.

What the result says. Testing each pixel at 0.05 calls 28 no-change pixels significant, so more than a fifth of the "changes" are false. Benjamini–Hochberg keeps the share of false discoveries near 5%. Here it still finds all 100 real trends because they are strong; with weaker real effects it would miss some, which is the price of controlling false discoveries. On a real map with millions of pixels, this is the difference between a map of change and a map of noise.

6.2 Reading any test result

Every test reported in a paper should come with the same set of facts. This table collects the ones used above.

In [13]
summary = pd.DataFrame([
    ["z-test", "Is one value unusual?", "a value, a mean, a known SD", "z = (x − μ)/σ", "none", "unitless (number of SDs)"],
    ["t-test for a slope", "Is there a trend?", "pairs (time, value)", "t = b / SE(b)", "n − 2 (fewer if autocorrelated)", "b in units of y per unit time"],
    ["Welch t-test", "Do two means differ?", "two samples", "t = (x̄₁ − x̄₂)/√(s₁²/n₁ + s₂²/n₂)", "Welch–Satterthwaite", "difference in units of y"],
    ["Mann–Whitney U", "Does one group tend to be larger?", "two samples", "U from ranks", "none", "U is a count of pairs"],
    ["Pearson r test", "Are two variables linearly related?", "pairs (x, y)", "t = r√(n−2)/√(1−r²)", "n − 2", "r unitless, −1 to 1"],
    ["Mann–Kendall", "Is there a monotonic trend?", "a time series", "Kendall τ of value with time", "none", "τ unitless, −1 to 1"],
    ["Chi-square", "Are two categories related?", "a table of counts", "χ² = Σ(O − E)²/E", "(rows − 1)(cols − 1)", "χ² unitless; Cramér's V 0 to 1"],
    ["Benjamini–Hochberg", "Which of many tests to believe?", "many p-values", "ranked p ≤ (k/m)·q", "none", "adjusted p-values"],
], columns=["test", "question", "inputs", "statistic", "degrees of freedom", "units"])
print(summary.to_string(index=False))
Out [13]
test                            question                      inputs                          statistic              degrees of freedom                          units
            z-test               Is one value unusual? a value, a mean, a known SD                      z = (x − μ)/σ                            none       unitless (number of SDs)
t-test for a slope                   Is there a trend?         pairs (time, value)                      t = b / SE(b) n − 2 (fewer if autocorrelated)  b in units of y per unit time
      Welch t-test                Do two means differ?                 two samples t = (x̄₁ − x̄₂)/√(s₁²/n₁ + s₂²/n₂)             Welch–Satterthwaite       difference in units of y
    Mann–Whitney U   Does one group tend to be larger?                 two samples                       U from ranks                            none          U is a count of pairs
    Pearson r test Are two variables linearly related?                pairs (x, y)                t = r√(n−2)/√(1−r²)                           n − 2            r unitless, −1 to 1
      Mann–Kendall         Is there a monotonic trend?               a time series       Kendall τ of value with time                            none            τ unitless, −1 to 1
        Chi-square         Are two categories related?           a table of counts                   χ² = Σ(O − E)²/E            (rows − 1)(cols − 1) χ² unitless; Cramér's V 0 to 1
Benjamini–Hochberg     Which of many tests to believe?               many p-values                 ranked p ≤ (k/m)·q                            none              adjusted p-values

6.3 A checklist for reporting a test

  1. The question and the null hypothesis, in words.

  2. The data: what, where, when, how many observations, and how many of them are independent.

  3. The test, and why its assumptions hold, or how they were relaxed.

  4. The statistic with its degrees of freedom, for example \(t(25.0) = 13.3\).

  5. The p-value as a number, not only “\(p<0.05\)”.

  6. An effect size and a confidence interval, in the units of the data.

  7. If many tests were run, how false discoveries were controlled.

  8. What the result means for the question, and what it does not show.