4. Testing research code¶
4.1 Why test¶
A script that runs without errors can still be wrong. A test runs your code and checks the answer against something you know.
calc.py
"""A tiny calculation to test."""
def heat_loss(u, area, delta_t):
"""Heat loss in watts: Q = U A dT."""
if u < 0 or area < 0:
raise ValueError("U-value and area cannot be negative")
return u * area * delta_t
test_calc.py
"""Tests check the code against answers worked out by hand. Run: python -m pytest"""
import pytest
from calc import heat_loss
def test_known_answer():
assert heat_loss(2.1, 10, 15) == pytest.approx(315)
def test_no_temperature_difference_means_no_loss():
assert heat_loss(2.1, 10, 0) == 0
def test_impossible_input_is_refused():
with pytest.raises(ValueError):
heat_loss(-1, 10, 15)
pip install pytest
python -m pytest
What you should see
$ python -m pytest
============================= test session starts =============================
platform win32 -- Python 3.12.12, pytest-9.2.0, pluggy-1.7.0
rootdir: ...\py
plugins: platformdirs-4.14.2
collected 3 items
test_calc.py ... [100%]
============================== 3 passed in 0.09s ==============================
Each dot is a passing test. To see a failure, I changed u * area * delta_t to u * area + delta_t in calc.py, as if by a slip of the finger. The code still runs without any error, but the tests catch it:
What you should see when a test fails (first lines trimmed)
test_calc.py FF. [100%]
================================== FAILURES ===================================
______________________________ test_known_answer ______________________________
def test_known_answer():
> assert heat_loss(2.1, 10, 15) == pytest.approx(315)
E assert 36.0 == 315 ± 3.1e-04
E
E comparison failed
E Obtained: 36.0
E Expected: 315 ± 3.1e-04
test_calc.py:8: AssertionError
________________ test_no_temperature_difference_means_no_loss _________________
def test_no_temperature_difference_means_no_loss():
> assert heat_loss(2.1, 10, 0) == 0
E assert 21.0 == 0
E + where 21.0 = heat_loss(2.1, 10, 0)
test_calc.py:12: AssertionError
=========================== short test summary info ===========================
FAILED test_calc.py::test_known_answer - assert 36.0 == 315 ± 3.1e-04
FAILED test_calc.py::test_no_temperature_difference_means_no_loss - assert 21...
========================= 2 failed, 1 passed in 0.33s =========================
4.2 Kinds of test for research¶
Test |
Question |
Example |
|---|---|---|
Known answer |
Does it match a hand calculation? |
x 10 x 15 = 315 |
Limit case |
Does it behave at the extremes? |
No temperature gap gives no heat loss |
Invariant |
Does a rule always hold? |
Total price equals total value |
Simulated data |
Can it recover a truth you planted? |
A model finds a jump you added |
Data check |
Is the data as expected? |
No duplicates, values in range |
Independent recalculation |
Do two separate methods agree? |
The same estimate from different code |
Rendered output |
Does the result look right to readers? |
A browser opens each page |
4.3 Lessons from real faults¶
These all happened in my own projects:
-
An independent recalculation confirmed an estimate exactly, and adding a check for correlated data showed its confidence interval was too narrow.
-
A check of stated numbers against results found a sample size typed wrongly into a paper.
-
A test for labels found age groups named for the wrong years, though the numbers ran fine.
-
A browser test found broken links, unreadable buttons and maths that displayed as raw code. Its first version missed some faults, because it looked for error messages and these made none.
So a passing test only shows that the things you tested held. When a fault gets through, add a test for that kind of fault.
4.4 What tests cannot do¶
They can't show that a method is valid, or an argument sound. That takes reasoning and readers.