2. The data¶
Four files hold everything we need. This chapter downloads them, looks inside each one, and joins them into a single table with one row per year.
2.1 Download the data¶
Every number in this notebook comes from four public files. Each is downloaded once into a folder called data next to the notebook, and reused after that. Writing the download as code, rather than clicking in a browser, means anyone can repeat it exactly.
File |
What it holds |
Who publishes it |
|---|---|---|
|
Global temperature anomaly, one row per year since 1850 |
Met Office Hadley Centre and University of East Anglia |
|
Temperature of the contiguous United States, one row per year since 1895 |
NOAA |
|
CO₂ emitted by every country, every year |
Our World in Data |
|
Radiative forcing by cause, one row per year since 1750 |
Indicators of Global Climate Change |
DATA = Path("data")
DATA.mkdir(exist_ok=True)
SOURCES = {
"hadcrut5.csv": "https://www.metoffice.gov.uk/hadobs/hadcrut5/data/HadCRUT.5.0.2.0/analysis/diagnostics/"
"HadCRUT.5.0.2.0.analysis.summary_series.global.annual.csv",
"conus.csv": "https://www.ncei.noaa.gov/access/monitoring/climate-at-a-glance/national/time-series/110/tavg/12/12/1895-2024/data.csv",
"owid_co2.csv": "https://raw.githubusercontent.com/owid/co2-data/master/owid-co2-data.csv",
"forcing.csv": "https://raw.githubusercontent.com/ClimateIndicator/forcing-timeseries/main/output/ERF_best_aggregates_1750-2024.csv",
}
for name, url in SOURCES.items():
target = DATA / name
if not target.exists():
req = urllib.request.Request(url, headers={"User-Agent": "climate-data-notebook/1.0"})
target.write_bytes(urllib.request.urlopen(req, timeout=120).read())
print(f"{name:14s} {target.stat().st_size / 1e6:6.2f} MB")
hadcrut5.csv 0.01 MB
conus.csv 0.00 MB
owid_co2.csv 14.38 MB
forcing.csv 0.10 MB
2.2 Look inside each file¶
Before using any data, look at it: how many rows and columns, what the columns are called, and what a few rows contain. pd.read_csv reads a file into a table called a DataFrame; .shape gives (rows, columns); .head() shows the first rows.
HadCRUT5. The anomaly is the temperature minus the average of 1961–1990, so 0 means "normal", positive means warmer than normal. The two limits give the range within which the true value lies with 95% probability.
g = pd.read_csv(DATA / "hadcrut5.csv")
print(g.shape)
print(g.head(3).to_string())
print("...")
print(g.tail(2).to_string())
(176, 4)
Time Anomaly (deg C) Lower confidence limit (2.5%) Upper confidence limit (97.5%)
0 1850 -0.417711 -0.589256 -0.246166
1 1851 -0.233350 -0.411868 -0.054832
2 1852 -0.229399 -0.409382 -0.049416
...
Time Anomaly (deg C) Lower confidence limit (2.5%) Upper confidence limit (97.5%)
174 2024 1.180800 1.141589 1.220011
175 2025 1.190283 1.048175 1.332391
The last row is the current year, which is not finished, so we will stop at 2024.
NOAA contiguous US. The file starts with lines beginning #, which are notes; comment="#" skips them. The Date 189512 means the twelve months ending December 1895. Values are in degrees Fahrenheit.
us_raw = pd.read_csv(DATA / "conus.csv", comment="#")
print(us_raw.shape)
print(us_raw.head(3).to_string())
(130, 2)
Date Value
0 189512 50.33
1 189612 51.98
2 189712 51.54
Our World in Data. One row per country per year, with 79 columns. We keep only the rows for the whole World and two columns: co2, the CO₂ emitted that year, and cumulative_co2, the running total since records began. Both are in millions of tonnes; dividing by 1,000 gives gigatonnes (Gt).
owid = pd.read_csv(DATA / "owid_co2.csv", usecols=["country", "year", "co2", "cumulative_co2"])
print(owid.shape, "rows and columns;", owid.country.nunique(), "countries and regions")
world = owid[owid.country == "World"].drop(columns="country")
print(world[world.year.isin([1900, 1950, 2000, 2024])].to_string(index=False))
(50411, 4) rows and columns; 254 countries and regions
year co2 cumulative_co2
1900 1962.884 45105.176
1950 5930.406 231890.109
2000 25511.482 1045316.812
2024 38598.578 1849123.875
Radiative forcing. Radiative forcing is the extra energy, in watts per square metre of the Earth's surface (W/m²), flowing into the climate because something changed: more greenhouse gases (positive, warming), more sulfate pollution (negative, cooling), a brighter Sun, a volcanic eruption. Each column is one cause; anthro is all human causes together and total adds the natural ones.
f = pd.read_csv(DATA / "forcing.csv").rename(columns={"Unnamed: 0": "year"})
f["year"] = f["year"].astype(int)
print(f.shape)
print(f[["year", "CO2", "aerosol", "solar", "volcanic", "anthro", "total"]].iloc[[100, 200, 274]].round(3).to_string(index=False))
(275, 19)
year CO2 aerosol solar volcanic anthro total
1850 0.140 -0.083 0.008 0.244 0.114 0.365
1950 0.648 -0.714 0.057 0.218 0.344 0.618
2024 2.332 -1.066 0.091 0.238 2.974 3.303
2.3 Prepare: units, baselines and one joined table¶
Three preparation steps. Units: convert NOAA's Fahrenheit to Celsius with °C = (°F − 32) × 5/9. Baseline: turn the US temperatures into anomalies by subtracting their 1901–2000 mean. Join: put everything into one table with one row per year, matching rows on the year column with merge.
A worked example of the conversion for 2024: NOAA gives 55.48 °F, so (55.48 − 32) × 5/9 = 13.04 °C.
us = pd.DataFrame({"year": us_raw["Date"] // 100, "conus": (us_raw["Value"] - 32) * 5 / 9})
base = us.loc[us.year.between(1901, 2000), "conus"].mean()
print(f"US 1901-2000 mean: {base:.2f} °C; 2024: {us.conus.iloc[-1]:.2f} °C, anomaly {us.conus.iloc[-1] - base:+.2f} °C")
us["conus"] -= base
d = (g.rename(columns={"Time": "year", "Anomaly (deg C)": "global"})[["year", "global"]]
.merge(us, on="year", how="left")
.merge(world, on="year", how="left")
.merge(f[["year", "anthro", "aerosol", "solar", "volcanic", "total"]], on="year", how="left"))
d = d[d.year <= 2024].reset_index(drop=True)
print(d.shape)
print(d[d.year.isin([1900, 1940, 1982, 2024])].round(3).to_string(index=False))
US 1901-2000 mean: 11.12 °C; 2024: 13.04 °C, anomaly +1.93 °C
(175, 10)
year global conus co2 cumulative_co2 anthro aerosol solar volcanic total
1900 -0.234 0.417 1962.884 45105.176 0.124 -0.366 -0.037 0.179 0.265
1940 0.076 -0.072 4884.662 181025.359 0.304 -0.658 0.040 0.227 0.571
1982 0.034 -0.378 18729.742 640191.688 0.809 -1.393 0.067 -0.244 0.632
2024 1.181 1.928 38598.578 1849123.875 2.974 -1.066 0.091 0.238 3.303
how="left" keeps every year of the first table; where another table has no value for that year (the US series starts in 1895) the cell is NaN, which means "no value". Missing values are skipped by the calculations below.