Skip to content
Ahmad Ammar

2. The data

Four files hold everything we need. This chapter downloads them, looks inside each one, and joins them into a single table with one row per year.

2.1 Download the data

Every number in this notebook comes from four public files. Each is downloaded once into a folder called data next to the notebook, and reused after that. Writing the download as code, rather than clicking in a browser, means anyone can repeat it exactly.

File

What it holds

Who publishes it

hadcrut5.csv

Global temperature anomaly, one row per year since 1850

Met Office Hadley Centre and University of East Anglia

conus.csv

Temperature of the contiguous United States, one row per year since 1895

NOAA

owid_co2.csv

CO₂ emitted by every country, every year

Our World in Data

forcing.csv

Radiative forcing by cause, one row per year since 1750

Indicators of Global Climate Change

In [2]
DATA = Path("data")
DATA.mkdir(exist_ok=True)
SOURCES = {
    "hadcrut5.csv": "https://www.metoffice.gov.uk/hadobs/hadcrut5/data/HadCRUT.5.0.2.0/analysis/diagnostics/"
                    "HadCRUT.5.0.2.0.analysis.summary_series.global.annual.csv",
    "conus.csv": "https://www.ncei.noaa.gov/access/monitoring/climate-at-a-glance/national/time-series/110/tavg/12/12/1895-2024/data.csv",
    "owid_co2.csv": "https://raw.githubusercontent.com/owid/co2-data/master/owid-co2-data.csv",
    "forcing.csv": "https://raw.githubusercontent.com/ClimateIndicator/forcing-timeseries/main/output/ERF_best_aggregates_1750-2024.csv",
}
for name, url in SOURCES.items():
    target = DATA / name
    if not target.exists():
        req = urllib.request.Request(url, headers={"User-Agent": "climate-data-notebook/1.0"})
        target.write_bytes(urllib.request.urlopen(req, timeout=120).read())
    print(f"{name:14s} {target.stat().st_size / 1e6:6.2f} MB")
Out [2]
hadcrut5.csv     0.01 MB
conus.csv        0.00 MB
owid_co2.csv    14.38 MB
forcing.csv      0.10 MB

2.2 Look inside each file

Before using any data, look at it: how many rows and columns, what the columns are called, and what a few rows contain. pd.read_csv reads a file into a table called a DataFrame; .shape gives (rows, columns); .head() shows the first rows.

HadCRUT5. The anomaly is the temperature minus the average of 1961–1990, so 0 means "normal", positive means warmer than normal. The two limits give the range within which the true value lies with 95% probability.

In [3]
g = pd.read_csv(DATA / "hadcrut5.csv")
print(g.shape)
print(g.head(3).to_string())
print("...")
print(g.tail(2).to_string())
Out [3]
(176, 4)
   Time  Anomaly (deg C)  Lower confidence limit (2.5%)  Upper confidence limit (97.5%)
0  1850        -0.417711                      -0.589256                       -0.246166
1  1851        -0.233350                      -0.411868                       -0.054832
2  1852        -0.229399                      -0.409382                       -0.049416
...
     Time  Anomaly (deg C)  Lower confidence limit (2.5%)  Upper confidence limit (97.5%)
174  2024         1.180800                       1.141589                        1.220011
175  2025         1.190283                       1.048175                        1.332391

The last row is the current year, which is not finished, so we will stop at 2024.

NOAA contiguous US. The file starts with lines beginning #, which are notes; comment="#" skips them. The Date 189512 means the twelve months ending December 1895. Values are in degrees Fahrenheit.

In [4]
us_raw = pd.read_csv(DATA / "conus.csv", comment="#")
print(us_raw.shape)
print(us_raw.head(3).to_string())
Out [4]
(130, 2)
     Date  Value
0  189512  50.33
1  189612  51.98
2  189712  51.54

Our World in Data. One row per country per year, with 79 columns. We keep only the rows for the whole World and two columns: co2, the CO₂ emitted that year, and cumulative_co2, the running total since records began. Both are in millions of tonnes; dividing by 1,000 gives gigatonnes (Gt).

In [5]
owid = pd.read_csv(DATA / "owid_co2.csv", usecols=["country", "year", "co2", "cumulative_co2"])
print(owid.shape, "rows and columns;", owid.country.nunique(), "countries and regions")
world = owid[owid.country == "World"].drop(columns="country")
print(world[world.year.isin([1900, 1950, 2000, 2024])].to_string(index=False))
Out [5]
(50411, 4) rows and columns; 254 countries and regions
 year       co2  cumulative_co2
 1900  1962.884       45105.176
 1950  5930.406      231890.109
 2000 25511.482     1045316.812
 2024 38598.578     1849123.875

Radiative forcing. Radiative forcing is the extra energy, in watts per square metre of the Earth's surface (W/m²), flowing into the climate because something changed: more greenhouse gases (positive, warming), more sulfate pollution (negative, cooling), a brighter Sun, a volcanic eruption. Each column is one cause; anthro is all human causes together and total adds the natural ones.

In [6]
f = pd.read_csv(DATA / "forcing.csv").rename(columns={"Unnamed: 0": "year"})
f["year"] = f["year"].astype(int)
print(f.shape)
print(f[["year", "CO2", "aerosol", "solar", "volcanic", "anthro", "total"]].iloc[[100, 200, 274]].round(3).to_string(index=False))
Out [6]
(275, 19)
 year   CO2  aerosol  solar  volcanic  anthro  total
 1850 0.140   -0.083  0.008     0.244   0.114  0.365
 1950 0.648   -0.714  0.057     0.218   0.344  0.618
 2024 2.332   -1.066  0.091     0.238   2.974  3.303

2.3 Prepare: units, baselines and one joined table

Three preparation steps. Units: convert NOAA's Fahrenheit to Celsius with °C = (°F − 32) × 5/9. Baseline: turn the US temperatures into anomalies by subtracting their 1901–2000 mean. Join: put everything into one table with one row per year, matching rows on the year column with merge.

A worked example of the conversion for 2024: NOAA gives 55.48 °F, so (55.48 − 32) × 5/9 = 13.04 °C.

In [7]
us = pd.DataFrame({"year": us_raw["Date"] // 100, "conus": (us_raw["Value"] - 32) * 5 / 9})
base = us.loc[us.year.between(1901, 2000), "conus"].mean()
print(f"US 1901-2000 mean: {base:.2f} °C; 2024: {us.conus.iloc[-1]:.2f} °C, anomaly {us.conus.iloc[-1] - base:+.2f} °C")
us["conus"] -= base

d = (g.rename(columns={"Time": "year", "Anomaly (deg C)": "global"})[["year", "global"]]
       .merge(us, on="year", how="left")
       .merge(world, on="year", how="left")
       .merge(f[["year", "anthro", "aerosol", "solar", "volcanic", "total"]], on="year", how="left"))
d = d[d.year <= 2024].reset_index(drop=True)
print(d.shape)
print(d[d.year.isin([1900, 1940, 1982, 2024])].round(3).to_string(index=False))
Out [7]
US 1901-2000 mean: 11.12 °C; 2024: 13.04 °C, anomaly +1.93 °C
(175, 10)
 year  global  conus       co2  cumulative_co2  anthro  aerosol  solar  volcanic  total
 1900  -0.234  0.417  1962.884       45105.176   0.124   -0.366 -0.037     0.179  0.265
 1940   0.076 -0.072  4884.662      181025.359   0.304   -0.658  0.040     0.227  0.571
 1982   0.034 -0.378 18729.742      640191.688   0.809   -1.393  0.067    -0.244  0.632
 2024   1.181  1.928 38598.578     1849123.875   2.974   -1.066  0.091     0.238  3.303

how="left" keeps every year of the first table; where another table has no value for that year (the US series starts in 1895) the cell is NaN, which means "no value". Missing values are skipped by the calculations below.