Reading Scientific Papers Is Hard

A Tale Of Two Studies

We’re working on an analysis of forestation – i.e., planting trees – as a solution to climate change. This of course entails reading a bunch of papers. Making sense of what they say has been even harder than I’d expected.

We’re specifically looking for projections of how quickly new forests can absorb CO₂. One paper gave numbers that seemed to work out to 6.61 GtC/year¹. Another paper reported 0.12 GtC/year. The difference between 6.61 and 0.12 is equivalent to the difference between the population of the United States, and the population of Colorado. That’s a pretty big discrepancy!

Climate science is tricky, but it’s usually not “off by a factor of 55” tricky. It turns out that most of the discrepancy came down to subtle reading errors on our part, along with at least one important omission in one of the papers. Either we’re dumb, or reading these papers is hard. I respectfully argue for the latter.

What you are about to read is my third attempt to explain how we came out with a 55x error in our reading of these two papers. I had to throw out my first draft and start over, because I realized I was looking at the situation incorrectly. Then I had to throw out my second draft for the same reason. In Never Trust a Number and Never Trust a Projection, I’ve talked about how how easy it is for numbers to mislead. Boy did that play out here!

Note: there’s a lot of technical detail in this post. Don’t worry about following the details; if you just skim along, you’ll still pick up the important points. In fact, feel free to skip over all of the numbers if you aren’t interested in that level of detail.

How Much Wood Would a Wood Planter Plant?

Let’s begin by briefly summarizing each paper, and the figures they give for carbon absorption by forests.

The first paper, published in Science in 2019, is The Global Tree Restoration Potential. It states that forestation can store 205 GtC by 2050, which works out to 6.61 GtC / year.

The second paper, Limited climate change mitigation potential through forestation of the vast dryland regions, was published in 2022. It states that between 2020 and 2100, trees could provide cooling “equivalent to the sequestration of only 9.7 Gt C”. 9.7 GtC over 80 years is 0.12 GtC / year.

As best I can understand, both papers use a similar approach. They use satellite images to create a detailed map of growing conditions – topography, soil, and climate – for the entire Earth. The map has a resolution of roughly half a mile², which is a pretty good level of detail when we’re considering the entire planet.

They then feed in known data about existing forests: in one case, “78,774 direct photo-interpretation measurements of tree cover”. Using this, they train a machine learning model to extrapolate the potential for tree cover on each half-mile square of the Earth. By adding up the numbers for all land that is not already in use (for agriculture or other purposes), they get an estimate for the global potential of tree planting.

In what follows, I’ll refer to the two papers by their first listed authors: Bastin for the first paper, and Rohatyn for the second. This is shorthand for “Bastin et al” and “Rohatyn et al”.

We Thought 2050, We Thought Wrong

Allowing for the temporal flux correction, that reads 2050, right?

I mentioned that the Bastin paper models storage of 205 Gt carbon by 2050. It turns out that we hallucinated the 2050 part.

The paper doesn’t actually say how long new forests take to grow. It does talk about the year 2050 in other contexts³. But it never connects 2050 with tree growth.

Because no time frame other than 2050 is mentioned in the paper, we assumed that 2050 was also the time period for which they were modeling forest growth. But after reading a bunch of other sources (including the Rohatyn paper), it’s clear that a reasonable time for a forest to reach maturity is more like 80 years.

In our defense, the first paragraph of the paper mentions both “205 gigatonnes of carbon” and “by 2050”, and makes no effort to clarify that they are not connected. Academic writing frequently omits important context, presumably on the assumption that you’re already deeply familiar with the subject matter. If not, it’s easy to go astray.

If we divide Bastin’s 205 Gt by 80 years, we get 2.56 GtC / year. That’s still a lot more than Rohatyn’s 0.12 GtC / year, but at least we’re getting closer.

Bastin Didn’t Model Forests Correctly

The issue of Science in which the Bastin paper appears, also contains some comments pointing out several important flaws in the paper.

First, the paper miscalculates carbon sequestered per acre. It assumes that if a mature forest contains X tons of carbon, then planting a similar forest will absorb another X tons. However, the areas to be planted – mostly “sparse vegetation, grasslands, and degraded bare soils” – already contain quite a bit of carbon (much of it below ground). If a certain patch contains 200 tons, and eventually grows into a mature forest containing 500 tons, then forestation has only pulled down 300 tons of new carbon. According to two of the comments⁴, the Bastin paper fails to take this into account, and as a result gives estimates that are about double what they should be.

Second, it ignores “albedo effects”. Leaves are relatively dark, so planting trees on light-colored land has an effect similar to wearing dark clothing on a sunny day: the trees absorb sunlight and get warm. This heating effect reduces the climate benefit; in many areas, additional tree cover would be outright detrimental.

Third, they earmark natural grasslands and savannas for forestation. One comment notes that “herbivores and browsers—including insects—are ecological engineers stopping huge areas of natural grasslands and savannas from converting to forest”; another points out that in these areas, “fires and large herbivores have maintained low tree cover for millions of years”. Basically, there's a reason these areas, when pristine, weren't forests, and so attempts to turn them into forests are likely to fail.

At this point, I was developing the impression that the Bastin paper was written by a team that mostly understood machine learning, and didn’t know how to properly apply their results to climate change. The comments suggest that the estimate of carbon absorption is too high by factors ranging from 2x to 5x. If we compromise on 3x, then 205GtC becomes 68GtC; over 80 years, that’s 0.85 GtC/year, vs. 0.12 GtC/year for Rohatyn et al. Getting closer!

We Misunderstood Rohatyn’s Model

Quick, what color is the first word?

The next problem lies in our reading of the Rohatyn paper’s abstract. It presents a number which we thought applied to the entire Earth, but it turns out we were way off. Here’s the relevant section:

Forestation of the vast global drylands has been considered a promising climate change mitigation strategy. However, its actual climatic benefits are uncertain because the forests’ reduced albedo can produce large warming effects. Using high-resolution spatial analysis of global drylands, we found 448 million hectares suitable for afforestation. This area’s carbon sequestration potential until 2100 is 32.3 billion tons of carbon (Gt C), but 22.6 Gt C of that is required to balance albedo effects.

The last sentence is fairly clear in projecting a net climate benefit of 9.7 Gt of carbon over 80 years⁵. And the first sentence refers to “forestation”, a broad term which basically refers to any planting of new trees. However, the last two sentences switch to discussing afforestation, defined as “the act or process of establishing a forest especially on land not previously forested”. In other words, “afforestation” excludes the restoration of historical forest land (reforestation) or the planting of additional trees in land that already includes some tree cover (densification). Initially, none of us picked up on this subtle shift from “forestation” to “afforestation”; I finally noticed it on a fourth reading!

The other issue is that they are only modeling afforestation on “drylands”, defined as “an aridity index of less than 0.65”. Basically, this seems to exclude rain forests and other humid areas. So, 9.7 Gt is not their model of global carbon drawdown potential, it is only their model for afforestation in dry areas. They do go on to present a projection for all possible forestation activity across the entire globe, and it is a much higher number: 113.6 GtC.

Using 113.6 GtC for the Rohatyn paper, and dividing by 80 years, we get a carbon drawdown rate of 1.42 GtC/year, vs. our updated 0.85 GtC/year for Bastin. Our corrections have pushed the numbers so far that they’ve actually switched places; the Rohatyn figure now looks higher.

Bastin Strikes Back

Remember when I said that Bastin et al didn’t model forests correctly? Turns out that might be wrong.

Recall that the criticisms of the paper were as follows:

1. Giving forests credit for carbon that was present before the forest was planted.

2. Ignoring the impact of herbivores and fire.

3. Ignoring albedo effects (forests are dark, and so absorb the sun’s heat).

It turns out that Bastin et al posted a detailed response, which directly rebuts the first two points. They go into detail as to how they computed carbon per forest acre. It was a sophisticated analysis, and does take those first two factors into account.

My idea that Bastin’s team were experts in machine learning rather than climate science was way off base. If I had bothered to look as far as far as the footnotes on their first page, I would have seen that Bastin himself is at the “Department of Environmental Systems Science, Institute of Integrative Biology, ETH-Zürich”; he has a degree in tropical forestry.

They do acknowledge that they ignore albedo effects, stating that this is “beyond the scope of the present study”. Another example of academic writing omitting important context.

The details of the response are interesting. Bastin et al call out some errors that their critics made in interpreting their own paper – showing that even academics in the field can misread one another’s work. I’ve included some snippets in an appendix at the end of this post.

I don’t know how to update our math based on this. Earlier, I downgraded Bastin’s figure by 3x based on the three criticisms above. Accepting the rebuttal of the first two, but noting that the third criticism stands, I’m going to wave my hands and change the 3x to 1.5x. That gets us to a Bastin figure of 1.7 GtC/year, vs. 1.42 GtC/year for Rohatyn. Pretty close!

Boy, I’m Glad That’s Sorted

After many rounds of review, we’ve refined our interpretation of the two papers until they yield estimates that are only 20% apart (1.7 vs. 1.42 GtC/year). Such close agreement must mean that we’ve gotten to the actual truth, right?

Hah! Don’t count on it. Depending on how you count, we’ve identified up to eight distinct errors in our initial analysis⁶. If there were eight, there were probably more than eight. It took four readings of Rohatyn to figure out that the headline number only applied to afforestation; who knows what I’d find on a fifth reading?

Eight corrections in, the figures more or less agree, but that could easily be coincidence. If they had happened to agree after five or six corrections, I’d probably have stopped there, meaning I might have missed two or three errors. As the saying goes, “if you torture the data long enough, it will confess to anything”.

It’s also worth remembering that much of the fundamental science here is not settled. For instance, there is a lot that’s still not understood about underground carbon stores. Both papers are rough attempts to model a complicated planet, and they could both be off in the same direction.

At the end of the day, the fact that we’ve found two numbers that more or less match doesn’t mean that we have truth, it just means we have two numbers that match.

Let’s Be Careful Out There

No, I’m not dating myself here, why would you say that?

On a first reading of the Bastin paper, we determined that forestation can remove about 6.61 GtC / year. On a first reading of Rohatyn, we got 0.12 GtC / year. After a lot more work, we’re converging on a number that is nowhere near either of those initial figures.

In other words: imagine that you have a scientific question in mind. You find a paper published in a prestigious journal. Your question is answered in the first paragraph, but you want to make sure you’re understanding it correctly, so you take the time to read the entire paper. Even so, there’s quite a good chance that the conclusion you’re coming away with is way off. On these two papers, in our attempt to read them correctly, we went zero for two.

One big problem is that these papers omit context. Bastin didn’t mention their time frame, nor that they were ignoring albedo effects; Rohatyn buried the fact that their headline figures only covered dryland afforestation.

The various comments and rebuttals were helpful, but you have to dig around to find them – they’re not in the PDF.

How do you guard against these problems? The best advice I can give:

1. Do your best to find multiple sources. If they disagree, dig in to understand why.

2. Maintain an attitude of humility. Your first reading may be incorrect. Your second reading may be differently incorrect.

3. Never Trust a Number!

Appendix: Snippets from Bastin Et Al’s Response To Critics

As promised earlier, here are some excerpts from Response to Comments on “The global tree restoration potential” (Bastin et al’s response to the comments on their paper). I’m not going to provide full context or explanation, but by skimming these excerpts, you should get some idea of the nature of the (extensive!) disagreements and misunderstandings.

The discrepancies between our estimate and their estimates arise from (i) misinterpretations or confusion between the definitions of forest cover and associated carbon pools, and (ii) a lack of sufficient detail in the original manuscript on how existing carbon in potential restoration areas was removed for estimating the global restoration potential. We clarify these points here.

Three of the four examples provided are based on a different definition of forest: namely forest area, rather than tree canopy cover.

We estimated that there is 1657 Mha of forest area available [table S2 of (1)], which contains 900 Mha available as cumulative tree canopy cover. Because these papers (7, 10, 11) were addressing forest area, the carbon density estimates would need to be scaled to 1657 Mha instead of 900 Mha. Correcting for this consideration of forest area almost doubles the carbon estimates proposed by Lewis et al.

The numbers provided by Lewis et al. in their restoration study (7) concern only two of the five carbon pools for vegetation ecosystems (i.e., aboveground and belowground plant biomass). Restoring forest ecosystems would actually have an impact on all five pools of carbon, including soil, litter, and dead wood (12). In our analysis, we included all five, which drastically increases the amount of carbon expected to be stored in restored forests.

After subtraction of the existing carbon content from the potential global carbon content that could be stored in areas available for restoration, the global carbon gain from tree restoration potential ranges between 133.2 and 276.2 GtC with a mid-range value of 204.7 GtC. This range reflects the uncertainty in calibrating the biome-specific carbon density values to a baseline percent tree cover (see Table 1).

We completely agree that changes in forest cover resulting from restoration would also affect the climate through a range of mechanisms including changes in surface albedo and evapotranspiration. … Calculating the changes in albedo and evapotranspiration associated with restoration is beyond the scope of the present study.

Veldman et al. stress that our model had low predictive power across many of the open-canopy biomes, suggesting that it fails to account for natural fire and the presence of large mammals. Here, they have misinterpreted the uncertainty of our model. First, natural fires and large mammals exist in protected areas. They are therefore indirectly accounted for in our model. Second, as natural fire cannot be distinguished from human-made fire, it cannot be accounted for as a variable of the model to extrapolate the natural tree cover outside protected areas. Third, the high uncertainty in intermediate tree cover is due to the general low occurrence of intermediate tree cover.

1. GtC is short for “billions of tons of carbon”. Like all figures in today’s post, this is a measurement of carbon, not carbon dioxide. As I’ll explain in a later post, one ton of carbon corresponds to 3.67 tons of CO₂. So if you want to put these numbers in the context of CO₂ metrics, you need to multiply by 3.67.

2. To be precise: 30 arc seconds.

3. For example: “We estimate that if we cannot deviate from the current trajectory, the global potential canopy cover may shrink by ~223 million hectares by 2050, with the vast majority of losses occurring in the tropics.”

4. this comment and this comment

5. 32.3 Gt of actual carbon absorption, minus 22.6 Gt to balance the albedo effects from planting dark forest on light-colored land.

6. (1) we misinterpreted Bastin as giving results for 2050, (2) Bastin ignored pre-existing soil carbon, (3) Bastin ignored herbivores and fires, (4) Bastin ignored albedo effects, (5) Rohatyn only modeled afforestation, (6) Rohatyn only considered drylands, (7) point 2 was incorrect, (8) point 3 was incorrect.