Never Trust a Projection
Never Trust a Projection
If you have no intervals, then I have no confidence
No one knows how much energy we'll use in 2050.
Stated like that, it may seem obvious; the future is, famously, not yet written. But… isn’t that the sort of thing that “they” are supposed to know? I mean, I’m not sure who “they” are – I’ve never actually met a “they”1 – but, there’s, like, economists, and all the people who work on the IPCC reports, and so on. They have, you know, models and stuff, right? I mean, someone must know, because we’re always reading articles about how this and that is going to happen by such and such year.
Indeed, people do routinely act as if they did know things like how much energy we’ll use in 2050. Even in academic settings. For instance, I googled “global solar energy in 2050”, clicked the first interesting link, and arrived at the paper Low-cost renewable electricity as the key driver of the global energy transition towards sustainability. Skipping to the Conclusions section, I find:
The solar PV industry is capable of providing all required capacities, as shown by Verlinden [98], since 70 TW of PV capacities can be ramped up by 2050, which is about 10% more than 63.38 TW found in this research.
This confident statement – “the solar PV industry is capable of providing all required capacities” – appears to be based on the fact that the projected capacity is 10% larger than the projected need. If either figure turns out to be off by 10% in the wrong direction, the conclusion would be invalidated. How likely is it that projections of energy usage in 2050 will be off by 10%? Well, it’s easy to find projections that disagree with one another by much more than that, so the probability that at least some of these projections are off is 100%!
Some Questions Have No Answers; Some Have Too Many
I’ve long wanted to have a clear idea of how much energy the world will use a few decades down the road. Energy usage is an important input to many other interesting questions about how the green transition will play out. However, I’d hesitated to dig in, because I suspected it would be difficult to find a useful answer.
That changed recently, when – as I’m excited to report – Eliana Schwartz and Janki Patel joined to help with research and analysis for this blog (see below). With folks less lazy than me more collective time and energy to invest, it was reasonable to dig into the published literature on 2050 energy usage. I’ll go through the findings in a subsequent post, but if you’d prefer to dive in yourself, you can find the data Eliana gathered (along with her extensive notes) here.
Across the various sources, there are figures ranging from 3262 to 8863 quadrillion BTU. Both are from credible sources: the International Energy Agency (IEA) and the U.S. Energy Information Administration (EIA), respectively.
The difference between these numbers is huge: a ratio of 2.7 : 1. That’s far more than the 10% margin in the “Low-cost renewable electricity” paper. But even this huge discrepancy is only one part of the problem.
Confidence Intervals
The EIA projection for 2050 energy usage is not really 886 quadrillion BTU. If you click through the footnote to find the original source, you’ll see that it says “886.3”.
This figure probably comes from some complex mathematical model that actually spit out a number like 886,319,455,081,534,769 BTU. Obviously the model is not accurate to that level of precision, so someone rounded it.
When presenting figures like this, you’re really supposed to round down to a level of precision that matches the accuracy of your model. 886.3 has one digit after the decimal point, implying that true figure should be between 886.2 and 886.4. Obviously, the model can’t possibly be that accurate.
In any case, rounding is a blunt tool for conveying precision. You can better convey the expected accuracy of a figure by using a “confidence interval”. As an example of the right way to do it, page 41 of the IPCC WG1 AR6 technical summary states:
For the decade 2011–2020, the increase in global surface temperature since 1850–1900 is assessed to be 1.09 [0.95 to 1.20] °C.
“1.09” is the estimate, and “0.95 to 1.20” is the confidence interval. A footnote explains that the IPCC’s modeling work projects a 90% probability that warming in 2011-2020 was between 0.95 and 1.20 degrees C. This gives us a clear idea of the level of uncertainty in this estimate – about 12% in either direction. (And this is an estimate, a very careful and thorough estimate, of what happened in the past! The uncertainty for a projection three decades into the future will be much larger.) Computing confidence intervals adds extra work, but when working with uncertain figures, it’s vitally important to communicate how uncertain the values are.
Unfortunately, it’s only possible to evaluate known sources of uncertainty. So even when a confidence interval is reported (and was computed correctly), you shouldn’t treat it as gospel. But if projection doesn’t include a confidence interval, you can’t tell how seriously to take it, and you have to suspect that the author doesn’t know, either.
Assumptions
A broad statistic like “global energy usage” depends on, well, everything: population growth, economic growth, relative importance of different economic sectors, efficiency improvements, shifting consumer preferences… everything. To project this into the future requires making assumptions about how every aspect of the global economy will evolve. Sometimes these assumptions are explicit, sometimes they’re implicit, but they all affect the result.
Unfortunately, sometimes these assumptions aren’t very good. Consider the US EIA figure cited above. It comes from this report, dated October 6, 2021. Poking around, on slide 11, I find:
Electric vehicle stock… represents 31% of total passenger vehicle stock by 2050
That is, the EIA is assuming that in the year 2050, 31% of passenger vehicles will be electric. Given current rates of EV adoption, this strikes me as implausibly low (consider that GM has pledged to stop selling gasoline vehicles entirely by 2035, and EVs have already reached 35% market share in China, the largest market)4. EVs use far less energy than internal combustion vehicles, so by under-estimating the shift to EVs, the EIA is over-estimating overall energy usage.
These particular publications don’t attempt to keep their assumptions secret. The EIA report explicitly assumes we continue “business as usual”, whereas the IEA projects a scenario in which we reach net zero emissions. But that still leaves the onus on us, the reader, to decide which projection (if either) to rely on. And it’s not always easy to tease out all of the relevant assumptions; even the authors may not be explicitly aware of every assumption they’re making.
All Models Are Wrong; Are Some Useful?
You may have seen the quote “all models are wrong, but some are useful”. I think the intended meaning is that some models, under some conditions, provide answers which are sufficiently accurate for some purposes. One way of thinking about this is that a model is useful if you apply it under the conditions and purposes where it provides reasonable answers. In other words, the onus is on the user, not the model.
If you open the EIA report, scroll down, read that global energy consumption in 2050 will be 886.3 quadrillion BTU, and use that to start making firm conclusions about precisely what the world will look like, you’re relying on the model for a purpose for which it is not useful.
On the other hand, if you compare the conclusions of the EIA report with other analyses, you might be able to get an idea of the range of plausible trajectories, and you might be able to learn something about how different assumptions affect projected energy usage. That is a purpose for which the model might be useful. I’ll explore this further in an upcoming post.
For today, the key takeaway is to remember that the future is uncertain. The Internet is littered with confident, quantitative projections of the future, awash in numbers that have no confidence intervals and don’t clearly state their assumptions. Don’t place much confidence in these numbers, and don’t assume they apply to your purpose. In Never Trust a Number, I explain how easy it is to be led astray by numbers in general. When a number attempts to describe the future, it’s even less trustworthy.
A corollary is that we tend to be overly confident in our ability to predict the future, and thus, overly hasty to dismiss certain technologies as unnecessary. The inherent uncertainty in our models is another reason to err on the side of developing multiple paths to decarbonizing every sector of the economy.
Postscript
The Climateer team has expanded! I’d been doing this in my spare time, but there are now three of us working in our spare time:
Eliana Schwartz is a Senior Product Manager at WattBuy, working on home electrification and decarbonization. She is passionate about contributing to research that helps demystify climate and energy data. Her LinkedIn is here.
Janki Patel works with firms on bringing sustainability into their business strategy. She recently graduated from UC Berkeley's MBA program, where she focused her time on trying to understand the intersection between equity and climate change. You can connect with her on LinkedIn.
I’m very excited to have Janki and Eliana on board. They’re able to engage in research and analysis that I wasn’t previously able to undertake, which will expand the scope of what we can explore here. We’ll be posting more frequently. And it’s wonderful to have their additional perspectives.
On reflection, over the years I’ve met a number of people who work in or around politics and policy and are probably about as close to “they” as you can get, but they were just people. I think “they” is kind of like “true AI”, every time you meet an example, you change the definition to exclude that example.
IEA. "Net Zero by 2050: A Roadmap for the Global Energy Sector Annex A." iea.org, May 2021. https://www.iea.org/data-and-statistics/data-product/net-zero-by-2050-scenario. Accessed 19 Nov. 2022.
U.S. Energy Information Administration. “International Energy Outlook 2021 Table A01. World total primary energy consumption by region (quadrillion Btu).” eia.gov, 6 Oct. 2021, https://www.eia.gov/outlooks/ieo/tables_side_xls.php. Accessed 19 Nov. 2022.
To be fair, the EIA figure is for “electric vehicle stock”, i.e. cars on the road. This lags behind the figure for new cars, because a gas car sold in previous years may still be on the road in 2050. But the average car lifetime, especially in growing markets like China, is not nearly long enough to make up the difference here.
Great article. It is so difficult to truly translate science into meaning and action. The devil is in so much of the detail and gets so easily lost. I don't know if there is an easy solution for this beyond expanding scientific literacy and talking in ore depth about what models really mean.