Quantified results: how reliable are they?¶
Authors and date
- Submitted on: July 26th 2021
- Françoise Berthoud, research engineer, GRICAD, CNRS
This sheet, automatically translated by the Deepl tool, has not been proofread by its author
« Give us figures! » is a frequent request from journalists looking to show sensationalism, it is a frequent request from the public looking to compare and it is a frequent request as well from scientists looking to understand, to measure, to evaluate and to design.
Yet, figures do not tell the whole story and sometimes they do not represent what we imagined: a minimum of "expertise" is required to understand and to interpret figures. There is also a question about the degree of their reliability and of trust that we grant them.
When it comes to the environmental impact of digital technology, figures are everywhere: from the greenhouse gas emissions (GHG) generated by the manufacture of a smartphone, to the percentage of the world's electricity consumed by digital technology, to the number of years of indium reserves. There is also the percentage of greenhouse gases saved thanks to digital technology in the transportation sector, the amount of fresh water used to manufacture a tablet...
Are these figures reliable?¶
We must distinguish several aspects:
Methodology issues: a figure will always depend on the perimeter taken into consideration (what types of equipment in particular), the reference years, the very many assumptions made, extrapolations, the model studied, the input data taken into account for the calculations, anything that was not taken into account, the choices made for the design, etc. In other words, the better the quality of the input data and the more robust the methodology, the more reliable a figure will be.
Uncertainty issues: the uncertainties related to a result are of different natures. There are, for example, uncertainties related to the approximations made in measurements, estimations, etc. And then there are the uncertainties related to the methods used or to the model used, for example if the model forgets to integrate an important piece of data or when an average is taken. In all cases, the uncertainties accumulate in the calculations.
Finally, when it is calculated, this uncertainty is rarely submitted to the public. If we take a simple example of statistical data synthesized in a single figure (their average), the deviations from the average will be hidden1.
In summary: the reliability of a figure is all the stronger when the uncertainties are small and when they are published.
Transparency and openness of data: finally, the confidence that can be granted to the figures is all the stronger if the quality of the data, the methodologies and the uncertainties are visible by everyone, updated by cross-review systems and in any case with full transparency.
In other words, in this field, a figure published alone is not enough. It must have all elements that allow a reader to estimate its reliability and the conditions of its calculation. The use of the figure should differ greatly depending on these elements.
We will show this through three examples: the first one taken from the scientific literature, the second one from the software world and the third one from an independent study. These examples show weaknesses commonly observed in the large world of numerical data generation.
Example 1: An article on the worldwide electricity consumption of communication networks¶
Article: "Worldwide electricity consumption of communication networks". Sofie Lambert, Ward Van Heddeghem, Willem Vereecken, Bart Lannoo, Didier Colle, and Mario Pickavetnetworks. Published in: Optics Express Vol. 20, Issue 26, pp. B513-B524 (2012) 2
The article specifies the methodology used and analyzes the reliability of the results, clearly indicating the sources of uncertainties. This scientific article, published in a scientific journal, has been reviewed by two reviewers, usually anonymously. The scientific article must allow its reader to find the sources of all the data and the data itself, and also to reproduce the result or the experiment (in theory because in practice it is generally very complicated). This is called reproducible research3.
Here the article is in open access, i.e. freely accessible to everyone.
So, obviously, reading or understanding is less simple because everything must be (in principle) stated, argued, justified and verifiable. In this case, we have access to the hypotheses, to the data sources, to the design.
The result may be open to criticism, but when this is the case, a new article will probably be published in the near future to correct and complete the information, and this is also what makes science progress.
Example 2: a very practical application Carbonalyser¶
Carbonalyser: a browser extension (or add-on) that allows a user to see the power consumption and greenhouse gas emissions associated with his internet browsing4.
In reality, no network operator, no service provider, no software is able today to measure the precise energy consumption of real-time data transport and its specific processing. The calculation is therefore built from global, averaged data and assumptions that are clearly explained in the methodology. The source code and all the data used for the calculation are freely available.
The authors specify that « Carbonalyser is not an evaluation or audit tool. It is a support for individual awareness, which helps to visualize a certain aspect of our online uses, through comparisons and indications on power consumption and associated emissions. These results are not measures of the true impact of online activities, but indicators of average magnitudes obtained by extrapolating a macroscopic model built on worldwide statistical averages. Carbonalyser should only be used to inform individual reflections on one's own usage or to inspire the development of tools to accurately assess the impacts of online activities. »
In this example, the results are estimated from an available and transparent model, even if the model is imperfect.
To mesure the amount of GHGs emitted by the digital sector, the authors specify the scope of the digital sector (types of equipment taken into account), and then, based on assumptions about the duration of use of each type and sales volumes, calculate (on the basis of partially non-public data such as the average GHG emission values for the manufacture and transport of this equipment or its average annual consumption) the digital sector's share of global greenhouse gas emissions.
There is no transparency on the basic data, the assumptions are not explained and the sources (especially the input data) are not available in part. Finally, the uncertainties are not quantified. Nevertheless, this study has the merit of being one of the few to provide key figures and has been widely communicated and used.
From the preliminary remarks in this document and the examples in the field of the environmental impacts of digital technology, we can conclude that a figure is the result of an estimation made under specific conditions, with a specific methodology, uncertainties and sources; the figure represents only what it represents, with all the relevant limits. It is therefore important to be aware of all these limitations and sometimes the conclusion is the unreliability of the figure in question; for example, estimates of the potentially positive impacts of digital technology are based on so many extrapolations and overly optimistic assumptions that they do not really make sense6,7.
Often in this field numbers must be taken as orders of magnitude. Also the decimals do not make sense in general. For example, the digital sector as a whole (excluding connected objects) generates about 4% of GHGs: it may be 2% or 6% but not 50%, nor 0.4%.
What is an order of magnitude?
The Wikipedia definition is: an order of magnitude is a number that represents in a simplified but approximate way the measurement of a physical quantity.
An order of magnitude is easier to remember than a precise value and is sufficient for many purposes.
Scientifically, an order of magnitude corresponds to a range of values. This is, commonly, from one tenth to ten times the size. Thus, an object whose length is about 1 m (a table) is larger than an object whose length is about 1 dm (a pencil) and smaller than an object whose length is about 10 m (a truck).
Knowing and having in mind these orders of magnitude (and the conditions / limits) can be sufficient to make political choices or to take relevant legislative resolutions. On the other hand, to be able to exercise one's critical mind on the figures communicated with great publicity is necessary to avoid making counterproductive choices!
Finally, as you will have understood, in the vast majority of cases, it is not possible to compare two figures from different studies because of the very likely differences in assumptions, scope and input data. On the other hand, using the same methodology, conditions, etc. over time to compare progress on GHG emissions from a given facility, for example, is very relevant.
To remember: quality of input data, robustness and transparency of calculation methodologies, openness and accessibility, clarification of all assumptions, and communication precautions are the basic rules for any communication of figures in order to make progress together on the knowledge of our environmental impacts.
Sofie Lambert, Ward Van Heddeghem, Willem Vereecken, Bart Lannoo, Didier Colle, and Mario Pickavet, "Worldwide electricity consumption of communication networks," Opt. Express 20, B513-B524 (2012). Available at osapublishing ↩
Video suggestion of @ScienceEtonnante on the peer review process: Comment fact-checker une étude scientifique ?, ScienceEtonnante, David Louapre. Avril 2019. [accessed 29/03/2022] ↩