How AI could fuel global warming
New large models are energy intensive. How much CO2 is needed for their training?
Data and cloud are not virtual technology. They need costly infrastructure and electricity. The same for training an AI model. Researchers forecast that in the future their emissions could be much more than expected. A short overview
Who does not make us sleep the night?
A few weeks ago the UK break the temperature record, for the first time the temperature rose over 40 °C. The summer nights are warm and humid and it is hard to sleep on similar days. This is the fifth warmest year so far but the other four are in the same decade (chance are that we might remember this summer as one of the coolest). It is undoubtful that year after year we are facing higher temperatures, this tendency is also defined as part of global warming. Global warming is not meaning only warmer summers but also an increase in extreme events (such as hurricanes, tornadoes, droughts, bushfires, and so on).
Who is the culprit? A small molecule consisting of one carbon atom and two oxygen atoms, better known as carbon dioxide or co2. There are other greenhouse gases, but carbon dioxide remains by far the most important one. The graph below shows how carbon dioxide has grown exponentially over the past two centuries.
In a nutshell, the more carbon dioxide increases, the greater the retained part of the infrared component of solar radiation striking the Earth. This extra energy warms the Earth. Who produces carbon dioxide?
Human and all its activities. Leaving aside the fact that by breathing in we exhale carbon dioxide, the consumption of energy causes the production of carbon dioxide. Car and air travel, businesses, livestock farms and so on all contribute to the release of co2 into the atmosphere.
There is something that produces considerable carbon dioxide production as well, but it is little known. Data.
The cloud is made of carbon dioxide
Data appear to be virtual objects, almost metaphysical entities, but they require to be stored, processed, and transmitted, and this requires infrastructure. For example, when you save some files in the cloud, they have to traverse thousands of kilometers of optic cables before arriving at a data center. There are thousands of data centers around the globe, but essentially they are buildings filled with a huge number of hard disks. These hard disks are continuously in activity and they are producing heat.
"The more storage you have, the more stuff you accumulate." – Alexis Stewart
The estimated cost of a GigaByte stored in the cloud is 7kWh (no more than a hundred high-resolution photos). We produce 2.5 quintillion bytes per day (2.5 followed by 18 zeros). Without doing the math, it is a lot to store, and a lot of carbon dioxide is produced in the process. In fact, it is predicted that the communication industry will soon produce more than the automotive, aviation, and energy sectors combined.
In fact, there are today around 8 million data centers (in 2012 there were 800.000), showing how much at which pace we are increasing the production and the storage of data. Some models predict that by 2030 more than 10 % of the global electricity supply will be dedicated to data centers. These predictions are only taking into account the energy consumption required by storing the data, but data travel on the internet which is also consuming energy.
There are many researchers that are looking at how we can reduce the environmental impact of data storage. However, data are not only stored. In fact, when you have so much available data you can use it to train a very large model. Then is arising the question: how much artificial intelligence is consuming?
(artificial) Intelligence devours energy to sustain itself
The human brain is one of the most sophisticated things that has evolved on the face of the earth. Its complexity allows us to range between abstract reasoning, science, and art. If having such a developed brain is an evolutionary advantage, why do most species have far fewer neurons? One answer is because it costs a lot; the human brain alone consumes 20–30% of the body’s entire energy. Not cheap for a single organ.
We can suspect that artificial intelligence is doing the same: consuming a lot of energy to do all the calculations. The question is becoming more and more relevant since now almost all companies are investing in machine learning. Then, how much AI is consuming?
In one of the first works on the topic, Emma Strubell calculated that a transformer model trained using a Neural Architecture search will be comparable to the carbon dioxide emission of five cars during their lifetime (the 2019 paper link is here). In a successive article, Patterson expanded the analysis on different popular model architectures (T5, BERT, GPT-3) comparing the cost of their training and their carbon footprint.
In the article, Patterson showed how many factors have to be considered to calculate the energy cost of a model (the accelerator, the optimization method, dimension of the training set, number of hyperparameters, and so on). In the article, they compared the cost of training to the emission of a jet, which is worrying considering that the newer model has much more hyperparameters and the training datasets are also increasing in size.
In the article, they highlight also other interesting points: geographic location and infrastructure matter (using the cloud or not). Now, there are many services offering the possibility to train models on the cloud (Azure, AWS, and so on). Indeed, for a small company is easier to train a model on the cloud than to buy an expensive stack of GPUs and set the in-house infrastructure. Since this is a more and more popular choice, different researchers studied the carbon intensity of artificial intelligence in cloud instances.
It turned out that location still matters, even when using the cloud. In their works, they monitored the training of 11 algorithms (ranging from language models to vision algorithms) on Microsoft Azure and they monitored the electricity grid power at different locations. The differences were substantial, showing that performing the training at the US-based data center was generating double the emission of the same training performed in Norway.
"the most efficient regions produced about a third of the emissions of the least efficient" – Jesse Dodge, one of the co-author said to Nature
In addition, when you are training your model is also changing your carbon footprint. For instance, in training the model during the day in Washington the energy is coming from the gas-fired station while during the night the energy is produced by hydroelectric power.
"the team carried out only 13% of the transformer’s training process; training it fully would produce emissions "on the order of magnitude of burning an entire railcar full of coal", says Dodge". (source: Nature)
Conclusions and perspectives
The cloud would be the preferred choice to train the AI models for many small/medium companies. However, the cloud and AI models are increasing their carbon footprint at a moment when the global warming effect is becoming more and more marked. Thus we need to think about how we can reduce the impact of both technologies.
Companies need to invest in reducing the impact of energy. As suggested by researchers, one first step is the possibility to select and use the data center with the lowest carbon footprint when training AI on the cloud. Moreover, the training should be flexible and scheduled when there is lower demand for energy or the data center is powered by green energy.
"The less we do to address climate change now, the more regulation we will have in the future." – Bill Nye
Since the AI market size is predicted to expand at a compound annual growth rate (CAGR) of 38.1% from 2022 to 2030, we should address its energy consumption as soon as we can. The good news is that companies and researchers are aware of the problem and working on the solution. Moreover, there is also an institutional effort that is taking into account the use of green energy for training (such as the BLOOM model). Finally, AI models could also be beneficial to optimize energy consumption and contain carbon dioxide emissions.
Additional resources
- If you are interested in tracking or predicting energy consumption and/or carbon footprint by training a deep learning model you can try: carbontracker ([article](https://arxiv.org/abs/2103.16435), repository), codecarbon (website, [GitHub repository](https://github.com/poloclub/EnergyVis), documentation), ML CO2 Impact (website), or EnergyVis (article, GitHub repository)
if you have found it interesting:
You can look for my other articles, you can also subscribe to get notified when I publish articles, and you can also connect or reach me on LinkedIn. Thanks for your support!
Here is the link to my Github repository, where I am planning to collect code and many resources related to machine learning, artificial intelligence, and more.
GitHub – SalvatoreRa/tutorial: Tutorials on machine learning, artificial intelligence, data science…
Share This Article
Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.
Write for TDS
