Our mission to make business better is fueled by readers like you. To enjoy unlimited access to our journalism, subscribe today.
Artificial intelligence algorithms that power some of the most cutting-edge applications in technology, such as writing coherent passages of text or generating images from descriptions, can require vast amounts of computing power to train. And that in turn requires large amounts of electricity, leading many to worry about the carbon footprint of these increasingly popular ultra-large A.I. Systems make them environmentally unsustainable.
New research from scientists at the University of California at Berkeley and Google, which deploys many of these large A.I. Systems, provides the most accurate estimates to date for the carbon footprint of some of these state-of-the-art systems.
For instance, GPT-3, a powerful language model created by the San Francisco-based A.I. Company OpenAI, produced the equivalent of 552 metric tons of carbon dioxide during its training, according to the study. That’s the same amount that would be produced by driving 120 passenger cars for a year. Google’s advanced chatbot Meena consumed 96 metric tons of carbon dioxide equivalent, or about the same as powering more than 17 homes for a year.
While those figures are frightening large, they are smaller than some previous estimates from researchers who did not have access to the same detailed information from inside Google and OpenAI. The research paper, which was posted to the non-peer reviewed research repository arxiv.org on Wednesday, also shows that the climate impact of A.I. Can be mitigated.
The researchers conclude that the carbon footprint of training these algorithms varies tremendously depending on the design of the algorithm, the type of computer hardware used to train it, and the nature of electricity generation where that training takes place.
Altering all three of these factors can reduce the carbon footprint of training one of these very large A.I. Algorithms by a factor of up to 1,000, the Google scientists found. Simply changing the datacenter used to train the algorithm from a place where power generation is coal intensive, like India, to one where the electrical grid runs on renewable power, such as Finland, can reduce it by a factor of between 10 and 100 times, the study concluded.
“It’s like that old joke about the three most important things in real estate: location, location, location,” David Patterson, the Google scientist who is lead author on the new paper, told Coins2Day. “Location made such a big difference.”
Patterson, who is also an emeritus professor at U.C. Berkeley, says that’s ultimately good news because most A.I. Algorithms are trained “in the cloud,” with the actual processing taking place in data centers that can be hundreds or even thousands of miles away from where the person creating the system is sitting. “In cloud computing, location is the easiest thing to change,” he says.
But if environmental sustainability becomes a major consideration in training A.I. Systems it is also to further cement the market position of the largest cloud service providers. That’s because companies such as Microsoft, Google, IBM and Amazon Web Services have dozens of data centers in many different places, including those in areas with colder average temperatures, reducing the cost of cooling all those server racks, and greener energy.
The environmental impact of ultra-large A.I. Systems designed for processing language was one of the criticisms of such algorithms raised by a group of A.I. Ethics specialists inside Google that played a role in the events that precipitated the ouster of Timnit Gebru and the subsequent firing of Margaret Mitchell, the two co-heads of the A.I. Ethics research team.
Jeff Dean, a senior executive vice president at Google who heads the company’s research division and has been faulted by Gebru and her supporters for his role in forcing her out, is one of the nine authors credited on the new research paper on reducing the carbon footprint of these A.I. Systems. One of his alleged criticisms of Gebru’s earlier paper is that it did not discuss ways to mitigate the negative ethical impacts of large language models.
Besides shifting to a location with a greener electricity grid, another way to improve the energy consumption of these models is to use computer chips that are specifically-designed for neural networks, a kind of machine learning software loosely modeled on the human brain that is responsible for most recent advances in A.I. Today, the majority of A.I. Workloads are trained on computer chips that were originally designed for rendering the graphics in video games. But increasingly new kinds of computer chips designed just for neural networks are being installed in datacenters run by large cloud-computing companies such as Google, Microsoft, and Amazon Web Services.
Changing from graphics processing chips to these new neural network-specific chips can cut the energy needed to train an ultra-large algorithm by a factor of five, and it can be cut in half again by shifting from the earliest generation of these new A.I. Chips to the latest versions of them, the researchers found.
An even bigger savings—a factor of 10—can be found by redesigning the neural network algorithms themselves so that they are what computer scientists call “sparse.” That means that most of the artificial neurons in the network connect to relatively few other neurons, and therefore need a smaller number of these neurons to update how they weight data for each new example the algorithm encounters during training.
Maud Texier, another Google researcher who worked on the study, says she hopes the paper helps drive the entire industry towards standardized benchmarks for measuring the energy consumption and carbon footprint of A.I. Algorithms.
But she emphasized that this is not easy. To get an accurate estimate for the carbon footprint, it is important to know not just how green the electric grid in any particular location is in general, but exactly what the mix of renewable energy and fossil fuel-based electricity was during the specific hours when the A.I. Algorithm was being trained. Obtaining this information from cloud service providers has been difficult, she says, although the large cloud service companies are starting to provide more detailed information on carbon dioxide emissions to customers.
