Annual index finds AI is ‘industrializing’ but needs better metrics and testing

China has overtaken the United States in total number of AI research citations, fewer AI startups are receiving funding, and Congress is talking about AI more than ever. Those are three major trends highlighted in 2021 AI Index, an annual report released today by Stanford University. Now in its fourth year, the AI Index attempts to document advances in artificial intelligence as well as the technology’s impact on education, startups, and government policy. The report details progress in performance of major sub domains of AI like deep learning, image recognition, object detection, and progress in areas like protein folding.

The AI Index is compiled by the Stanford Institute for Human-Centered Artificial Intelligence in and a 11-member steering committee with contributors from Harvard University, OECD, the Partnership on AI, and SRI International. The AI Index utilizes datasets from a range of sources, like AI research data from arXiv, funding data from Crunchbase, and surveys of groups like Black in AI and Queer in AI. A major trend also identified in the report, Clark said, is the industrialization of AI.

“I think the story to me is that AI is industrializing, and we don’t quite know how to assess the industrialization of it holistically, because we sort of lack a lot of the data that you’d expect would exist, and I think that’s because AI has just gone from doesn’t work to works well enough for commercial deployment way more quickly than you might expect, and that means that we’re in a kind of everyone’s racing including the research community to keep up with the pace of commercial deployments,” he said.

Other major takeaways include:

• Brazil, India, Canada, Singapore, and South Africa saw the highest levels of AI hiring from 2016 to 2020, according to data provided by LinkedIn
• Total global investment like private investment and mergers and acquisitions grew 40% in 2020, but for the third year in a row, AI startup funding is going to fewer startups
• In 2019, about two of three graduating North American PhDs in AI went into industry, up from 44 percent in 2010
• The majority of AI PhD graduates come from outside the United States, and four of five stay in the United States after graduating
• A news analysis found AI ethics stories among the most popular stories related to AI of 2020 like Google firing Timnit Gebru, and ethics initiatives introduced by the European Commission, United Nations, and Vatican
• Attendance at major AI research conferences doubled in 2020 as most chose to hold virtual gatherings
• Women made up 18% of AI PhD graduates
• China overtook the U.S in total paper citations, but the U.S. continued a two decade lead in citations at AI research conferences
• TensorFlow is still the most popular AI software library followed by Keras and PyTorch
• AI-related papers on arXiv has grown from roughly 5,500 in 2015 to nearly 35,000 in 2020
• A Queer in AI 2020 member survey found that roughly half of respondents experienced harassment or discrimination and have encountered issues related to inclusiveness
• Academic researchers lead in total papers published worldwide, but in the U.S. corporate research ranks second, whereas government research ranks second in Europe and China

In the portion of the report dedicated to progress toward technical challenges, highlights include advances in effective chemical and molecular synthesis.

The AI Index shows progress in how AI systems can impact surveillance technology like performance of objection made for deployment like YOLO. Considerable progress has been made toward VoxCeleb, which measures the ability to identify the voice of an individual among 1,000 people.

“This metric is telling us that AI systems have gone from having like an eight percent equal error rate to a less than 0.5% one, which tells you that this capability is going to be being deployed quietly across the world,” Clark said.

A panel of experts on technical progress called AlphaFold’s ability to predict how proteins fold and GPT-3 two of the most talked about AI systems of 2020. Though the AI Index acknowledges few- and zero-shot learning gains made by GPT-3, it also cites a paper by authors including former Ethical AI team Cole and Timnit Gebru critical of large language models and their ability to perpetuate bias, as well as a paper published last month by OpenAI and Stanford warning about the need to address the societal impact of large language models. In an interview with VentureBeat in 2019, AI Index founding director Yoav Shoham was critical of judging language models based on performance on narrow tasks.

Both of these research papers have been extensively reported on by VentureBeat. Other reports previously covered by VentureBeat and cited in the report include McKinsey’s State of AI report that found little progress among business leaders to address risk associated with deploying AI. Another is about the de-democratization of AI in the age of deep learning that coauthors say can perpetuate inequality.

Among major conclusions about the current state of AI drawn in the report: Computer vision, ethics, and NLP need more benchmarks and testing to move the industry forward. As demonstrated by benchmarks like GLUE and SuperGLUE, Clark said, “we’re running out of tests as fast as we can build them.” Creating benchmarks and testing is an opportunity to make metrics reflective of people’s values or that make progress toward a grand challenges like deforestration.

“I think one of the ways to get holistic accountability in a space is to have like the same test that you run everything against, or the same set of tests, and until we have that, it’s going to be really fuzzy to talk about biases and other ethical issues in these systems, which I think would just hold us back as a community and also make it easier for people who want to pretend these issues don’t exist to continue to pretend they don’t exist or not mention them,” he said.

In previous years, the AI Index expanded to include tools like an arXiv monitor for searching prerprint papers. The AI Index’s Global Vibrancy Tool, which serves up comparisons between national AI initiatives, now works for 26 countries across 23 categories.

Perhaps as interesting as what’s included in the report is what’s missing. This year, the report removed data related to progress on self-driving cars, while Clark said the report does not include information about fully autonomous weaponry due to a lack of data.