Year in Review

Individual deep learning models proved their mettle in hundreds of tasks. The scope of multi-task models expanded dramatically in the past year.

Researchers pushed the boundaries of language models to address persistent problems of trustworthiness, bias, and updatability.

Work on vision transformers exploded in 2022. Researchers published well over 17,000 ViT papers during the year. A major theme: combining self-attention and convolution.

Behind schedule on a software project? There’s an app for that. Language models fine-tuned on computer code proved capable of generating software routines similar to the work of experienced developers — though the results can be hit-or-miss.

Pictures produced by AI went viral, stirred controversies, and drove investments. A new generation of text-to-image generators inspired a flood of experimentation, transforming text descriptions into mesmerizing artworks and photorealistic fantasies.

The transformer architecture extended its reach to a variety of new domains.What happened: Originally developed for natural language processing, transformers are becoming the Swiss Army Knife of deep learning.

Musicians and filmmakers adopted AI as a standard part of the audio-production toolbox. What happened: Professional media makers embraced neural networks that generate new sounds and modify old ones. Voice actors bristled.

While models like GPT-3 and EfficientNet, which work on text and images respectively, are responsible for some of deep learning’s highest-profile successes, approaches that find relationships between text and images made impressive

In 2021, transformers were harnessed to discover drugs, recognize speech, and paint pictures — and much more.

Legislators worldwide wrote new laws — some proposed, some enacted — to rein in societal impacts of automation.What happened: Authorities at all levels ratcheted up regulatory pressure as

The trend toward ever-larger models crossed the threshold from immense to ginormous. Google kicked off 2021 with Switch Transformer, the first published work to exceed a trillion parameters, weighing in at 1.6 trillion.

Neural networks for natural language processing got bigger, more prolific, and more fun to play with. Language models, which already had grown to gargantuan size, continued to swell, yielding chatbots that mimic AI luminaries and have very strange ideas about horses.

Institutional hurdles to AI for medicine began to fall, setting the stage for widespread clinical use of deep learning in medical devices and treatments. DeepMind’s AlphaFold model determined the three-dimensional shape of a protein in just hours.

The worldwide pandemic and a contentious U.S. election whipped up a storm of automated disinformation, and some big AI companies reaped the whirlwind.

Some of deep learning’s bedrock datasets came under scrutiny as researchers combed them for built-in biases. Researchers found that popular datasets impart biases against socially marginalized groups to trained models due to the ways the datasets were compiled, labeled, and used.