Data showing how new pretrained language models might learn facts like weight and cost
BERT

The Measure of a Muppet

The latest pretrained language models have shown a remarkable ability to learn facts. A new study drills down on issues of scale, showing that such models might learn the approximate weight of a dog or cost of an apple, at least to the right order of magnitude.
2 min read
Examples of contrastive learning
BERT

Learning From Words and Pictures

It’s expensive to pay doctors to label medical images, and the relative scarcity of high-quality training examples can make it hard for neural networks to learn features that make for accurate diagnoses.
2 min read
Data related to Nvidia's Pay Attention When Required (Par) approach
BERT

Selective Attention

Large transformer networks work wonders with natural language, but they require enormous amounts of computation. New research slashes processor cycles without compromising performance.
1 min read
Bert (muppet) and information related to BERT (transformer-based machine learning technique)
BERT

Do Muppets Have Common Sense?

Two years after it pointed a new direction for language models, Bert still hovers near the top of several natural language processing leaderboards. A new study considers whether Bert simply excels at tracking word order or or learns something closer to common sense.
2 min read
Graphs and data related to language models and image processing
BERT

Transforming Pixels

Language models like Bert, Ernie, and Elmo have achieved spectacular results based on clever pre-training approaches. New research applies some of those Sesame Street lessons into image processing.
2 min read
Examples and explanation of an automatic headline generation
BERT

AI Makes Headlines

Which headline was written by a computer? A: FIFA to Decide on 2022 World Cup in March B: Decision in March on 48-team 2022 World Cup, Says Infantino
2 min read
Illustration of a broken heart with a smirk in the middle
BERT

Outing Hidden Hatred

Facebook uses automated systems to block hate speech, but hateful posts can slip through when seemingly benign words and pictures combine to create a nasty message. The social network is tackling this problem by enhancing AI’s ability to recognize context.
2 min read
Illustration of two translators on a scale
BERT

Choosing Words Carefully

The words “big” and “large” have similar meanings, but they aren’t always interchangeable: You wouldn’t refer to an older, male sibling as your “large brother” (unless you meant to be cheeky). Choosing among words with similar meanings is critical in language tasks like translation.
2 min read
Single Headed Attention RNN (SHA-RNN)
BERT

Language Modeling on One GPU

The latest large, pretrained language models rely on trendy layers based on transformer networks. New research shows that these newfangled layers may not be necessary.
2 min read
Yann LeCun
BERT

Yann LeCun: Learning From Observation

How is it that many people learn to drive a car fairly safely in 20 hours of practice, while current imitation learning algorithms take hundreds of thousands of hours, and reinforcement learning algorithms take millions of hours? Clearly we’re missing something big.
2 min read
Illustration of a fireplace with "Happy holidays" cards in English, Spanish and French
BERT

Language Models Get Literate

Earlier language models powered by Word2Vec and GloVe embeddings yielded confused chatbots, grammar tools with middle-school reading comprehension, and not-half-bad translations. The latest generation is so good, some people consider it dangerous.
2 min read
Sesame Street characters together
BERT

Inside AI’s Muppet Empire

As language models show increasing power, a parallel trend has received less notice: The vogue for naming models after characters in the children’s TV show Sesame Street.
1 min read
Automatically generated text summary
BERT

Keeping the Facts Straight

Automatically generated text summaries are becoming common in search engines and news websites. But existing summarizers often mix up facts. For instance, a victim’s name might get switched for the perpetrator’s.
2 min read
Pipeline for identifying sentences containing evidence of SDIs and SSIs
BERT

Hidden Findings Revealed

Drugs undergo rigorous experimentation and clinical trials to gain regulatory approval, while dietary supplements get less scrutiny. Even when a drug study reveals an interaction with supplements, the discovery tends to receive little attention.
2 min read
GPT-2 text generator
BERT

Putting Text Generators on a Leash

Despite dramatic recent progress, natural language generation remains an iffy proposition. Even users of the muscular GPT-2 text generator have to press the button a number of times to get sensible output. But researchers are figuring out how to exert greater control over generated text.
2 min read

Subscribe to The Batch

Stay updated with weekly AI News and Insights delivered to your inbox