GPT-3
Weak Foundations Make Weak Models: Foundation AI Models Pass Flaws to Fine-Tuned Variants
A new study examines a major strain of recent research: huge models pretrained on immense quantities of uncurated, unlabeled data and then fine-tuned on a smaller, curated corpus.