ZeRO
Bigger is Better: A research summary of Microsoft's Turing-NLG language model.
Natural language processing lately has come to resemble an arms race, as the big AI companies build models that encompass ever larger numbers of parameters. Microsoft recently held the record — but not for long.