A team in the United Arab Emirates, a seven-state federation on the Arabian Peninsula, built the latest top-performing open source large language model.
What’s new: The UAE’s Falcon edged out Meta’s LLaMA in the Hugging Face Open LLM Leaderboard to take the top spot. It’s available via the Apache 2.0 license, which is free for commercial applications. You can try it here.
How it works: Developed by Abu Dhabi’s Technology Innovation Institute (TII), Falcon is a pretrained model based on transformers. A paper is forthcoming.
- The 40-billion parameter model was pretrained to predict the next token on 1 trillion tokens of text. 820 billion tokens came from RefinedWeb, a curated subset of Common Crawl. The remainder came from books, code, academic papers, technical documents, and conversations on sites including Reddit and StackOverflow.
- The architecture is similar to OpenAI’s GPT-3 with a few differences. For instance, it uses the FlashAttention algorithm with multiquery attention, both of which cut memory requirements at inference.
- Four versions are available: the general-purpose Falcon-40B the chat-ready Falcon-40B-Instruct, the more compact Falcon-7B, and Falcon-7B-Instruct.
- Initially, TII required users whose commercial applications of Falcon earn more than $1 million to pay a 10 percent “authorization fee.” However, the developer quickly reverted to a more permissive license.
Behind the news: Open source licenses, particularly those that are free for commercial use, are enabling independent teams to build systems that are competitive with those produced by big tech companies. A recently leaked Google memo went so far as to call open source development a threat to the company’s business.
Why it matters: Amid a surge in open source language models, Falcon offers higher performance (on the four benchmarks tracked by Hugging Face) and lower training cost relative to its chief rival, LLaMA. The fact that it was built by a team in Abu Dhabi highlights the fact that AI talent is everywhere and elite skills are spreading to all parts of the globe.
We’re thinking: AI development is a global enterprise. It gives us hope that people around the world can come together to meet other common challenges.