Large language models often are called upon to gather news. In this task, researchers found, their ability to find relevant reports is the weakest link.
What’s new: Mirac Suzgun and colleagues at Stanford University and the cloud platform Together AI tested the ability of six popular LLMs outfitted with web-search tools to answer questions about the daily news. The LLMs generally answered accurately, especially when the questions themselves contained accurate information and were posed in English. When they answered inaccurately, usually it was because they retrieved unhelpful articles.
Key insight: An LLM’s ability to answer factual questions that weren’t addressed in its training data depends on at least three steps: (i) receiving a well formed question, (ii) retrieving a relevant document, and (iii) extracting facts from it. Compromising any of these steps will degrade the output. For instance, a question that includes incorrect facts — say, a mistaken name — will make it difficult for the LLM to answer correctly. Similarly, retrieving a document that doesn’t include necessary facts is more likely to yield an incorrect answer. And the LLM won’t be able to extract facts from a retrieved document if it wasn’t well trained in the document’s language. Building a high-performance news-reporting system requires mastering all three steps.
How it works: The authors performed tests daily between February 9 and February 22, 2026. They posed questions based on BBC News reports in six languages (Arabic, English, French, Hindi, Russian, and Turkish). They tested Google’s Gemini 3 Flash and Pro, xAI’s Grok 4, Anthropic’s Claude 4.5 Sonnet, and OpenAI’s GPT‑5 and GPT‑4o mini, all with web-search tools.
- Each day, Gemini 3 Flash produced 25 multiple-choice questions per language (150 total) based on current news. Each question began with temporal context like “Today is February 10, 2026…” and offered five potential answers. The correct answer was a verifiable detail such as a number, location, or quotation.
- The authors tested the LLMs on (a) the five-option, multiple-choice questions, (b) alterations that introduced a single incorrect premise (such as an incorrect actor, timeline, or detail) and added the potential answer “insufficient information to answer,” and (c) the original questions without potential answers.
- They used majority votes by Claude Opus 4.7, GPT-5.4, and Gemini 3 Pro to classify the causes of incorrect outputs into eight categories, such as failures of retrieval, retrieval of a relevant source that contained different details than the one used to compose the question, inadequate comprehension, and mistaken timeline.
Results: The models generally performed with high accuracy on well formed questions in English. However, (i) they underperformed in other languages, especially Hindi, (ii) most errors stemmed from retrieval rather than intelligence, and (iii) they underperformed when questions contained false premises.
- Answering the original, unaltered multiple-choice questions, the top four models exceeded 90 percent accuracy: Gemini 3 Flash (95.6 percent), Grok 4 (95.0 percent), Gemini 3 Pro (93.7 percent), and Claude 4.5 Sonnet (90.4 percent). GPT-5 achieved 85 percent and GPT-4o mini achieved 69 percent. (Given free-response questions, the models maintained their relative rank, but their accuracy fell by 11 to 22 percentage points.)
- Answering the unaltered multiple-choice questions, every model showed its worst performance when questions were in Hindi (average 79.3 percent). The models frequently cited English-language sources (such as English Wikipedia) even when questions were posed in other languages (especially Hindi).
- The most common cause of errors were retrieval failures (38.8 percent). The second-most common cause was retrieval of a topically relevant source that contained “smart but wrong” details that did not correctly answer the specific question (32.7 percent).
- Given altered questions that contained false premises, Grok 4 (70 percent accuracy) outperformed the other models by at least 15 percent. GPT‑5 (19 percent accuracy) performed barely above random chance for six potential answers.
Why it matters: LLMs outfitted with web search are well suited to finding factual information, but their performance depends on receiving factual input, finding relevant documents, and extracting facts properly. Since most errors arise in retrieval, improving retrieval may yield a larger performance increase than scaling model parameters (at least when it comes to responding to queries that involve timely information). The authors point to three approaches to improving retrieval: (i) better indexing coverage (which documents or web pages are included in the search), (ii) better ranking of sources (the order in which retrieved documents are presented to the model), and (iii) better handling of queries in languages other than English.
We’re thinking: Web search engines built for agentic rather than human use have become an exciting and rapidly growing category. It looks like there's still ample room to build better web search for agents!