On November 2, the New York Times filed a lawsuit against Brave, the company behind the Brave browser and Brave Search engine. The newspaper claims that Brave copied more than 30 billion web pages ‘without identifying itself or offering website owners a mechanism to block its crawler, ‘ and then resold this content to artificial intelligence companies. Brave has not yet responded, and these claims remain allegations for now.
- The New York Times accuses Brave of copying over 30 billion web pages and reselling them to AI companies.
- Brave sells access to its search index through its API, which several AI firms use.
- Brave’s crawler has no unique name, so sites cannot block it while still allowing Google.
- A site owner found his pages in Brave’s index even though he blocks thirteen AI crawlers.
Brave does indeed sell access to its search index-the database of pages it has collected-through a product called Brave Search API. When an AI assistant needs recent information, it can pay to query this index rather than crawling the web itself. According to Brave’s ‘About’ page, this interface is ‘used by some of the biggest names in AI, including Cohere, Mistral, Perplexity, and You.com.’ The page dedicated to AI states that the index contains ‘more than 30 billion pages, with millions of new pages indexed daily.’
The Times’ complaint names five AI companies allegedly supplied by Brave: Anthropic, Perplexity, Mistral, xAI, and Cohere. None of these are defendants in the lawsuit. The case of Anthropic is the most delicate, as the evidence is older: the newspaper points to a 2023 Brave article about training Claude 2. In March 2025, Anthropic added Brave Search to its list of subcontractors, as reported by TechCrunch. That same month, developer Simon Willison asked Claude for information about pelicans: the ten cited results exactly matched those from an identical search on Brave. Whether Claude still uses Brave today could not be confirmed.
A Nameless Crawler
Sites indicate which web crawlers can read them via a small file called robots.txt. This mechanism works because each crawler has a name. OpenAI and Anthropic publish the names of their crawlers, allowing sites to permit one and deny the other.
Brave’s help page states that its crawler ‘does not announce a differentiated user agent.’ In other words, it is not identifiable by a unique name. Brave justifies this choice by saying some sites allow only Google in: a crawler signed as Brave would therefore be blocked. Brave claims to follow Googlebot’s rules, meaning it only reads what Google is allowed to read. The result, according to the page, is that a site cannot allow Google while blocking Brave. To be removed from the index, a site must use the noindex tag, which also asks Google to remove the page. The Times describes this situation as ‘refusing to disclose the name of its web crawler.’
The Case of an Indexed Site
The author of the source article, who runs the site nohacks.co, verified this. A site:nohacks.co search performed on November 3 displayed sixteen of his pages on the first results page, including thirteen articles. The result even included a page description that had been replaced that same day, indicating a stored copy rather than a live consultation. His robots.txt lists thirteen AI crawlers, but none correspond to Brave.
The complaint also addresses ‘LLM Context, ‘ a feature described by Brave as providing ‘extracted page snippets and source metadata, in a form and length suited to an LLM’s needs, without requiring additional scraping.’ The classic search engine sends users to websites, whereas reselling the index does not guarantee this referral, as the decision rests with the AI company.
The Times is suing Brave for copyright infringement and circumventing its subscription system, among other grievances. The Daily News, Chicago Tribune, and Orlando Sentinel filed a separate complaint on the same day. In a prior dispute with News Corp, Brave initiated the lawsuit first, claiming it was merely indexing the web. In July 2026, News Corp counter-sued, accusing Brave of hiding its crawlers to prevent publishers from detecting or blocking them, according to Semafor.
