Brave Search Help

Brave Search Crawler

Like any other search engine, Brave search has a crawler to discover new pages and index their content.

The Brave Search crawler does not advertise a differentiated user-agent because we must avoid discrimination from websites that allow only Google to crawl them. However, if a domain or page is not crawlable by any search engine (it has a noindex tag), or if it is not crawlable by googlebot, then Brave Search’s bot will not crawl it either. One last option is to contact privacy@brave.com to solicit a delisting of your website, which will stop crawling.

Brave Search’s crawler is partially powered by information provided by users enrolled in the Web Discovery Project (WDP) option in Brave browser’s search settings, which is an off-by-default (aka opt-in), privacy-preserving system. WDP has multiple mechanisms to prevent Brave from knowing who is contributing what (WDP is also open-source for auditing and inspection by anyone).