Building a web index is one of the most expensive things a technology company can decide to do, which is why almost nobody does it. Google has one. Microsoft has one. OpenAI built one rather than keep renting Bing. If the log evidence that surfaced this week reads the way it appears to, the Meta AI crawler is the front end of a fourth, and that changes the arithmetic of who gets to see your site directly.
The evidence is thin but specific. Pieter Levels, an independent developer who runs a portfolio of small sites, posted screenshots of Meta crawler activity across them and put forward the reasoning himself: Meta wants its own web index so that when its AI performs a web search, the query does not end up at Google, where Google could then use it for its own training. Barry Schwartz covered it at Search Engine Roundtable on August 10, noting the history of Meta's earlier and unsuccessful search partnership with Bing.
What turned up in the logs
One developer's server logs are not a confirmation, and it is worth being precise about what is established. Meta crawlers hitting a set of sites is observed. Meta building a general web index for AI retrieval is an inference from that observation plus a plausible motive. Meta has not announced anything. If you repeat this in a client meeting, repeat it with that hedge attached.
That said, it is a good inference. The motive Levels describes is the same one that pushed OpenAI off Bing: an AI company that routes its retrieval through a competitor's index hands that competitor a live feed of what its users are asking. For a company that has spent two years building frontier models and has no search surface of its own, owning the retrieval layer is the missing piece.
| CLAIM | STATUS | SOURCE |
|---|---|---|
| Meta crawlers active across independent sites | Observed, screenshots published | Pieter Levels, via Search Engine Roundtable, Aug 10, 2026 |
| Motive is avoiding Google as a retrieval intermediary | Levels' stated reasoning, not a Meta statement | Same coverage |
| Meta is building a general web index | Inference, not confirmed by Meta | No primary source |
| Meta crawlers at roughly nine billion requests quarterly | Secondary report, not independently verified here | Anicca weekly roundup, Aug 11, 2026 |
| Googlebot reached 2.99x more unique URLs than Meta-ExternalAgent | Published measurement | Cloudflare Radar, January 2026 |
Why the Meta AI crawler matters more than the headline
The competitive story is the least interesting part. What matters operationally is that each independent index produces an independent reading of your site, and those readings already disagree more than most teams realize. We have documented engines disagreeing on which sources to cite for the same query, and index divergence is the mechanism underneath it. Two engines that crawl on different schedules, render JavaScript differently, and apply different canonical logic will end up believing different things about the same page.
Add a fourth and the divergence widens. Practically, that means a site fix does not propagate uniformly: you correct a product page, Google reflects it in days, one assistant reflects it in weeks, and one is still quoting the old claim in three months. Anyone measuring AI visibility as a single number is averaging over four different pictures of reality.
A fourth index changes the robots.txt calculation
Most enterprise robots.txt files fall into one of three states: written in 2019 and never touched, hardened against AI crawlers in a panic during 2025, or maintained by someone who reads release notes. Only the third group is in good shape, and a new crawler arriving is the event that exposes the other two.
The default posture in a wildcard-heavy file is accidental. A blanket disallow written to stop training scrapers will also stop retrieval crawlers, which is the distinction we drew in training crawlers versus retrieval crawlers. Blocking a training bot costs you nothing you can measure. Blocking a retrieval bot removes you from an answer surface. Those two decisions live one line apart in the same file and get made as if they were the same decision.
Meta AI crawler volume against the referral you get back
The economics are the honest objection to letting any new crawler in, and they are getting worse rather than better. Anicca's August 11 roundup attributed roughly nine billion requests quarterly to Meta crawlers with minimal referral traffic in return. Treat that figure as a secondary report rather than a measurement you can defend, but the direction is consistent with everything else in this category.
Relative unique-URL reach by crawler, expressed as a share of Googlebot's reach, derived from the Cloudflare Radar comparison published for January 2026. Shown to compare coverage breadth, not request volume.
Set that against what we found in the crawl-to-referral economics work: the ratio between pages crawled and visitors returned runs into the thousands to one for several AI companies, and in the worst measured case into the tens of thousands. On a pure bandwidth-for-traffic trade, none of these crawlers pay their way. That is why publisher-side metering exists and why it keeps growing.
But bandwidth-for-traffic is the wrong ledger for most B2B companies. If a buyer asks an assistant which vendors handle a problem and your competitor is named because your pages were readable and yours were not, the cost of the block is a deal, not a server bill. Publishers with millions of pages and ad-funded models face a genuinely different calculation from a software company with four hundred pages and six-figure contracts. Most crawler advice on the internet is written for the first group and read by the second.
| SITE TYPE | CRAWL COST MATTERS | CORRECT DEFAULT | THE REAL RISK |
|---|---|---|---|
| Large ad-funded publisher | Yes, materially | Meter or negotiate access | Giving away the archive for nothing |
| B2B software, few hundred pages | No, negligible | Allow retrieval bots explicitly | Absence from answers where buyers decide |
| Ecommerce catalog | Sometimes, at scale | Allow, and feed structured product data | Competitor specs quoted instead of yours |
| Docs and developer content | No | Allow everything that retrieves | Being wrong in an assistant's answer forever |
What to do before the index solidifies
There is a version of this article that ends with a recommendation to block Meta until it explains itself, and it is the wrong recommendation for almost every company that will read this. Blocking is a lever with an asymmetric payoff: the saving is a bandwidth line you were never going to notice, and the cost is being absent from a retrieval surface that may matter in eighteen months. Reserve it for cases where the content itself is the product and the archive is the asset. Everywhere else, the correct posture is deliberate permission with good instrumentation, so you can see what a crawler took and what it sent back.
New indexes are most malleable early. The first crawl of your site establishes a baseline understanding that later crawls update incrementally, and correcting a wrong early impression is materially harder than establishing a right one. This is the same reason a poorly handled site migration haunts a domain for a year.
So the window is now, and the work is unglamorous. Make sure the pages that define what you do are reachable without JavaScript, state your category and differentiators in plain sentences near the top rather than only in a hero graphic, and keep your entity markup consistent across the site so a new crawler resolves you to one company rather than three. We ran exactly this sequence for Zenity ahead of a category that was still being named, and the point of doing it early was that there was nothing to correct later.
For companies in fast-defining categories, the stakes are higher than for established ones, because a new index forms its idea of a young category from whatever it finds first. That is the argument we make to B2B software teams who think GEO is a next-year problem: next year the index will already have an opinion.
One developer's logs are a weak basis for a strategy and a perfectly good basis for a check. Spend the afternoon on the crawler enumeration whether or not Meta is doing what it appears to be doing. The original coverage is at Search Engine Roundtable's August 10 report.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Josh leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.