---
title: "1% of Cloudflare’s network put hundreds of millions of new pages into ChatGPT’s index"
description: "Cloudflare reports early results from its July pilot with OpenAI: discovery signals work, and the crawl got cheaper"
author: "Luis Goa"
published: 2026-10-15T13:00:00Z
modified: 2026-10-15T16:23:19Z
url: https://rews.cc/a/1-of-cloudflare-s-network-put-hundreds-of-millions-of-new-pa-f74935
language: en
tags: ["ai", "openai", "cloudflare", "search", "crawling", "tech"]
publisher: "Rews (https://rews.cc)"
---

# 1% of Cloudflare’s network put hundreds of millions of new pages into ChatGPT’s index

*Cloudflare reports early results from its July pilot with OpenAI: discovery signals work, and the crawl got cheaper*

By Luis Goa · October 15, 2026 · https://rews.cc/a/1-of-cloudflare-s-network-put-hundreds-of-millions-of-new-pa-f74935

## In brief

- Pilot covered about 1% of Cloudflare's network and got hundreds of millions of unknown pages indexed by OpenAI's crawler
- It targets indexing coverage: valuable pages like manuals and live menus often lack the inbound links crawlers rely on
- Cloudflare's network derives signals from request traffic, which OpenAI's search crawler then uses
- Filtering removes junk before fetching: nonexistent files, soft-404 redirects and duplicate URLs
- Change signals aim to recrawl only what changed, though Cloudflare concedes the pilot initially increases total crawling

A pilot covering roughly 1 percent of Cloudflare’s network was enough to get hundreds of millions of previously unknown web pages discovered and indexed by OpenAI’s search crawler, [according to Cloudflare’s account](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/) of early results from a research pilot the two companies launched in July 2026. The figure is self-reported, from the party that ran the pilot, with no outside verification — treat “hundreds of millions” and the accompanying “resounding yes” as a vendor’s measurement of its own experiment. The mechanism underneath it is nonetheless worth reading closely, because it describes where the economics of the AI web are heading.

The problem the pilot addresses is indexing coverage. Valuable pages — the installation manual for one exact washing-machine model, current passport fees, a restaurant’s live menu — often have almost no inbound links, the signal traditional search crawlers depend on. If AI systems are displacing the search-results page as the place questions get answered, a page no crawler finds effectively does not exist, and the person asking gets a wrong answer instead.

![A flashlight casts a wide yellow beam across a teal background, illuminating cream paper sheets among dark sheets scattered outside the light.](https://rews.cc/img/a60df120e86f16989cd026aea146e024ece0f523.webp)

## How the signals flow

The pilot’s data flow, reduced to its parts: Cloudflare’s network observes request traffic to participating customer sites → it derives signals (a URL exists and serves distinct content; a page changed since the last crawl; a page matches patterns associated with junk or compromise) → OpenAI’s search crawler uses those signals to decide what to fetch → OpenAI independently evaluates and ranks pages for its index → ChatGPT answers cite or link the source, sending the user back to the publisher. Cloudflare calls the round trip a feedback loop. The loop’s output so far is measured entirely at the middle — pages entering OpenAI’s index — not at the ends, where publishers would see visits and money.

The more concrete engineering result is what the filtering removes. Automated traffic reliably asks for files that do not exist; some sites redirect arbitrary addresses to their homepage, making a soft-404 look like a page; other URLs serve duplicates. Cloudflare says recognizing these patterns before a fetch is attempted cuts wasted crawl work — which costs the site owner origin load and the crawler its fetch budget. On the safety side, the companies are testing signals that flag single compromised pages — a planted fake login form on an otherwise legitimate site — so that one page can be excluded without penalising the whole domain. Early results, again self-reported, “suggest” the signals work; no error rates are given.

![Dark blue tweezers lift a red document bearing a worm symbol from a row of cream documents against a teal background.](https://rews.cc/img/3f891f1e9464af024f949e33bbc25ff2861c3698.webp)

## The crawl budget

Keeping an index current by refetching everything more often costs both sides: repeated downloads of unchanged pages are pure load for the site and pure expense for the crawler. Change signals are the intended fix — recrawl only what changed. Cloudflare concedes the pilot initially increases total crawling, because newly discovered pages must be fetched, evaluated and indexed; the claim is that filtering and change signals then reduce the steady-state work. Whether that nets out below the old baseline is the part no number in the announcement settles.

On consent, the line Cloudflare draws is metadata versus content: network-derived signals about pages are shared for search and discovery; page content is not shared for model training, and a signal granting that a page exists does not grant permission to access it — domain owners manage that through Cloudflare’s AI Crawl Control. The company situates the work alongside OpenAI’s GPT-6 and Intelligent UI push to make ChatGPT a venue for shopping and task completion. The announced next phase — better page selection, faster change signals, measuring “real business outcomes” for publishers — is also the test that matters. An index growing by hundreds of millions of pages is OpenAI’s win. Whether a discovered page earns its owner anything is the number the pilot has not yet produced.