Pay per crawl: how HTTP 402 lets sites charge AI crawlers

Pay per crawl is a Cloudflare feature that lets a site charge AI crawlers for each page they fetch. The site sets a price. A crawler that has not agreed to pay gets an HTTP 402 Payment Required response with a crawler-price header. A verified crawler that retries with a signed payment header gets the page and a crawler-charged receipt, and Cloudflare bills it.

The feature has been in closed beta since Cloudflare announced it on July 1, 2025. Two kinds of reader land on it. One is the engineer whose crawl logs filled with 402s overnight and needs to know which requests to pay for, fix or drop. The other is the site owner deciding which bots to charge, and at what price.

Both need the same pieces: the headers on the wire, the Web Bot Auth signature that decides who pays, the error codes, and how the model sits next to x402 and RSL. A proxy changes none of the billing, for reasons the signature section makes clear.

What pay per crawl is

Pay per crawl sits inside Cloudflare’s AI Crawl Control. According to Cloudflare’s docs (checked September 29, 2026), a site owner sets one price per zone, and each AI crawler request either carries payment intent in its headers and gets a 200, or gets a 402 with the price. Cloudflare is the Merchant of Record: it charges the crawler operator and pays the site owner, with Stripe underneath.

Three actions per crawler. For every AI crawler Cloudflare knows, the site owner picks Charge, Allow or Block. Charge applies the price. Allow serves the page free. Block refuses it outright.

One price for every charged crawler. The FAQ is explicit: a site cannot set a different price per crawler. It can set a different price per page, which comes up under site owner setup.

Security rules win. A crawler blocked by the WAF or Bot Management stays blocked even when the crawler is set to Charge.

The HTTP 402 request flow

Status code 402 has sat in the HTTP specs since the 1990s without a job. RFC 9110 (June 2022) still says only that it “is reserved for future use”. Pay per crawl gives it one, with four custom headers. The exchange, as the crawl pages reference documents it:

GET /article HTTP/2
host: example.com
user-agent: ExampleBot/1.0

HTTP/2 402
crawler-price: USD PRICE

The crawler decides whether to pay and retries with one of two headers, signed (next section):

GET /article HTTP/2
host: example.com
user-agent: ExampleBot/1.0
crawler-exact-price: USD PRICE
signature-agent: "https://bot.example.com"
signature-input: sig1=("@authority" "signature-agent" "crawler-exact-price");created=1759140000;expires=1759140060;keyid="KEY_THUMBPRINT";alg="ed25519";tag="web-bot-auth"
signature: sig1=:BASE64_SIGNATURE:

HTTP/2 200
crawler-charged: USD PRICE

crawler-exact-price agrees to the price quoted on the 402. It has to match exactly, or the retry fails.

crawler-max-price sets a ceiling. The crawler can send it on the first request and skip the 402 round trip on every page priced at or below it, and it is charged the page price, not the ceiling. A page priced above the ceiling still returns 402.

crawler-charged on the 200 is the amount billed for that request. Cloudflare’s docs recommend logging every value, because the crawler’s own records are the only running total it has until the invoice.

Web Bot Auth verification

The payment header alone proves nothing. Anyone can type crawler-max-price into curl. What makes the request billable is a signature that ties it to a registered crawler, using Web Bot Auth, which relies on two IETF drafts: a key directory draft and an architecture draft on top of HTTP Message Signatures (RFC 9421).

The Web Bot Auth setup (checked September 29, 2026) runs in four steps:

1. Generate an Ed25519 key pair and convert the public key to a JSON Web Key.
2. Host the public keys at /.well-known/http-message-signatures-directory over HTTPS, as a signed JSON Web Key Set.
3. Register the crawler and its key directory through the Bot Submission Form, choosing Verified Bot or Signed Agent, with the user agent strings it sends.
4. Sign every request with three headers: Signature-Agent (the directory URL, in double quotes), Signature-Input (the covered components, keyid, created, expires, tag="web-bot-auth") and Signature.

The payment header must be one of the signed components in Signature-Input, and the crawl pages reference flags this as the step to get right. A request that carries crawler-exact-price outside the signature fails. Cloudflare also recommends a short expires, about a minute, because it does not track nonces against replay.

Identity moved from IP to key

Verified bots have long been recognised by where they come from, through published IP ranges and reverse DNS, two of the inputs described in how anti-bot systems work. That only works when the operator owns its addresses. Web Bot Auth moves identity into the request. The signature covers @authority and the chosen headers, so it verifies the same from any network path, and the exit network no longer decides who the crawler is.

Site owner setup

For a site on Cloudflare, the path in the docs is: enable the feature in account settings, set a price, select crawlers to charge, then monitor activity and payouts. The docs set a minimum price per crawl, and the charge applies to each successful retrieval, meaning a 200.

Dynamic pricing. The zone price is a default. With dynamic pricing on, the origin (or a Worker in front of it) can return its own crawler-price header per response, and that price wins. Cloudflare adds a cf-pay-per-crawl request header to every origin request, with pricing=zone-default, in-band or bypass, so the origin knows when a price is wanted.

Free paths. /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json are always free. Other paths can be exempted with a Configuration Rule, for example a free homepage and category pages that help crawlers find the paid articles.

Search crawlers. Cloudflare’s tip on the crawler list warns that setting search engine crawlers to Charge or Block can hurt how the site is indexed. Charging applies per crawler, so a site can charge training crawlers and leave search crawlers on Allow.

Is charging worth it?

Revenue follows the count of charged 200 responses per month, and that count is the hard number. AI Crawl Control shows requests per crawler, so a month of logs before switching on Charge gives the real volume instead of a guess. For most sites AI crawlers are a small share of requests and the per-page price is small, so the direct income is modest.

Large sites weigh it differently. Stack Overflow, which co-launched the model, described its setup in February 2026: bot categories and WAF rules decide which crawlers see a 402, and pay per crawl sits next to enterprise data licensing as the programmatic option for crawlers without a contract. The value there is control over who takes the data on what terms, with income as a second effect.

Crawler operator setup

The operator side has five steps in the docs: create a Cloudflare account, verify the crawler with Web Bot Auth, discover payable content, connect Stripe, crawl.

Discovery. A verified crawler can call GET https://crawlers-api.ai-audit.cfdata.org/charged_zones, signed with the same Web Bot Auth headers, to list the domains that charge it. The discovery docs suggest caching the list and refreshing it. A domain missing from the list does not charge that crawler, but it may still block it in the WAF or in robots.txt, which the crawler should respect.

Spend limits are the crawler’s job. Cloudflare’s Stripe setup page states that it does not configure spending limits, and invoices arrive through Stripe. Every re-crawl of the same page is billed again. Error responses are not billed, even when a price header was sent.

User agent. The FAQ says to send the standard user agents that were onboarded to Cloudflare and identified through Web Bot Auth, not a new string per job.

Error codes and 402 handling

Every failed paid request carries a crawler-error header. The error code reference lists eleven. The ones a pipeline meets most:

crawler-error	Status	What the pipeline should do
CrawlerForbidden	403	The site blocked this crawler. Stop; do not retry.
MissingCrawlerPrice	402	No price header sent. Decide to pay, or drop the URL.
InvalidCrawlerExactPrice	402	The quote changed. Re-read crawler-price before paying.
InvalidCrawlerMaxPrice	402	Page costs more than the ceiling. Skip or raise the ceiling.
StrongAuthRequired / InvalidSignature	400	Signature missing or broken. Fix signing; retrying the same request fails.
ConflictingPriceHeaders	400	Both price headers sent. Send one.
PaymentFailed	403	Billing setup problem on the crawler account. Pause paid fetches.
InternalError	500	The only one to retry, with exponential backoff.

A crawler that treats every non-200 as “retry later” burns requests on answers that will not change. A small decision layer fixes that:

def handle(resp, budget):
    err = resp.headers.get("crawler-error")
    if resp.status_code == 200:
        charged = resp.headers.get("crawler-charged")
        if charged:  # absent on free or unbilled requests
            budget.record(charged)
        return "ok"
    if err == "InternalError":
        return "retry_with_backoff"
    if resp.status_code == 402 and err in (None, "MissingCrawlerPrice",
                                            "InvalidCrawlerExactPrice"):
        price = resp.headers.get("crawler-price")  # currency code and amount
        return "pay" if budget.allows(price) else "skip"
    return "stop"  # blocked, bad signature or billing: do not loop

The pay branch re-sends the request with a signed crawler-exact-price. Everything else logs and moves on.

Pay per crawl vs other models

Cloudflare’s model is one of several attempts to price machine access. They differ on who sets the price, how the payer proves identity, and where the money moves.

x402 is an open standard now hosted by the x402 Foundation (README checked September 29, 2026). The server answers 402 with a PAYMENT-REQUIRED header, the client pays and retries with PAYMENT-SIGNATURE, and a PAYMENT-RESPONSE header comes back with the 200. It suits agents that pay per API call without holding an account with each site; the network side of those agents is covered in proxies for AI agents.

RSL (Really Simple Licensing) puts licensing terms, including per-crawl or per-use payment, in a file referenced from robots.txt. Plagiarism Today’s comparison (September 11, 2025) makes the practical point: RSL states the terms, and enforcement depends on the infrastructure in front of the site.

Pay per use. On July 1, 2026 Cloudflare said it is evolving Pay Per Crawl into Pay Per Use, paying publishers when their content is used in an AI answer rather than per fetch. The 402 mechanics above are what the docs describe today; the billing event may move.

Where proxies fit

A proxy changes the network origin of a request. It does not change the key that signs it, the user agent the crawler registered, or the action the site picked for that crawler. For billing, the exit network has no bearing on whether the crawler pays.

Proxies still have honest jobs next to paid crawling. A site owner testing the setup needs to see what a crawler sees from outside the office network: which paths return 402, whether the free paths stay free, whether a Configuration Rule exempts the right URIs, and whether regional pages behave the same in every market. Fast, fixed datacenter proxies cover repeatable checks from several countries; rotating residential proxies show how the same pages answer consumer networks. A data team collecting public data from sites that allow it has the same choice between categories.

Status and availability

The feature is in closed beta. Sites and crawler operators join through Cloudflare’s sign-up form or, for Enterprise customers, their account executive. The headers and error codes above are what the docs specify today. Two things are still moving: the Pay Per Use shift, and how many sites keep Cloudflare’s new default blocks from September 15, 2026 or switch them to Charge.