Proxies for AI training data: throughput, sizing and rules

For most crawls, the right proxies for AI training data are dedicated datacenter IPv4 ports. The job is millions of small HTML fetches from lenient hosts, and a flat monthly port does not charge per page. Residential per GB covers the slice of the corpus that needs a country or city the ports do not offer. And no proxy changes a site’s opt-out.

A crawl for a training corpus looks nothing like a price monitor. It touches millions of URLs across tens of thousands of hosts, fetches each page once or twice, and throws away everything but the text. The pitch pages skip that part and sell residential bandwidth by the gigabyte, and pool sizes in the tens of millions, as if every target were strict retail.

Most of the web a training crawler reads is lenient. Blogs, documentation, forums, public records. Page size and politeness decide the cost. How the crawler treats robots.txt, the newer opt-out signals and its own identity decides whether the corpus is defensible.

Sizing proxies for AI training data

Start with bytes, not requests. Training data is text, so the crawler needs the HTML and nothing else: no images, no scripts, no fonts. The HTTP Archive Web Almanac 2025 page weight chapter (published January 15, 2026) puts the median mobile HTML transfer size at 22 KB for home pages and 20 KB for inner pages. That is compressed size on the wire, which is what a proxy meters when the client sends Accept-Encoding.

The median full page, with every image and script, is 2,164 KB on mobile in the same chapter. A crawler that drives a full browser for every URL pays for about 100 times the bytes it keeps. For web data for LLM training, a plain HTTP client that fetches HTML only is the first cost decision, and it is bigger than any tier choice.

Throughput. A 10 million page crawl spread over 30 days is 10,000,000 / 2,592,000 seconds, about 3.9 pages per second. At 22 KB each that is roughly 85 KB per second, under 1 Mbit/s. At this volume large scale web scraping is a scheduling problem, set by how many hosts are in flight at once and how often each one is hit.

Concurrency. If the crawler waits 5 seconds between requests to the same host, it needs about 20 hosts in flight to hold 3.9 pages per second (3.9 x 5 = 19.5). The scheduler handles that; the proxy count follows from blast radius instead.

Common Crawl before a custom crawl

Common Crawl’s August 2026 archive (published August 24, 2026) holds 2.14 billion pages from 33.1 million registered domains, free to download. For broad text, filtering an existing archive is cheaper than any proxy.

A custom crawl earns its cost when the corpus needs pages Common Crawl skipped, fresher captures, or a domain list it covers thinly. Size the proxy bill for that gap, not for the web.

Why datacenter carries the crawl

A training crawl sends a lot of small, polite requests, and a flat line per port fits that, which is how datacenter proxies are billed. Dedicated IPv4 ports are billed per port per month with no bandwidth cap, so the 10 millionth page adds nothing to the bill. How the addresses work, and where they fail, is in what are datacenter proxies.

How many ports. Bandwidth does not set the number; blast radius and shared infrastructure do. One large CDN fronts thousands of sites, and its edge sees every request from one IP across all of them. Spreading the crawl over 20 to 50 ports limits the blast radius: a port that lands on a blocklist takes out a twentieth of the crawl, not all of it.

What ports do not buy. More IPs must never mean more requests per host. The per-host crawl budget belongs to the crawler, so the rate limit has to be one shared counter per host across every port. Splitting one site’s crawl over 50 addresses to go 50 times faster is the pattern a rate limiter exists to stop, and it is the fastest way to get a crawler’s whole range blocked.

Proxy types by pipeline phase

A training pipeline is not one crawl. Each phase has a different volume and a different host list, and the tier follows from those.

Pretraining. Breadth: millions of pages from thousands of hosts, fetched once. This is the case above. Common Crawl first, then datacenter ports for the gap.

Fine-tuning corpora. Depth on a few hundred hosts in one domain, such as documentation sites, legal texts or a language community. Same rule. Most of these hosts are lenient, so they stay on ports; a host that refuses the crawler is skipped, not moved.

Evaluation sets. Small and one-off, often a few thousand pages. At that size the tier barely moves the bill, so pick on the locations the set needs.

Retrieval and agents. Fetching pages at query time for retrieval or an agent is a different job: low volume, real-time, often on consumer sites. It is not training data collection, and the sizing here does not apply.

When residential is worth it

Some text only exists by location: regional news, local listings, pages a site serves in one language to one country. When the corpus needs a country or city the datacenter ports do not offer, rotating residential proxies carry that slice, with country, region and city targeting; datacenter vs residential proxies compares the two tiers in full. The job becomes a per-GB line for a small slice of the corpus.

Every host starts on datacenter. A host that answers the first polite requests with a 403 or a challenge page has refused this crawler, even when robots.txt allows the path: log it, skip it, and ask through the site’s contact or licensing route if its text matters. Moving a refused host to another exit type is the routing-around the code of practice rules out. In most large scale web scraping for training corpora the skip list stays short, because anti-bot vendors cluster on strict retail and social platforms, which are rarely the text a corpus needs.

Mobile and sticky sessions

No training crawl needs a carrier origin. The text a corpus wants does not sit behind checks for a mobile network, and mobile is built for app-first platforms that do.

Rotation matters less than it looks. A stateless HTML fetch does not need a sticky session. On the residential residual, a timed session per host (5 to 60 minutes on proxymint’s residential tier) keeps each host seeing one visitor at the rate it agreed to, instead of a new address on every request.

Robots.txt and AI crawler tokens

The rules layer is where training crawls differ most from ordinary scraping. RFC 9309 (published September 2022) standardised the Robots Exclusion Protocol. Four of its rules shape a crawler’s code:

• Cache for at most 24 hours. A crawler should not use a cached robots.txt for more than 24 hours unless the file is unreachable.
• A 4xx means no rules. If robots.txt returns 400 to 499, the crawler may access any resource on the server.
• A 5xx means stop. If robots.txt returns 500 to 599, the crawler must assume a complete disallow.
• Parse at least 500 KiB. Files larger than the crawler’s limit get cut, and rules past the cut are lost.

The same RFC says the rules are not a form of access authorization. That cuts both ways. robots.txt does not lock anything, so nothing technical stops a crawler that ignores it. But a site that writes a Disallow has stated what it wants, and a training corpus built over that statement is hard to defend later.

The AI-specific tokens

Each AI crawler has its own robots.txt token, and site owners now write groups for them. OpenAI’s crawler documentation (checked September 2026) separates GPTBot, which crawls content that may be used to train its foundation models, from OAI-SearchBot, which surfaces sites in ChatGPT search, and says a robots.txt change takes about 24 hours to reach its search systems. Google’s common crawlers page (updated July 14, 2026) describes Google-Extended as a robots.txt control for Gemini training and grounding that has no separate user agent string and does not affect Search inclusion. Common Crawl crawls as CCBot.

A new crawler has its own token that no site has written a rule for yet. So the honest check reads two things: the group that matches the crawler’s own token, and whether the site disallows the known AI training tokens. A site that blocks GPTBot, Google-Extended and CCBot but allows everything else has not invited a different training crawler in. It has not heard of it yet.

Opt-out signals beyond robots.txt

robots.txt can say “do not crawl”, but not “crawl for search, do not train”. Two newer signals fill that gap, and a training crawler meets both in 2026.

Content-Usage. The IETF AI Preferences working group’s attachment draft (published September 14, 2026) adds a Content-Usage rule to robots.txt and an HTTP header of the same name. A line such as Content-Usage: train-ai=n states that the content should not train AI models. It is an Internet-Draft, not an RFC yet, but it costs a few lines to read.

Edge defaults. Cloudflare’s changelog of July 1, 2026 says that from September 15, 2026, new domains block Training and Agent bots by default on pages that display ads, while Search crawlers stay allowed. A training crawler can now meet a block on a site whose robots.txt says nothing about it. That block is the site’s answer, and routing around it through another exit is evasion.

Before a host joins the crawl, check each opt-out in turn:

• The crawler’s own token. Skip the host when its robots.txt group for your token disallows the path.
• The known AI training tokens. Skip it when it disallows GPTBot, Google-Extended and CCBot alike.
• Content-Usage. Skip it when robots.txt or a response header states train-ai=n.
• The edge. Treat a block from the site’s CDN as the site’s answer and log the host as refused.
• The terms. Read the site’s terms for a ban on AI training or automated collection before the first fetch.

A robots.txt gate in code

This gate follows RFC 9309’s status rules, caches per host for 24 hours, and records any site-wide Content-Usage preference. It uses Python’s standard library parser, which ignores Content-Usage lines, so a small reader handles those. The robots.txt fetch goes through the same proxy as the crawl, so the site sees one consistent crawler.

import time
import requests
from urllib.parse import quote
from urllib.robotparser import RobotFileParser

TOKEN = "ExampleTrainBot"
UA = "ExampleTrainBot/1.0 (+https://example.com/bot)"
USER, PASS = quote("USERNAME", safe=""), quote("PASSWORD", safe="")
PROXY = f"http://{USER}:{PASS}@HOST:PORT"
AI_TOKENS = ["GPTBot", "Google-Extended", "CCBot"]
CACHE = {}  # host -> (fetched_at, parser, train_ai)

def train_ai_pref(lines):
    # IETF aipref draft: "Content-Usage: train-ai=n". Path-scoped lines are skipped,
    # and every group is read, which is stricter than the draft requires.
    pref = None
    for line in lines:
        key, _, value = line.partition(":")
        if key.strip().lower() != "content-usage" or value.strip().startswith("/"):
            continue
        for item in value.split(","):
            k, _, v = item.strip().partition("=")
            if k.strip() == "train-ai":
                pref = v.strip()
    return pref

def rules_for(host):
    hit = CACHE.get(host)
    if hit and time.time() - hit[0] < 24 * 3600:
        return hit[1], hit[2]
    r = requests.get(f"https://{host}/robots.txt", headers={"User-Agent": UA},
                     proxies={"http": PROXY, "https": PROXY}, timeout=20)
    rp, lines = RobotFileParser(), []
    if r.status_code >= 500:
        lines = ["User-agent: *", "Disallow: /"]      # unreachable: full disallow
    elif r.status_code < 400:
        lines = r.content[:500 * 1024].decode("utf-8", "ignore").splitlines()
    rp.parse(lines)                                    # 4xx: no rules
    CACHE[host] = (time.time(), rp, train_ai_pref(lines))
    return rp, CACHE[host][2]

def may_fetch(host, url):
    rp, train_ai = rules_for(host)
    if train_ai == "n" or not rp.can_fetch(TOKEN, url):
        return False
    # a site that opts out of every known AI trainer gets skipped too
    blocked = [t for t in AI_TOKENS if not rp.can_fetch(t, url)]
    return len(blocked) < len(AI_TOKENS)

The last line is a policy choice, not a protocol rule. A crawler team can make it stricter (skip the site if any AI token is disallowed) and should write the choice down with the dataset. Pages can also send Content-Usage as a response header, so a full pipeline checks the header on each fetch before keeping the text.

Crawler identity and verification

An anonymous crawler gets treated like any other unknown traffic, scored the way how anti-bot systems work describes. A named one can be verified, and verification is where the choice of proxy stops being only about price.

Cloudflare’s verified bot policy (updated July 1, 2026) accepts two ways to prove identity: Web Bot Auth, a cryptographic signature on each request, or IP validation, meaning a published IP list with a stable user agent, or reverse DNS. It also requires the bot to obey robots.txt and crawl directives and keep reasonable request rates. Using the verified IPs for anything beyond that one service, or ignoring Crawl-delay, can get a bot removed.

That favours a fixed address. Dedicated datacenter ports can be published as a list, used for the crawler alone, and checked by any operator, the way OpenAI publishes the ranges GPTBot crawls from. Rotating residential exits are shared home connections that change on a timer, so they can be neither published nor sole-use. Web Bot Auth signatures work from any IP, but the crawler still needs a stable, honest identity behind them.

• Name the crawler. Send a user agent with a product token and a URL that explains the bot and gives a contact address.
• Publish the IPs. A plain JSON file of the port addresses, linked from that URL.
• Keep the ports single-purpose. No scraping jobs, tests or other clients on the crawler’s addresses.

A named crawler on a published set of dedicated IPs gives an operator a choice: allow it, rate-limit it, or disallow it in robots.txt. An unnamed crawler rotating through home IPs leaves only one choice, which is to block everything that looks like it.

Politeness sets the crawl budget

Crawl budget, for a training crawler, is how many requests per host per second it allows itself. Google’s own crawl budget documentation (updated July 22, 2026) describes a crawl capacity limit that rises when a site responds quickly and consistently and falls on slowdowns, 5xx errors and 429 responses. A training crawler should copy that rule rather than invent its own.

• One counter per host. Start at one request in flight per host with a pause of several seconds, and make the counter shared across every port.
• Back off on 429 and 5xx. Halve the rate for that host and honour a Retry-After header when it is present.
• Honour Crawl-delay. RFC 9309 does not define it, but plenty of sites write it, the parser exposes it through crawl_delay(), and Cloudflare’s verified bot policy counts ignoring it against AI crawlers.
• Speed up only on evidence. Raise the rate slowly while response times stay flat.

And the crawl budget is also a cost control. Every retry against a host that is already returning 503 is a wasted request, and on a per-GB tier it is wasted traffic.

Sizing a 10 million page crawl

The first cost question about proxies for AI training data is what 10 million pages move. The page size assumption decides almost everything, so here are two.

Lean: 22 KB per page. HTML only, compressed, at the Web Almanac median. 10,000,000 x 22 KB = 220 GB.

Heavy: 100 KB per page. A cautious planning figure that covers redirects, retries, error pages and larger-than-median documents. 10,000,000 x 100 KB = 1,000 GB.

Most crawls end up running a mixed setup. Say 10% of pages need a city the ports do not cover: 1,000,000 x 22 KB = 22 GB of per-GB traffic on top of the flat ports. Scaling to 50 ports for more spread across shared edges raises the flat line and still adds nothing per page.

Entry prices and every volume band for both tiers are on the pricing page.

Legal and compliance limits

A proxy changes the source address of a request. It does not change copyright, a site’s terms, or any opt-out the site has published. This section describes what the rules say; it is not legal advice.

The EU AI Act. Article 53 requires providers of general-purpose AI models to keep a policy for complying with EU copyright law, including the text and data mining opt-outs reserved under Article 4(3) of Directive 2019/790, and to publish a summary of the content used for training. Those obligations have applied since August 2, 2025.

The code of practice. The EU’s General-Purpose AI Code of Practice (published July 10, 2025) turns that into crawler behaviour for its signatories: crawlers that read and follow robots.txt as RFC 9309 specifies, compliance with other machine-readable opt-outs, no circumvention of paywalls or subscription walls, exclusion of sites that EU courts or authorities recognise as infringing copyright at commercial scale, and published information about the crawlers used.

Read together, these set the bar a training crawl gets measured against, whoever runs it. Rotating through exits to get past a block fails that bar no matter which tier the exits come from.

Keep a fetch log. For every page kept: URL, timestamp, exit IP, user agent, robots.txt status and the rule that allowed it, any Content-Usage value, and the HTTP status. It costs a few bytes per row, and it is the only way to answer a rightsholder, a regulator or a data-governance review with evidence instead of intent.

Proxies for AI training data, phase by phase

Pretraining and fine-tuning. Filter Common Crawl first, then run the gap on dedicated datacenter ports with a named user agent, a published IP list, one rate limit per host and a fetch log. That covers most crawls on its own.

Location slices. The part of the corpus that needs a country or city the ports do not offer runs on residential per GB, and only that part.

Evaluation sets. A one-off evaluation crawl of a few GB favours per GB; a larger crawl, or one that repeats every month, favours flat ports.

Every phase. When robots.txt disallows the crawler or the known AI training tokens, a Content-Usage rule says train-ai=n, the edge blocks training bots, or a host refuses the crawler with a 403 or a challenge, skip the host.

A different IP does not turn a stated no into a yes.