Crawlee proxy setup: rotation, sessions, tiers and SOCKS5

A Crawlee proxy is set with one object. Create a ProxyConfiguration with your proxy URLs (proxyUrls in Node, proxy_urls in Python), credentials inside each URL, and pass it to the crawler as proxyConfiguration or proxy_configuration. Crawlee then gives every new session the next URL in the list, keeps that session on it, and reports the choice in proxyInfo inside the request handler.

The official guide covers that setup, tiers and a custom proxy function. The support threads cover what happens next. A PlaywrightCrawler logs three different proxies in proxyInfo while every page leaves through the first one. A socks5:// URL that curl accepts fails four times in a default CheerioCrawler and never starts in Python. On that default client, one wrong password turns each request into eleven refused logins, whatever maxRequestRetries says.

Each behaviour below was reproduced on crawlee 3.18.2 (the npm latest, published September 29, 2026) and Crawlee for Python 1.11.0, against a local logging HTTP and SOCKS5 proxy, and traced to the ProxyConfiguration source at tag v3.18.2 (checked October 19, 2026). Placeholders are USERNAME, PASSWORD, HOST and PORT.

Crawlee proxy setup in one answer

In Node, the whole setup is a list of URLs and one crawler option:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ['http://USERNAME:PASSWORD@HOST:PORT'],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ request, proxyInfo, $ }) {
        console.log(request.url, proxyInfo?.hostname, $('title').text());
    },
});

await crawler.run(['https://example.com/']);

The same object works on HttpCrawler, JSDOMCrawler, PlaywrightCrawler and PuppeteerCrawler. In Python the class takes the same three options in snake case:

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration


async def main() -> None:
    proxy_configuration = ProxyConfiguration(
        proxy_urls=['http://USERNAME:PASSWORD@HOST:PORT'],
    )
    crawler = BeautifulSoupCrawler(proxy_configuration=proxy_configuration)

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        proxy = context.proxy_info.hostname if context.proxy_info else None
        context.log.info(f'{context.request.url} via {proxy}')

    await crawler.run(['https://example.com/'])


asyncio.run(main())

Credentials. Username and password go in the URL. Percent-encode @, : or / in a password; Crawlee and got-scraping decode them again before building the login. The proxy string formats guide converts the host:port:user:pass lines most dashboards export.

One source per object. ProxyConfiguration accepts exactly one of proxyUrls, tieredProxyUrls or newUrlFunction. Passing two throws in the constructor, in both SDKs.

What the handler sees. proxyInfo (context.proxy_info in Python) carries the URL, hostname, port, username, the session id and, with tiers, the tier number. It records what Crawlee chose. In a default browser crawler the page can leave through a different proxy, as the browser section shows.

How Crawlee rotates proxies and sessions

Rotation in Crawlee follows the session. With proxyUrls, the configuration hands the next URL in round robin to every session id it has not seen, then returns the same URL for that session from then on. Requests without a session (useSessionPool: false) just take the next URL each time.

Early in a crawl, almost every session is new. The session pool creates a fresh session for every request until it holds maxPoolSize sessions, and the default is 1,000 (session_pool/consts.ts). So the first thousand requests of a crawl rotate through the list one by one. In the local test, nine requests over three URLs went 1, 2, 3, 1, 2, 3, each on its own session.

To make sessions repeat, and so keep cookies and an IP together, shrink the pool:

const crawler = new CheerioCrawler({
    proxyConfiguration,
    sessionPoolOptions: { maxPoolSize: 20 },
    persistCookiesPerSession: true,
    async requestHandler({ session, proxyInfo }) {
        // the same session id always comes back with the same proxy URL
    },
});

Retired sessions. A response with status 401, 403 or 429 retires the session in both SDKs, and the retry gets a new session and therefore the next proxy URL. In the test, with one exit returning 403, every blocked page succeeded on its retry through the next URL.

One gateway URL. With a single rotating gateway in proxyUrls, every session receives the same URL and Crawlee’s round robin changes nothing. The exit then changes according to the provider’s rotation setting. On proxymint per-GB orders that setting lives on the order: a new IP every request, a timer from 5 to 60 minutes, or sticky while the device stays online, and it can be changed later without new credentials. Rotating vs sticky proxies covers which mode fits which crawl.

PlaywrightCrawler and browser proxies

A browser is launched with one proxy. Crawlee’s browser pool, by default, opens the next page in any browser that has free capacity, whichever proxy that browser was launched with. The session’s proxy URL is still written to proxyInfo.

In a local test, a default PlaywrightCrawler with three proxy URLs logged ports 8001, 8002 and 8003 in turn in proxyInfo, while the IP echo showed the first proxy’s exit on all six pages. At concurrency 1 and at concurrency 3 the result was the same.

Two launchContext options fix it. useIncognitoPages: true opens each page in its own browser context with the session’s proxy. browserPerProxy: true reuses only browsers launched with the page’s proxy and launches new ones otherwise. Both exist in 3.18.2, and both gave six pages on three exits in the order proxyInfo reported:

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
        'http://USERNAME:PASSWORD@HOST:PORT_1',
        'http://USERNAME:PASSWORD@HOST:PORT_2',
        'http://USERNAME:PASSWORD@HOST:PORT_3',
    ],
});

const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    launchContext: {
        // one browser context per page, opened with that page's proxy
        useIncognitoPages: true,
    },
    async requestHandler({ page, proxyInfo }) {
        const exit = await page.textContent('body');
        console.log(proxyInfo?.port, exit);
    },
});

await crawler.run(
    Array.from({ length: 6 }, (_, i) => ({
        url: 'https://api.ipify.org',
        uniqueKey: `exit-check-${i}`,
    })),
);

Incognito pages share no cache with each other. browserPerProxy keeps a shared context per proxy at the cost of more browser processes. The Playwright proxy setup guide covers the scopes Playwright itself offers outside Crawlee.

Credentials in browsers. Chromium’s proxy flag takes no username and password, so the browser pool starts a small local proxy (the proxy-chain package) that adds the credentials and forwards to yours. That is also why authenticated SOCKS5 works in browser crawlers, covered below.

Background traffic. In the test, headless Chromium opened a tunnel to accounts.google.com through the proxy before loading the first page. On a per-GB plan that traffic is metered like any other.

Tiered proxies in Crawlee

tieredProxyUrls takes a list of lists. Crawlee tracks errors per target domain, starts each domain at tier 0, moves it up a tier when requests there fail, and probes the lower tier again after a while. null in a tier means no proxy at all.

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    tieredProxyUrls: [
        [null], // tier 0: no proxy
        ['http://USERNAME:PASSWORD@DC_HOST:DC_PORT'], // tier 1: datacenter
        ['http://USERNAME:PASSWORD@HOST:PORT'], // tier 2: residential
    ],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ request, proxyInfo }) {
        console.log(request.url, 'tier', proxyInfo?.proxyTier ?? 0);
    },
});

await crawler.run(['https://example.com/']);

In Python, ProxyConfiguration(tiered_proxy_urls=[[None], [DATACENTER], [RESIDENTIAL]]) does the same. With the lowest tier blocked in the test, both SDKs moved the domain up during the first request’s retries and served all 40 pages from the next tier.

Order tiers by what the target accepts. A target that serves hosting networks belongs on datacenter proxies, which are built for volume. A target that scores consumer origins goes up to residential. Tiers let one crawl cover both kinds of domain without a rule per site.

Crawlee 4. The 4.0.0-rc.1 release candidate (October 6, 2026) throws on tieredProxyUrls: “The tieredProxyUrls option has been removed in Crawlee v4”, pointing to named sessions instead. Code that depends on tiers should pin 3.x until the upgrade guide settles.

Custom proxy function per request

newUrlFunction returns a URL, or null for no proxy, and can be async. In 3.18.2 it receives the session id and an object with the request, so a label can choose the pool:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const DATACENTER = 'http://USERNAME:PASSWORD@DC_HOST:DC_PORT';
const RESIDENTIAL = 'http://USERNAME:PASSWORD@HOST:PORT';

const proxyConfiguration = new ProxyConfiguration({
    newUrlFunction: (sessionId, { request } = {}) => {
        if (request?.label === 'PRODUCT') return RESIDENTIAL;
        return DATACENTER;
    },
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ request, proxyInfo }) {
        console.log(request.label ?? 'LIST', proxyInfo?.port);
    },
});

await crawler.run([
    { url: 'https://example.com/list', label: 'LIST' },
    { url: 'https://example.com/item/1', label: 'PRODUCT' },
]);

Python’s new_url_function gets (session_id, request) and may also be a coroutine:

from crawlee import Request
from crawlee.proxy_configuration import ProxyConfiguration

DATACENTER = 'http://USERNAME:PASSWORD@DC_HOST:DC_PORT'
RESIDENTIAL = 'http://USERNAME:PASSWORD@HOST:PORT'

tiered = ProxyConfiguration(
    tiered_proxy_urls=[
        [None],  # tier 0: no proxy
        [DATACENTER],  # tier 1
        [RESIDENTIAL],  # tier 2
    ],
)


def pick_proxy(session_id: str | None = None, request: Request | None = None) -> str | None:
    if request is not None and request.label == 'PRODUCT':
        return RESIDENTIAL
    return DATACENTER


routed = ProxyConfiguration(new_url_function=pick_proxy)

The function result is not cached per session the way proxyUrls is. If a session must keep one exit, return the same URL for the same sessionId. The current docs on the main branch describe the v4 signature, which takes no arguments, so check which version a snippet was written for.

SOCKS5 in Crawlee

A socks5:// Crawlee proxy URL behaves differently in each SDK and HTTP client.

Node HTTP crawlers, default client. HttpCrawler, CheerioCrawler and JSDOMCrawler send requests through got-scraping by default, whose proxy hook accepts http: and https: proxy URLs only. A socks5:// URL fails with Proxy URL protocol "socks5:" is not supported. Please use HTTP or HTTPS., and because that is not treated as a proxy error, Crawlee retries it three times before giving up.

Node HTTP crawlers with impit. Crawlee 3.x ships a second HTTP client, ImpitHttpClient in the @crawlee/impit-client package, and it took SOCKS5 with a login in the lab (install it with npm install @crawlee/impit-client). With socks5h:// the SOCKS5 server received the target as a hostname; with socks5:// it received an IP address, so the lookup happened on the crawler’s machine.

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
import { ImpitHttpClient } from '@crawlee/impit-client';

const crawler = new CheerioCrawler({
    httpClient: new ImpitHttpClient(),
    proxyConfiguration: new ProxyConfiguration({
        // socks5h:// sends the hostname to the proxy, socks5:// resolves it locally
        proxyUrls: ['socks5h://USERNAME:PASSWORD@HOST:PORT'],
    }),
    async requestHandler({ request, proxyInfo }) {
        console.log(request.url, proxyInfo?.hostname);
    },
});

await crawler.run(['https://example.com/']);

Python. ProxyConfiguration validates every URL as an HTTP URL, so socks5:// and socks5h:// raise a pydantic ValidationError (“URL scheme should be ‘http’ or ‘https’”) before the crawl starts, with either the default ImpitHttpClient or HttpxHttpClient.

Browser crawlers. PlaywrightCrawler accepted socks5://USERNAME:PASSWORD@HOST:PORT through the local proxy-chain bridge. The SOCKS5 server logged a username and password login and the target as a hostname (address type DOMAIN), so DNS resolved on the proxy side.

For Python, and for Node on the default client, the fix is the provider’s HTTP endpoint. With an HTTP proxy and an https:// target, the client sends CONNECT host:443, so the proxy resolves the hostname and the lookup never leaves as a DNS leak, the same property people want from socks5h://. proxymint serves HTTP and SOCKS5 on every tier, and proxymint Lab #1 measured that CONNECT requests through the per-GB gateway resolve on the proxy side. SOCKS5 vs HTTP proxy compares the two protocols in more depth.

Crawlee proxy errors and 407 loops

A wrong password does not fail fast. Crawlee classifies proxy failures (Proxy responded with, ECONNREFUSED, ECONNRESET, ERR_PROXY_CONNECTION_FAILED, ERR_TUNNEL_CONNECTION_FAILED, listed in blocked.ts) as session errors. A session error rotates the session and tries again, up to maxSessionRotations, which defaults to 10, and that limit applies instead of maxRequestRetries.

In the local test, one request with a wrong password produced 11 CONNECT attempts on the default Node client, each answered with 407, and a PlaywrightCrawler with a bad SOCKS5 login looped the same 11 times. Python made 10. Node’s ImpitHttpClient words the error as ProxyAuthRequired, which matches none of those strings, so Crawlee counts it as an ordinary failure: 4 attempts, the maxRequestRetries budget, with no session rotation. With a list of one URL, every new session lands on the same proxy with the same bad password, so each queued URL repeats the cycle.

Message	Crawler	What it means	Fix
Proxy responded with 407 Proxy Authentication Required	Node HTTP crawlers, default client	Proxy refused the login	Check the username and password in the URL, percent-encode special characters
net::ERR_TUNNEL_CONNECTION_FAILED	PlaywrightCrawler	The proxy refused the CONNECT, often a 407 behind the local bridge	Test the same URL with curl, then fix the credentials
impit.ProxyError: Proxy authentication required.	Python, default client	Proxy refused the login	Same as above
ProxyAuthRequired: Proxy authentication required.	Node with ImpitHttpClient	Proxy refused the login	Same as above
connect ECONNREFUSED HOST:PORT	Node, default client	Nothing listens on that host and port	Check the port and protocol from the order
ConnectError: Failed to connect to the server.	Node with ImpitHttpClient	Nothing listens on that port, or a SOCKS5 proxy refused the login	Check the port, then the credentials
Proxy URL protocol "socks5:" is not supported	Node HTTP crawlers, default client	got-scraping takes HTTP proxies only	Use the HTTP endpoint or ImpitHttpClient
URL scheme should be 'http' or 'https'	Python	Validation rejects SOCKS URLs	Use the HTTP endpoint
Detected a session error, rotating session...	Node	A wrapper, the real cause is the next line	Read request.errorMessages

Two settings stop the loop from hiding a configuration error. Lower maxSessionRotations (for example to 2) while testing a new proxy, and log request.errorMessages in failedRequestHandler, because the final message is only the wrapper. The 407 Proxy Authentication Required guide lists the causes on the proxy side, including a package that has run out.

Verify the Crawlee proxy exit

proxyInfo says which proxy Crawlee picked. Only the target knows what arrived, so ask an IP echo endpoint from inside the crawler:

import { HttpCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ['http://USERNAME:PASSWORD@HOST:PORT'],
});

const crawler = new HttpCrawler({
    proxyConfiguration,
    maxConcurrency: 1,
    async requestHandler({ body, proxyInfo }) {
        console.log(proxyInfo?.port, body.toString());
    },
});

// uniqueKey stops Crawlee from deduplicating the same URL
await crawler.run(
    Array.from({ length: 10 }, (_, i) => ({
        url: 'https://api.ipify.org?format=json',
        uniqueKey: `exit-check-${i}`,
    })),
);

Without uniqueKey, the request queue drops repeats of the same URL and the test runs once. Use an https:// echo URL: through the proxymint per-GB gateway, plain http:// requests get a 301 to the https:// address (Lab #1), so start URLs should already be https://.

Tunnels per page. CheerioCrawler opened two CONNECT tunnels per HTTPS page in the test (40 for 20 pages): got-scraping first connects to learn whether the site speaks HTTP/2, then connects again for the request. Node’s ImpitHttpClient opened one tunnel per page (20 for 20). The Python clients reused tunnels instead (20 pages, 10 tunnels with impit and with httpx). On a gateway that gives a new exit per connection, the probe and the page can leave from different IPs on the default Node client, and consecutive pages can share one in Python. Run the ten-request test above against your own order to see which behaviour you get.

Choose the type by the target. Consumer-facing shops, search results by location and marketplaces that score origins fit rotating residential proxies, with country, region and city chosen on the order. Lenient APIs, sitemaps and bulk pages fit datacenter, built for speed and volume. A crawl that logs in and must hold one address for days fits static ISP proxies. Authentication on every tier is username and password, the same URL format shown above.