Web scraping

Guides

Crawlee proxy setup: rotation, sessions, tiers and SOCKS5

Crawlee proxy setup for Node and Python: ProxyConfiguration, how sessions pin proxies, the browser trap where proxyInfo lies, tiers, SOCKS5 and 407 loops.13 min read
Guides

Privacy Pass tokens explained: roles, flow and proxies

Privacy Pass tokens let a client prove it passed a check without saying who it is. How issuance and redemption work, where tokens run, what proxies change.11 min read
News

PostgreSQL 18.7 security release: 52 CVEs, check your version

PostgreSQL 18.7, 17.12, 16.16, 15.20 and 14.25 fix 52 CVEs, including CVE-2026-6460 (CVSS 8.8). Check your version, your logins, and plan off 14.6 min read
News

Apache HTTP/2 RCE CVE-2026-119618: check your server

Apache HTTP/2 vulnerability CVE-2026-119618 is a critical RCE in mod_http2, fixed in 2.4.71. A one-minute check for version, module and h2/h2c.6 min read
Guides

Residential proxy botnets: how they work and how to spot one

A residential proxy botnet rents out hijacked routers, TV boxes and phones as proxy exits. How devices get enrolled, the 2024-2026 cases, and how to check.10 min read
News

Chrome XML zero-day CVE-2026-125350: patch headless too

Chrome XML zero-day CVE-2026-125350 is fixed in 157.0.8103.50. Auto-update skips headless Chrome in Playwright, Puppeteer and Docker. How to check yours.5 min read
Guides

Anubis proof of work: how the AI scraper challenge works

How Anubis proof of work runs: the SHA-256 puzzle, the week-long cookie bound to your IP, weights by country and ASN, its limits, and where proxies fit.11 min read
Guides

HTTP/2 fingerprinting: the frames and what a proxy changes

HTTP2 fingerprinting reads the SETTINGS, WINDOW_UPDATE, PRIORITY and pseudo-header order a client sends first. Real values from Chrome, Firefox and Go.8 min read
News

robots.txt AI training opt out: one token, four companies

One robots.txt AI training token asks OpenAI, Google, Microsoft and Anthropic not to train on your site. How to add it, when each honours it, how to test it.5 min read
Guides

HTTP CONNECT method: how HTTPS gets through a proxy

The HTTP CONNECT method turns a proxy into a blind tunnel. The request, the 200 reply, what the proxy can and cannot see, why tunnels fail, and HTTP/3.13 min read
Guides

CDP detection: how sites spot Chrome DevTools automation

How CDP detection spots Puppeteer, Playwright and Selenium through the Chrome DevTools Protocol, what the 2025 V8 change broke, and what a proxy changes.9 min read
News

Claude Code email in User-Agent: what happened, how to check

Claude Code's model put signed-in users' emails in User-Agent headers. What Anthropic confirmed, what is fixed, and how to check what your clients send.5 min read
Guides

Block AI crawlers: robots.txt, server rules and verification

How to block AI crawlers: which bots train and which cite you, copy-paste robots.txt and server rules, bot verification, and what to do about bots that lie.13 min read
Guides

Pay per crawl: how HTTP 402 lets sites charge AI crawlers

How Cloudflare pay per crawl uses HTTP 402, the crawler-price headers and Web Bot Auth, what sites and crawler operators set up, and where proxies fit.10 min read
News

sshole OpenSSH vulnerability: check Alpine and OpenWrt

sshole lets long passwords into OpenSSH on musl without PAM. Who is affected, the fixed Alpine and OpenWrt packages, and how to check your logs.5 min read
Guides

502 Bad Gateway proxy error: find who sent it, then fix it

A 502 through a proxy comes from the proxy or from the target's own gateway. How to tell them apart in one command, what causes each, and when to retry.11 min read
Guides

Proxy timeout: connect, read or 504, and which side to fix

A proxy timeout names the phase that stalled: connect, the CONNECT tunnel, TLS or first byte. Read the error, time each phase with curl, fix the right side.15 min read
Guides

Web Bot Auth explained: signed requests, keys and proxies

Web Bot Auth lets bots sign HTTP requests with a published key. How the three headers work, who verifies them in 2026, and what a proxy changes.11 min read
Guides

Proxy rotation strategy: sessions, pace and retries

A proxy rotation strategy decides when a job changes exit IP, how fast it hits each target and which failures it retries, with a failure table and Python code.10 min read
Guides

ERR_TUNNEL_CONNECTION_FAILED: what it means, how to fix it

ERR_TUNNEL_CONNECTION_FAILED means a proxy would not open the HTTPS tunnel. How to see the status Chrome hides, and the fix for each one, in Chrome and code.10 min read
Guides

Proxies for market research: seeing each market as locals do

Which market research jobs need residential exits, where datacenter is enough, how to design a multi-market collection, and a sized 5-country weekly run.11 min read
Guides

Web scraping architecture: components, patterns and proxies

Web scraping architecture splits a scraper into scheduler, fetcher, parser, storage and monitoring layers. How it changes with scale, and where proxies belong.12 min read
Guides

Backconnect proxy: one gateway, many exit IPs, how to test

A backconnect proxy is one gateway address that routes your traffic to exit IPs from a pool. Gateway vs exit, rotation, proxy lists, and how to test one.11 min read
Guides

How many proxies do I need? A sizing formula by job

How many proxies you need depends on the job: requests per hour, concurrent sessions, accounts or GB. Formulas, a pilot test and a worked example for each job.10 min read
Best Practices

Cheap datacenter proxies: billing, sizing and where they fit

Cheap datacenter proxies are billed per IP or per GB, but value is set by how many requests return data. How to measure it, size an order and pick the tier.10 min read
Guides

Proxy error codes: which hop sent it and how to fix it

Proxy error codes decoded by hop: which ones the proxy sends, which come from the target, how to read them with curl -v, and which ones are safe to retry.14 min read
Guides

How to choose a residential proxy provider: a buyer's checklist

How to choose a residential proxy provider: sourcing and consent, pool depth where you need it, sessions, geo accuracy, real cost per GB and a test plan.9 min read
Best Practices

Residential proxy pricing: what a GB buys for your job

Residential proxy pricing per GB explained: what a GB buys in pages, traffic per 1,000 pages, expiry and run-outs, and when another tier fits the job better.10 min read
Guides

IPv6 proxies: when they work and where they fail

IPv6 proxies reach only targets that serve IPv6, and many rate limiters count a whole /64 as one client. How to check both, and when IPv4 or residential fits.9 min read
Guides

Python requests proxy: auth, sessions, SOCKS5 and rotation

Set a proxy in Python requests the right way: the proxies dict and its https key, auth encoding, sessions, SOCKS5, rotation, retries and the silent bypass.14 min read
Comparisons

Best language for web scraping: Python, Node, Go and Java compared

Python, Node, Go and Java compared for scraping by parsing tools, concurrency, headless browsers and proxy support, with a verified proxy snippet for each.12 min read
Best Practices

Proxy performance tuning: concurrency, reuse and timeouts

Proxy performance is mostly decided on the client. Tune connection reuse, concurrency and timeouts in order, then measure each phase with curl.12 min read
Guides

Are free proxies safe? What a free list actually delivers

What studies of free proxy lists found: dead proxies, leaked IPs, injected scripts and swapped certificates, what HTTPS hides, and when free is fine.13 min read
Guides

Scrape Google search results: API, parsing, pacing

How to scrape Google search results in 2026: the Custom Search JSON API and its limits, what Google's terms say, parsing, pacing and setting location.14 min read
Guides

Proxies for AI agents: giving an agent reliable web access

Proxies for AI agents give a cloud-hosted agent a home or ISP origin. Which jobs need one, which type and rotation fit, and how to wire Playwright and MCP.13 min read
Guides

Proxies for AI training data: throughput, sizing and rules

How to size proxies for AI training data: why datacenter ports carry most of the crawl, where residential fits, opt-out rules and a sized 10M page job.12 min read
Guides

OkHttp proxy: auth, SOCKS5, rotation and 407 fixes

OkHttp proxy setup in Kotlin and Java for OkHttp 4.12 and 5.x: proxyAuthenticator done right, SOCKS5 logins, the system proxy, rotation and the 407 traps.15 min read
Guides

PHP cURL proxy: CURLOPT_PROXY, auth, SOCKS5 and errors

PHP cURL proxy setup that works: CURLOPT_PROXY and PROXYUSERPWD, special characters, SOCKS5 and socks5h, env variables, reuse, rotation and the 407 errors.9 min read
Guides

C# HttpClient proxy: WebProxy, auth, SOCKS5 and rotation

C# HttpClient proxy setup for .NET 8 and 10: WebProxy credentials, SOCKS5, environment proxies, IHttpClientFactory, rotation and the 407 and https:// traps.14 min read
Best Practices

Geo targeted proxies: country, region or city level

Geo targeted proxies filter exits by country, region or city. How accurate each level is, why city pools run thin, and when country targeting is enough.11 min read
Guides

Java proxy setup: HttpClient, Authenticator and SOCKS5

Java proxy setup that works: system properties, java.net.Proxy, HttpClient, a scoped Authenticator, the 407 fix for Basic over CONNECT, SOCKS5, OkHttp, Apache.13 min read
Best Practices

Proxy DNS leak test: check and fix it in your own client

A proxy DNS leak test you run through your own client: where curl, requests, httpx, Playwright and Node resolve names, what a leak exposes, and each fix.12 min read
Guides

How to scrape Google Maps: Places API first, then scraping

How to scrape Google Maps: how the Places API is billed now, what Google's terms allow, and a scraping method for public business data with city-level proxies.14 min read
Guides

USA datacenter proxies: which US targets accept them

USA datacenter proxies are fast US server IPs. Which US targets accept hosting networks, which filter them, how to test one exit and when another tier fits.10 min read
Guides

How to scrape Amazon: Python, the law and official APIs

Scrape public Amazon product and price data with Python: what the terms and courts say, the official API routes, polite rate limits and where proxies fit.16 min read
Guides

Germany proxies: targeting, sessions and exit checks

Germany proxies give a German exit IP. Which proxy type fits, how state and city targeting works, why DE is the code, and how to check the exit before a job.12 min read
Guides

407 Proxy Authentication Required: causes and client fixes

A 407 means the proxy, not the site, rejected or never got your login. Causes in the order to check, and the fix for curl, Python, Node, browsers and apt.15 min read
Guides

UK residential proxies: regions, cities and sessions

UK residential proxies route through British home broadband. How UK targeting works, why GB is the country code, which session fits and how to check the exit.10 min read
Guides

USA residential proxies: state, city and session guide

USA residential proxies route through US home connections. How state and city targeting works, which session type fits, and how to check the exit.9 min read
Comparisons

SOCKS5 vs HTTP proxy: DNS, speed, security, which to use

SOCKS5 vs HTTP proxy compared: what each carries, where DNS resolves (socks5 vs socks5h), speed, auth, client support, and which one your job needs.11 min read
Guides

Proxy format: IP, port, username and password, every form

Read and write proxy strings: host:port:user:pass, user:pass@host:port, http:// vs socks5h:// URLs, separate fields, password encoding and IPv6 brackets.12 min read
Comparisons

Rotating vs sticky proxies: what breaks under each mode

Rotating vs sticky proxies by job: per-request IPs, 5-60 minute timers or sticky while online, what breaks under the wrong mode, and a test to run.10 min read
Guides

Proxy provider security review: 6 checks before you buy

A proxy provider security review for privacy-first buyers: terms, abuse handling, credentials, measured traffic, logs, IP sourcing and what certificates prove.11 min read
Guides

Pay as you go proxies: per-GB billing, no subscription

Pay as you go proxies bill per GB with no renewal. How to estimate GB, how per-GB rates scale, whether GB expire, and when a per-IP subscription fits better.7 min read
Guides

What is a proxy server? How it works and when to use one

What is a proxy server: how a request travels through one, forward vs reverse, what it hides and what it does not, proxy vs VPN, the risks and what it costs.11 min read
Guides

Switching proxy providers without downtime: a checklist

Switching proxy providers without breaking jobs: what to inventory, how to map credentials and rotation, the parallel test to run, and when switching fails.12 min read
Guides

Proxy acceptable use policy: what providers ban and why

What a proxy acceptable use policy bans, which targets get blocked, how KYC and enforcement work, what a breach costs, and what to read before you pay.10 min read
Guides

What are datacenter proxies? Uses, limits and pricing

Datacenter proxies use IPs owned by hosting companies. What they are used for, why strict sites block them, how to test a target, and IPv4 port pricing.11 min read
Guides

Best proxies for web scraping: match the proxy to the job

The best proxies for web scraping depend on the target. Datacenter, residential, ISP and mobile compared by job, with billing models and a test to run first.13 min read
Guides

What is a residential proxy? How it works and when you need one

A residential proxy sends traffic out through a home ISP connection. How it works, where the IPs come from, whether it is legal, and what it costs per GB.11 min read