9 Quick Tips to Reduce Proxy Bandwidth Cost (Comprehensive Guide) in 2026

TL;DR: 

These nine tips cut proxy bandwidth from megabytes to kilobytes per page.

• Switch from headless browser to direct HTTP request – 10-50x less data per page

• Block images, fonts, media, and trackers – 2-10x in headless browsers

• Enable response compression – 60-80% smaller text responses

• Cache unchanged pages – 50-90% fewer requests on stable targets

The remaining five techniques (connection reuse, proxy type, streaming, retry logic, monitoring) compound at scale.

Savings by tips (ordered by typical bandwidth reduction)

Measure your proxy bandwidth first

The gap between what your HTTP client reports (decompressed body) and what your proxy provider bills (compressed wire bytes plus TLS overhead) can be 2-5x. If you optimize against the wrong number, you won't see the savings on your invoice.

At the 90th percentile, pages reach 8-9 MB (HTTP Archive page weight data), and images and JavaScript make up over 70% of that. Those numbers assume compression is working. If a proxy in your chain strips Accept-Encoding without you realizing, wire bytes can be 3-5x larger. The compression section covers how to verify this.

Each tool reports wire bytes differently:

• httpx (Python): response.num_bytes_downloaded – compressed bytes on the wire.

• Playwright: (await request.sizes())["responseBodySize"] – works on Chromium. Check if the values are accurate on your target browser – size reporting varies between browser engines.

• Puppeteer: Use a Chrome DevTools Protocol (CDP) session (page.createCDPSession()) with Network.enable, then sum encodedDataLength from Network.dataReceived events – this is the compressed wire bytes (dataLength is decompressed).

What your proxy provider bills

Your proxy provider measures bandwidth at the network level. Network-level bandwidth includes the response body and several overhead layers: the CONNECT tunnel setup, Proxy-Authorization headers, TLS record framing, and HTTP headers. On sites with heavy analytics or session management, Cookie / Set-Cookie headers alone can add 2-10 KB per request.

On typical HTML responses (50-500 KB), expect the total overhead to be a small percentage of transfer. On small API responses under 5 KB, the overhead percentage is much higher – sometimes exceeding the payload itself.

Most providers show billed bytes in their dashboard or API – compare that number against your client-side measurements to find the gap. If it's larger than expected, check whether the proxy is stripping compression.

If bandwidth is under 10-15% of your total scraping costs, optimizing success rates or reducing compute may save more. But if proxy bandwidth is a large part of your bill – or if you're on per-GB billing – start with the extraction method. It determines whether each page costs 50 KB or 5 MB through the proxy.

Rough cost estimate: (pages/day) × (avg MB/page) × 30 ÷ 1,000 × ($/GB) = monthly bandwidth cost. At 10,000 pages/day and 3 MB/page through $5/GB proxies, that's roughly $4,500/month.

1. Pick the lightest extraction method to reduce bandwidth

Three tiers, lightest to heaviest:

Tier 1: Direct HTTP requests

In Python, httpx works for server-rendered pages when the target doesn't check TLS fingerprints. For sites that check JA3/JA4 fingerprints (TLS handshake signatures that identify your HTTP client – see Section 8), use curl_cffi. In Node.js, the built-in fetch (backed by undici) handles most targets.

Typical bandwidth: 50-200 KB per page.

Tier 2: Find the underlying API

Many sites load data through underlying JSON APIs. Once you identify the endpoint, check whether it requires session cookies, auth tokens, or cross-origin resource sharing (CORS) headers. Replicate those in your client – otherwise the request returns an error or empty response.

Typical bandwidth: 15-50 KB per request.

Tier 3: Headless browser (Playwright, Puppeteer, Selenium)

Typical bandwidth: 1.5-8 MB per page.

Only use headless browsers when the page renders entirely on the client with no accessible API. The other case is when the anti-bot system checks for a real browser environment – canvas fingerprinting, behavioral analysis (see Section 8).

When a site upgrades its anti-bot system, the extraction method you need can change quickly. See Section 6 for how proxy type and extraction method affect cost together.

How to check: curl the target URL. If your data is in the response, use Tier 1. If the body is a JS shell with no data, look for JSON API endpoints (Tier 2). If neither works, use Tier 3.

Scrape from listing pages, not detail pages. A single listing page with 50 products might be 200 KB. Fetching all 50 product pages individually would transfer 10-50 MB.

2. Block non-essential resources to reduce headless browser bandwidth

Blocking images, fonts, stylesheets, media, and tracking scripts reduces headless browser wire bytes – 5-10x on image-heavy e-commerce pages, 2-3x on text-heavy pages with minimal assets. The Playwright page.route() handles this with request interception. Puppeteer has equivalent interception via page.setRequestInterception().

Playwright (Python) (the Node.js API is equivalent – use route.request().resourceType() and route.continue() without the underscore):

async def block_unnecessary(route):
    blocked_types = {"image", "media", "font", "stylesheet"}
    if route.request.resource_type in blocked_types:
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_unnecessary)

async def block_unnecessary(route):
    blocked_types = {"image", "media", "font", "stylesheet"}
    if route.request.resource_type in blocked_types:
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_unnecessary)

async def block_unnecessary(route):
    blocked_types = {"image", "media", "font", "stylesheet"}
    if route.request.resource_type in blocked_types:
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_unnecessary)

Each resource type has different blocking considerations:

Block entire tracking domains too. The following pattern matches both google-analytics.com and subdomains like ssl.google-analytics.com:

from urllib.parse import urlparse

blocked_domains = [
    "google-analytics.com",
    "googletagmanager.com",
    "doubleclick.net",
    "connect.facebook.net",
    "hotjar.com",
    "segment.com",
    "clarity.ms",
    "cdn.cookielaw.org",
]

async def block_unnecessary(route):
    resource_type = route.request.resource_type
    hostname = urlparse(route.request.url).hostname or ""

    if resource_type in {"image", "media", "font", "stylesheet"}:
        await route.abort()
    elif any(hostname == d or hostname.endswith("." + d) for d in blocked_domains):
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_unnecessary)

from urllib.parse import urlparse

blocked_domains = [
    "google-analytics.com",
    "googletagmanager.com",
    "doubleclick.net",
    "connect.facebook.net",
    "hotjar.com",
    "segment.com",
    "clarity.ms",
    "cdn.cookielaw.org",
]

async def block_unnecessary(route):
    resource_type = route.request.resource_type
    hostname = urlparse(route.request.url).hostname or ""

    if resource_type in {"image", "media", "font", "stylesheet"}:
        await route.abort()
    elif any(hostname == d or hostname.endswith("." + d) for d in blocked_domains):
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_unnecessary)

from urllib.parse import urlparse

blocked_domains = [
    "google-analytics.com",
    "googletagmanager.com",
    "doubleclick.net",
    "connect.facebook.net",
    "hotjar.com",
    "segment.com",
    "clarity.ms",
    "cdn.cookielaw.org",
]

async def block_unnecessary(route):
    resource_type = route.request.resource_type
    hostname = urlparse(route.request.url).hostname or ""

    if resource_type in {"image", "media", "font", "stylesheet"}:
        await route.abort()
    elif any(hostname == d or hostname.endswith("." + d) for d in blocked_domains):
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_unnecessary)

page.route() doesn't intercept WebSocket connections or service worker requests – if your target loads data through either channel, you'll need to handle those separately.

To find which resources transfer the most data, log (await request.sizes())["responseBodySize"] per request in your Playwright scripts. Aggregate by domain and resource type to see what to block first (see 'Measure your proxy bandwidth first' for caveats on browser engine differences).

3. Verify that response compression is saving proxy bandwidth

Most HTTP clients send Accept-Encoding and handle decompression automatically. The problem is proxies in forwarding mode – where the proxy terminates and re-originates the request. These proxies can strip the Accept-Encoding header without any client-side error, inflating every text response by 3-5x.

If your proxy provider uses CONNECT tunneling (the default on most modern proxy gateways), the TLS connection is end-to-end. The proxy can't interfere with compression negotiation.

Compare wire bytes against decompressed body size to verify compression:

The most reliable way to check is to compare response.num_bytes_downloaded (the compressed wire bytes) against len(response.content) (the decompressed body). On text-heavy pages, expect a ratio of 0.2-0.4. If the ratio is close to 1:1, compression isn't working – either the server is sending uncompressed data or a proxy is stripping the Accept-Encoding header. Also, check your proxy provider's bandwidth dashboard for the raw transfer size.

Every text-fetching request should include:

Accept-Encoding: gzip, deflate, br, zstd

Accept-Encoding: gzip, deflate, br, zstd

Accept-Encoding: gzip, deflate, br, zstd

Python (httpx):

import httpx

# httpx sends Accept-Encoding automatically and decompresses responses.
#
# IMPORTANT: You need optional packages for some encodings:
#   pip install httpx[http2]    (required for http2=True)
#   pip install httpx[brotli]   (required for Brotli decompression)
#   pip install httpx[zstd]     (required for Zstandard decompression)
#
# httpx only advertises encodings it can decompress. Without httpx[brotli],
# it won't include br in Accept-Encoding, so no silent failures.

proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)
response = client.get("https://example.com")

import httpx

# httpx sends Accept-Encoding automatically and decompresses responses.
#
# IMPORTANT: You need optional packages for some encodings:
#   pip install httpx[http2]    (required for http2=True)
#   pip install httpx[brotli]   (required for Brotli decompression)
#   pip install httpx[zstd]     (required for Zstandard decompression)
#
# httpx only advertises encodings it can decompress. Without httpx[brotli],
# it won't include br in Accept-Encoding, so no silent failures.

proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)
response = client.get("https://example.com")

import httpx

# httpx sends Accept-Encoding automatically and decompresses responses.
#
# IMPORTANT: You need optional packages for some encodings:
#   pip install httpx[http2]    (required for http2=True)
#   pip install httpx[brotli]   (required for Brotli decompression)
#   pip install httpx[zstd]     (required for Zstandard decompression)
#
# httpx only advertises encodings it can decompress. Without httpx[brotli],
# it won't include br in Accept-Encoding, so no silent failures.

proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)
response = client.get("https://example.com")

If you use the requests library instead of httpx: it's HTTP/1.1 only. It supports Brotli and Zstandard if you install the brotli and zstandard packages separately.

4. Reuse connections to reduce proxy bandwidth overhead

Every new connection costs 3-7 KB for the TLS 1.3 handshake alone (depending on certificate chain and extension sizes). The latency hit is larger: 50-150 ms per new connection (200-500+ ms through residential proxy gateways). That slows throughput – fewer requests per hour from the same proxy allocation.

Most HTTP clients reuse connections by default. The main thing to tune is keepalive_expiry: httpx defaults to 5 seconds, which is too short for batched scraping jobs with pauses between pages.

Connection reuse helps most when you hit the same domain repeatedly. If you scrape across hundreds of different domains, you only benefit from reusing the proxy gateway connection itself.

Python (httpx with connection reuse):

import httpx

# Reuses connections across requests to the same host.
# Requires: pip install httpx[http2]
proxy = "http://USER:PASS@IP:PORT"
limits = httpx.Limits(
    max_connections=100,
    max_keepalive_connections=20,
    keepalive_expiry=30  # Default is 5s; longer expiry reuses connections between batched requests
)
client = httpx.Client(proxy=proxy, limits=limits, http2=True)

for url in urls:
    response = client.get(url)

import httpx

# Reuses connections across requests to the same host.
# Requires: pip install httpx[http2]
proxy = "http://USER:PASS@IP:PORT"
limits = httpx.Limits(
    max_connections=100,
    max_keepalive_connections=20,
    keepalive_expiry=30  # Default is 5s; longer expiry reuses connections between batched requests
)
client = httpx.Client(proxy=proxy, limits=limits, http2=True)

for url in urls:
    response = client.get(url)

import httpx

# Reuses connections across requests to the same host.
# Requires: pip install httpx[http2]
proxy = "http://USER:PASS@IP:PORT"
limits = httpx.Limits(
    max_connections=100,
    max_keepalive_connections=20,
    keepalive_expiry=30  # Default is 5s; longer expiry reuses connections between batched requests
)
client = httpx.Client(proxy=proxy, limits=limits, http2=True)

for url in urls:
    response = client.get(url)

For async, use httpx.AsyncClient with the same Limits configuration. Connection reuse behavior is identical.

HTTP/2 multiplexes multiple requests over a single TCP connection. Check whether your proxy provider supports persistent connections (keepalive) and HTTP/2 on their gateway. Without both, you lose most of the benefit of client-side connection reuse. Enable HTTP/2 in your client (for example, http2=True in httpx) once you have confirmed gateway support.

Common misconception: When using HTTPS proxies with CONNECT tunneling, the CONNECT handshake itself usually runs over HTTP/1.1, even if the client-to-target connection inside the tunnel uses HTTP/2. The HTTP/1.1 tunnel means http2=True in httpx enables HTTP/2 for the target connection, not the proxy connection. HTTP/2 CONNECT (where the proxy connection itself is HTTP/2) is possible per RFC 9113 (HTTP/2), but few proxy providers support it yet.

5. Stop wasting bandwidth on pages you already have

On stable targets – product catalogs, real estate listings, and reference pages – 50-90% of pages may not change between daily runs. Four approaches eliminate those redundant downloads.

Use an HTTP caching library

Using hishel (HTTP caching for httpx):

from hishel.httpx import SyncCacheClient
from hishel import SyncSqliteStorage

# hishel adds transparent HTTP caching to httpx.
# pip install hishel[httpx]
storage = SyncSqliteStorage(database_path="/tmp/scrape_cache.db")
client = SyncCacheClient(storage=storage)

# Repeated requests to the same URL are served from cache
response = client.get("https://example.com/products")

from hishel.httpx import SyncCacheClient
from hishel import SyncSqliteStorage

# hishel adds transparent HTTP caching to httpx.
# pip install hishel[httpx]
storage = SyncSqliteStorage(database_path="/tmp/scrape_cache.db")
client = SyncCacheClient(storage=storage)

# Repeated requests to the same URL are served from cache
response = client.get("https://example.com/products")

from hishel.httpx import SyncCacheClient
from hishel import SyncSqliteStorage

# hishel adds transparent HTTP caching to httpx.
# pip install hishel[httpx]
storage = SyncSqliteStorage(database_path="/tmp/scrape_cache.db")
client = SyncCacheClient(storage=storage)

# Repeated requests to the same URL are served from cache
response = client.get("https://example.com/products")

For async, use AsyncCacheClient from the same module. For multi-worker setups or when cache lookups become a bottleneck, a dedicated caching proxy like Squid or the nginx proxy_cache outperforms an in-process cache.

For targets without reliable ETags, set a TTL-based cache expiry. Product catalog pages rarely change more than once a day – a 12-24 hour TTL eliminates most redundant fetches without getting stale data. If your provider uses per-IP pricing (see Section 6), each cache hit frees capacity for new extractions rather than lowering the bill directly.

Use HEAD requests or conditional GETs for change detection

A HEAD request transfers only headers – a few hundred bytes through the proxy – but requires a second round trip when content has changed. A conditional GET achieves the same result in one request. Send the stored ETag in an If-None-Match header. The server returns 304 Not Modified (no body) if unchanged, or 200 with the full body if changed.

import httpx

proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)

# First request: store the ETag
response = client.get("https://example.com/products/123")
etag = response.headers.get("etag")
# If the server doesn't return an ETag, the follow-up request
# falls back to a normal GET (no bandwidth savings on that target).

# Follow-up request: send the ETag back
headers = {"If-None-Match": etag} if etag else {}
response = client.get("https://example.com/products/123", headers=headers)

if response.status_code == 304:
    # Content unchanged, no body transferred
    pass

import httpx

proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)

# First request: store the ETag
response = client.get("https://example.com/products/123")
etag = response.headers.get("etag")
# If the server doesn't return an ETag, the follow-up request
# falls back to a normal GET (no bandwidth savings on that target).

# Follow-up request: send the ETag back
headers = {"If-None-Match": etag} if etag else {}
response = client.get("https://example.com/products/123", headers=headers)

if response.status_code == 304:
    # Content unchanged, no body transferred
    pass

import httpx

proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)

# First request: store the ETag
response = client.get("https://example.com/products/123")
etag = response.headers.get("etag")
# If the server doesn't return an ETag, the follow-up request
# falls back to a normal GET (no bandwidth savings on that target).

# Follow-up request: send the ETag back
headers = {"If-None-Match": etag} if etag else {}
response = client.get("https://example.com/products/123", headers=headers)

if response.status_code == 304:
    # Content unchanged, no body transferred
    pass

Real-world note: High-traffic Shopify Plus stores return strong ETags – conditional requests save the full response body on follow-up fetches. CDN-fronted sites (Cloudflare, Fastly, Akamai) are the most likely to return reliable ETags.

However, many dynamic sites return unstable values (rotating ETags, Last-Modified set to the current time). Verify these headers are stable before depending on them.

If you use Scrapy, its built-in HTTP cache middleware (RFC2616Policy) handles conditional requests automatically. Set HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy" and the cache stores ETags and sends If-None-Match on repeat visits.

Hash extracted data to detect changes client-side

If the server doesn't return reliable ETags or Last-Modified headers, hash the extracted data yourself. On each run, compare the hash against the previous run and skip pages where the content hasn't changed. Hashing also catches duplicate content served under different URLs – pagination variants, query string permutations, and canonicalization differences.

Check sitemaps before re-fetching

Many sites publish XML sitemaps with <lastmod> timestamps. Before re-crawling, fetch the sitemap and only request pages where <lastmod> is newer than your last fetch. Fetching only updated pages turns a full re-crawl into a differential update.

However, <lastmod> is often unreliable – many content management systems (CMSes) set it to the sitemap generation timestamp or never update it. Verify by fetching a sample of pages, hashing their content, and checking whether <lastmod> correlates with actual changes over several crawl cycles.

6. Choose the right proxy type for each target

Your success rate on the actual targets you scrape determines the right proxy type. Not the listed $/GB.

Cost comparison

Some providers charge per GB, some per IP with unlimited bandwidth, some per request. Calculate your cost per successful request on each target.

Two things make this expensive. Residential proxies cost roughly 10x more per GB than datacenter, and a headless browser transfers roughly 10x more data per page than HTTP. In practice, the combined cost is about 25-50x more, because not every request needs both residential proxies and headless browsers. Add an anti-bot bypass service and the total can reach 50-80x.

HypeProxies, for example, uses per-IP pricing with unlimited bandwidth. Under per-IP pricing, bandwidth optimizations mean more requests from the same allocation rather than a lower bill.

Residential proxy prices have dropped 60-75% over the past five years (ScrapeOps industry analysis). Prices are still shifting. In January 2026, Google and Cloudflare disrupted IPIDEA, one of the largest residential proxy networks, removing a major source of residential IPs from the market.

Heavier extraction methods reduce the per-GB savings. According to the same analysis, the share of requests needing residential proxies, JS rendering, or anti-bot bypasses is now nearly 25% – up from under 2%. The cost per successful payload is 2-3x higher on heavily protected verticals like e-commerce and travel.

From the same data, fewer top sites in each vertical can be scraped with datacenter proxies than before:

A datacenter proxy at $1/GB with a 40% success rate on a protected target needs on average 2.5 requests per success. That means 1.5 out of every 2.5 requests are wasted, and the scraper still fails on many requests after retry exhaustion. A residential proxy at $5/GB with 95%+ success costs more per GB but less per successful extraction, because nearly every request returns usable data.

How ISP proxy quality affects block rates

Providers that source IPs directly from ISPs and register them under residential ASNs get lower block rates than those reselling third-party pools.

HypeProxies sources every IP directly from ISPs – no third-party pools. A 7-day test with Camoufox (an anti-detect browser) showed challenge rates of 0.06% on Cloudflare-protected sites, 0.01% on PerimeterX, and 0.02% on DataDome. Results vary by target, request volume, and fingerprint configuration.

The hardest targets (Amazon, Google, major social platforms) reliably block datacenter IPs. Industry surveys confirm the trend. The State of Web Scraping 2026 found 86% of professionals reporting more anti-bot-protected websites year over year.

When evaluating ISP proxy providers, ask how they source IPs: directly from ISPs or through brokers. Run your own block-rate tests on your actual targets – a few hundred requests per target over 24 hours gives a baseline for comparing providers. HypeProxies maintains a ranked comparison of ISP proxy providers if you want a detailed breakdown.

Session strategy by proxy type

How you rotate IPs depends on your proxy type:

• ISP proxies (static IPs): Support sticky sessions natively – no session ID configuration needed.

• Rotating residential proxies: Require session IDs with provider-set TTLs.

For session length, match the rotation to the target's sensitivity:

• Multi-step flows (logins, pagination, checkout): Use sticky sessions to avoid repeating auth steps.

• Moderate-protection targets: Rotate every 5-10 requests or use sticky sessions of 10-30 minutes.

Changing IP on every request when the target doesn't require it adds unnecessary proxy gateway overhead. It can also trigger behavioral detection – the anti-bot system flags one-request-per-IP patterns as bot behavior.

7. Stream responses and abort early to save bandwidth

Streaming only saves bandwidth when your target data appears before the full response is buffered. In practice, the data needs to be in the <head> or early <body>.

import httpx
from io import BytesIO
from selectolax.lexbor import LexborHTMLParser

# Requires: pip install httpx selectolax
# Note: the httpx iter_bytes() method returns decompressed content by default,
# so this works even when the server sends gzip/brotli responses.
# Uses client.stream() for connection reuse (see the connection reuse section).
proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)
MAX_BYTES = 64 * 1024  # Safety limit: stop after 64 KB regardless

with client.stream("GET", "https://example.com/product/123") as response:
    buffer = BytesIO()
    for chunk in response.iter_bytes(chunk_size=8192):
        buffer.write(chunk)
        partial_html = buffer.getvalue().decode("utf-8", errors="replace")

        if "</head>" in partial_html:
            tree = LexborHTMLParser(partial_html)
            title = tree.css_first("title")
            if title:
                # Got the title, stop downloading
                break
if buffer.tell() > MAX_BYTES:
            break

# Downloads ~10-30 KB instead of the full 100-200 KB HTML document

import httpx
from io import BytesIO
from selectolax.lexbor import LexborHTMLParser

# Requires: pip install httpx selectolax
# Note: the httpx iter_bytes() method returns decompressed content by default,
# so this works even when the server sends gzip/brotli responses.
# Uses client.stream() for connection reuse (see the connection reuse section).
proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)
MAX_BYTES = 64 * 1024  # Safety limit: stop after 64 KB regardless

with client.stream("GET", "https://example.com/product/123") as response:
    buffer = BytesIO()
    for chunk in response.iter_bytes(chunk_size=8192):
        buffer.write(chunk)
        partial_html = buffer.getvalue().decode("utf-8", errors="replace")

        if "</head>" in partial_html:
            tree = LexborHTMLParser(partial_html)
            title = tree.css_first("title")
            if title:
                # Got the title, stop downloading
                break
if buffer.tell() > MAX_BYTES:
            break

# Downloads ~10-30 KB instead of the full 100-200 KB HTML document

import httpx
from io import BytesIO
from selectolax.lexbor import LexborHTMLParser

# Requires: pip install httpx selectolax
# Note: the httpx iter_bytes() method returns decompressed content by default,
# so this works even when the server sends gzip/brotli responses.
# Uses client.stream() for connection reuse (see the connection reuse section).
proxy = "http://USER:PASS@IP:PORT"
client = httpx.Client(proxy=proxy, http2=True, follow_redirects=True)
MAX_BYTES = 64 * 1024  # Safety limit: stop after 64 KB regardless

with client.stream("GET", "https://example.com/product/123") as response:
    buffer = BytesIO()
    for chunk in response.iter_bytes(chunk_size=8192):
        buffer.write(chunk)
        partial_html = buffer.getvalue().decode("utf-8", errors="replace")

        if "</head>" in partial_html:
            tree = LexborHTMLParser(partial_html)
            title = tree.css_first("title")
            if title:
                # Got the title, stop downloading
                break
if buffer.tell() > MAX_BYTES:
            break

# Downloads ~10-30 KB instead of the full 100-200 KB HTML document

When you break out of the streaming loop, httpx closes the underlying connection. Over HTTP/2, the close sends a RST_STREAM frame that cancels that single stream without tearing down the TCP connection. Other multiplexed requests on the same connection keep running. Over HTTP/1.1, breaking mid-stream closes the TCP connection, so the client opens a new one.

Useful for <title>, Open Graph tags, and structured data (<script type="application/ld+json">) – anything in the <head>. For data in the <body>, streaming rarely helps – you typically need most of the HTML before your target elements appear.

Request partial responses

Some APIs support partial responses. GraphQL lets you request only the fields you need, and many REST (Representational State Transfer) APIs accept field selection parameters like ?fields=name,price,url.

For static files and media, Range: bytes=0-1023 fetches only the first 1,024 bytes. Check that the server returns 206 Partial Content – most servers ignore Range headers on dynamic HTML.

8. Fix retry logic and request fingerprints

Most retry loops treat all failures identically, wasting bandwidth on retries that fail the same way.

Classify failures before retrying

def should_retry(status_code, attempt, max_attempts=5):
    # Client errors (except 429): don't retry
    if 400 <= status_code < 500 and status_code != 429:
        return False

    # Rate limited: retry after waiting
    if status_code == 429:
        return attempt < max_attempts

    # Server errors: retry with backoff, but cap at 3.
 # Unlike 429 (where the request itself is valid), repeated 5xx
    # usually means a deeper problem that more retries won't fix.
    if status_code >= 500:
        return attempt < 3

    return False

def should_retry(status_code, attempt, max_attempts=5):
    # Client errors (except 429): don't retry
    if 400 <= status_code < 500 and status_code != 429:
        return False

    # Rate limited: retry after waiting
    if status_code == 429:
        return attempt < max_attempts

    # Server errors: retry with backoff, but cap at 3.
 # Unlike 429 (where the request itself is valid), repeated 5xx
    # usually means a deeper problem that more retries won't fix.
    if status_code >= 500:
        return attempt < 3

    return False

def should_retry(status_code, attempt, max_attempts=5):
    # Client errors (except 429): don't retry
    if 400 <= status_code < 500 and status_code != 429:
        return False

    # Rate limited: retry after waiting
    if status_code == 429:
        return attempt < max_attempts

    # Server errors: retry with backoff, but cap at 3.
 # Unlike 429 (where the request itself is valid), repeated 5xx
    # usually means a deeper problem that more retries won't fix.
    if status_code >= 500:
        return attempt < 3

    return False

Retrying a 403 gets the same result – switch proxy type or fix your request fingerprint first. A 401 can mean detection or expired credentials; check which before retrying. On 429, wait for the Retry-After period and don't retry immediately.

For 5xx errors, use exponential backoff with half-jitter (delay = min(base * 2^attempt, max_delay) * random(0.5, 1.0)), capped at 3 attempts. Respect Retry-After headers on 429s. On connection timeouts (ConnectTimeout, ReadTimeout), rotate to a different proxy before retrying – the proxy itself likely failed.

The retry classification handles transient failures (overloaded servers, rate limits). Persistent 403s need a completely different fix.

When you're getting blocked: fix the fingerprint, not only the IP

If you're hitting 403s consistently, the cause is almost always your request fingerprint, not your proxy IP. The anti-bot systems you're most likely hitting – Cloudflare, Akamai, DataDome, and PerimeterX/HUMAN Security – all use fingerprint-based detection as a primary signal.

These systems rely on JA3/JA4 TLS fingerprinting, behavioral analysis, and canvas fingerprinting. Headless browsers in default configurations produce distinguishable fingerprints, which makes them detectable.

Beyond TLS, these systems also check HTTP header order, HTTP/2 connection settings (SETTINGS frame values, window sizes), and navigator properties (navigator.webdriver, navigator.plugins). Rotating IPs without fixing the fingerprint gives you the same 403 from a different address.

IP reputation alone is a weak detection signal compared to fingerprinting. But when your IP is flagged, proxy quality matters more than pool size – clean proxies let you focus entirely on the fingerprint layer.

If you're hitting 403s from TLS fingerprinting, try curl_cffi (Python bindings for curl-impersonate, linked in the extraction method section). It replicates Chrome and Firefox TLS fingerprints, which can bypass detection without a full headless browser. Fixing the fingerprint costs less bandwidth and money than upgrading to residential proxies.

Don't assume managed 'smart proxy' APIs handle fingerprint rotation for you. Test your setup against each target. Check for:

• CDP signals that leak headless mode

• Viewport dimensions no real device uses (for example, 0x0 or 800x1)

• HeadlessChrome in the user agent string

• navigator.webdriver set to true

9. Monitor proxy bandwidth and keep optimizing

A scraper using 50 KB per page can drift to 500 KB without any visible error. Websites add trackers, update anti-bot systems (Cloudflare Radar 2025 Year in Review), change API endpoints, and restructure content without warning. Every optimization in the previous sections breaks down unless you measure continuously.

Key metrics

The most important metric is bytes transferred per successful extraction. Also track:

• Success rate per proxy type per target – catches proxy quality degradation and new anti-bot deployments

• Bandwidth by resource type – reveals new third-party scripts or trackers to block (Section 2)

• Cache hit rate – a sudden drop means the target changed its URL structure, ETag behavior, or caching headers

• Retry rate – high retry rates mean most of the bandwidth goes to retries (Section 8)

• Gap between client-reported response size and proxy-billed bytes – catches compression stripping (Section 3)

Set up alerts

Alert on these:

• Bytes per request exceeds baseline by 30%+ – a site added resources or compression broke

• Success rate drops below 90% (adjust per target) – new anti-bot deployment or proxy quality change

• Cache hit rate drops below 40% on targets that previously cached well – investigate target-side changes

Monthly review

If you scrape 100 sites, expect to fix a few broken scrapers each month.

Each month, check:

• Whether API endpoints changed, forcing fallback to full page loads

• Whether any proxy types are underperforming on specific targets

• Whether TLS fingerprint changes are causing undetected failures

Watch for hidden costs

Some bandwidth increases happen without triggering visible errors. These increases are harder to catch than outright blocks:

• Response size inflation: A site adds new scripts or trackers. Detect by tracking median bytes per request per target.

• Data fragmentation: Content that used to be on one page gets split across pagination, overlays, or separate API calls. Detect by tracking the number of requests needed to extract the same data points.

• Soft bans (cloaking): The site returns 200 OK but with empty, truncated, or fake data. Your scraper thinks it worked. Detect by validating extracted field counts and value distributions against a known-good baseline.

For example, Google dropped support for the num=100 parameter in September 2025. Each request now returns ~10 results instead of 100 – a 10x increase in requests for the same data.

Responsible scraping

These techniques assume you're scraping publicly available data and not overloading the sites you target. Lighter scraping is less likely to trigger anti-bot escalation.

If a website explicitly prohibits automated access, respect that. Aggressive scraping that ignores access restrictions makes anti-bot protections stricter for everyone. HypeProxies enforces rate limits and access policies – accounts that violate target sites' terms of service are suspended.

Next steps to reduce your proxy bandwidth

You don't need to implement all nine at once. The extraction method (Section 1) makes the biggest difference. Moving a target from headless browser to HTTP request reduces bandwidth 10-50x. After that, compression (Section 3) and caching (Section 5) remove most of the remaining waste.

• Doing it yourself? Measure your baseline with the cost formula in "Measure your proxy bandwidth first". Pick the most effective technique for your setup and compare bytes per successful extraction before and after.

• Hitting block rates or overpaying per GB? HypeProxies ISP proxies – per-IP pricing, unlimited bandwidth.

• Want fully managed scraping? The Crawlbyte scraping API picks the extraction method, handles CAPTCHAs and anti-bot bypasses, and returns structured data – pay per successful extraction.

Frequently asked questions

How much bandwidth does Playwright or Puppeteer use per page?

1.5-8 MB per full page load, depending on images, scripts, and third-party resources (HTTP Archive). A direct HTTP request to the same page transfers 50-200 KB. At 10,000 pages/day, that's roughly 50 GB vs 2 GB – a 25x difference.

Why did my proxy costs spike suddenly?

Three common causes: a proxy update stripped Accept-Encoding headers, inflating every response 3-5x with no client-side error. A drop in success rate, causing more retries that waste bandwidth on data you never get. Or target sites added new resources that your blocking rules don't catch yet.

Do ISP proxies use less bandwidth than residential proxies?

Response size is the same regardless of proxy type. The bandwidth difference comes from success rates – best ISP Proxies fail less on protected targets, which means fewer retries. Each failed request wastes the full request and response bandwidth while returning no usable data.

Do connection reuse and caching work with rotating proxies?

Yes. Connection reuse keeps the TCP connection to the proxy gateway open regardless of which outbound IP the provider assigns. HTTP caching and conditional GET work at the URL level and are unaffected by IP rotation. Only session-based scraping (logins, pagination) requires static IPs.

What's cheaper at scale: per-GB or per-IP proxy pricing?

Per-GB pricing is simpler at low volume. At scale, per-GB costs become unpredictable – heavier pages or increased retries increase costs without warning. Per-IP pricing with unlimited bandwidth fixes costs. Bandwidth optimizations still help under per-IP: less data per request means faster extraction and higher concurrency.