Rotating proxies with a list that updates itself all day

Rotation at the list level

ProxySale rotates proxies through the list itself. The pool loses dead addresses and gains new ones all day, and each time your software requests the list link, it gets the set that works at that moment. You set no rotation rules on our side: reload the link every 8 to 10 minutes to keep the list fresh.

Your software then picks the line for each request. Most scrapers and SEO tools choose a random proxy per request or switch after an error, and the load spreads over thousands of addresses.

Why rotating proxies help scrapers

Search engines, marketplaces and price aggregators count requests per IP. One address that sends hundreds of requests a minute runs into a captcha or a block. Spread the same load over a rotating proxy list, and each address sends only a few, so the target sees something close to ordinary visitors.

Rotation also hides dead lines. A failed request goes out again through another address, and the next reload of the list drops the bad line.

Setting up rotation in your software

• A-Parser and ZennoPoster: add the list link as a proxy source with a 10-minute reload.
• Scrapy: load the list into a downloader middleware that picks a random line per request.
• Python requests: keep the list in memory and pass a random line in the proxies dictionary.
• curl in bash: take a line with shuf -n 1 before each call.

Leave the reload at 8 to 10 minutes. We throttle requests that come every minute, and the pool does not change that fast anyway.

Rotation strategies

• A random line per request: the simplest scheme with the most even load.
• One line for a short series: useful when a site expects a few related requests, such as a search page and the next page of results.
• Switch on error: keep a line while it answers and drop it after a captcha, a timeout or a 403.
• A cool-down: after a block, keep the address out of rotation for 10 to 15 minutes.

Most scrapers combine the first and the third: a random line per request and an instant switch after an error. Watch the share of captchas and failed requests. If it grows, add addresses or slow down.

How many addresses you need

Count the requests per minute to your hardest target and divide them by what one address can send without trouble. If a site tolerates five requests a minute from one IP and you need 5,000, plan for at least 1,000 live addresses, plus headroom for the night dip. Bigger jobs are covered on the bulk proxies page.

Pick the pool by the target: World MIX for global sites, Europe proxies for EU shops and search, CIS proxies for Yandex and regional marketplaces, China proxies for Baidu.

Tasks that need a fixed address

Some work needs one address for a long time: logging into accounts, keeping a shopping cart, running antidetect browser profiles. A rotating package list fits poorly here, because the address can leave the pool. For such work take residential proxies on the Max pool: each line runs its own session, lasting anywhere from three minutes to a day while the home device is online. The anti-detect proxies page shows how to set up profiles.

For anything that reads public pages, rotation fits better and costs less per request.

A list file that refreshes every 10 minutes

Some tools read proxies only from a file. Keep that file fresh with a small script that downloads the list every 10 minutes. Each script below writes to a temporary file first and replaces the main one only after a complete download, so your software never reads a half-written or empty list. Put the list link from the control panel in place of YOUR_LIST_LINK.

Linux: a shell script and cron

Save the script as /opt/proxies/update-proxies.sh, make it executable with chmod +x /opt/proxies/update-proxies.sh and run it once by hand. It prints how many proxies it saved, or an error if the link does not answer.

#!/bin/sh
LIST_URL="YOUR_LIST_LINK"
DEST="/opt/proxies/proxies.txt"
TMP="$DEST.tmp"

mkdir -p "$(dirname "$DEST")"
if curl -fsS --max-time 60 -o "$TMP" "$LIST_URL" && [ -s "$TMP" ]; then
  mv "$TMP" "$DEST"
  echo "$(date '+%F %T') saved $(wc -l < "$DEST") proxies"
else
  rm -f "$TMP"
  echo "$(date '+%F %T') download failed, kept the old list" >&2
  exit 1
fi

Open the schedule with crontab -e and add the line below. Cron starts the script every 10 minutes and appends its messages to a log. Run it as root, or change both paths to a folder in your home directory.

*/10 * * * * /opt/proxies/update-proxies.sh >> /var/log/update-proxies.log 2>&1

Windows: PowerShell and Task Scheduler

Save the script as C:\proxies\update-proxies.ps1 and run it once in PowerShell to check the link.

$ListUrl = "YOUR_LIST_LINK"
$Dest = "C:\proxies\proxies.txt"
$Tmp = "$Dest.tmp"

New-Item -ItemType Directory -Force -Path (Split-Path $Dest) | Out-Null
try {
    Invoke-WebRequest -Uri $ListUrl -OutFile $Tmp -UseBasicParsing -TimeoutSec 60
    if ((Get-Item $Tmp).Length -eq 0) { throw "empty list" }
    Move-Item -Force $Tmp $Dest
    $count = (Get-Content $Dest | Measure-Object -Line).Lines
    Write-Output "$(Get-Date -Format s) saved $count proxies"
} catch {
    Remove-Item $Tmp -ErrorAction SilentlyContinue
    Write-Output "$(Get-Date -Format s) download failed, kept the old list: $($_.Exception.Message)"
    exit 1
}

Register a task that runs the script every 10 minutes: paste the command into Command Prompt or PowerShell. The task survives reboots and shows up in Task Scheduler as UpdateProxies. It runs while you are logged in; to run it without a session, open the prompt as administrator and add /RU SYSTEM to the command.

schtasks /Create /TN "UpdateProxies" /SC MINUTE /MO 10 /F /TR "powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:\proxies\update-proxies.ps1"

Prefer the mouse? Open Task Scheduler, choose Create Basic Task, pick a daily trigger, then open the task properties and set Repeat task every 10 minutes for a duration of Indefinitely. The action starts powershell.exe with the arguments -NoProfile -ExecutionPolicy Bypass -File C:\proxies\update-proxies.ps1.

Any system: Python

The Python version runs in a loop and needs no scheduler. Start it in a terminal, inside screen or tmux, or as a service. It needs Python 3 and no extra packages.

import time
import urllib.request
from pathlib import Path

LIST_URL = "YOUR_LIST_LINK"
DEST = Path("proxies.txt")
EVERY = 600

while True:
    try:
        data = urllib.request.urlopen(LIST_URL, timeout=60).read()
        if not data.strip():
            raise ValueError("empty list")
        tmp = DEST.with_name(DEST.name + ".tmp")
        tmp.write_bytes(data)
        tmp.replace(DEST)
        print(time.strftime("%Y-%m-%dT%H:%M:%S"), "saved", len(data.split()), "proxies", flush=True)
    except Exception as error:
        print(time.strftime("%Y-%m-%dT%H:%M:%S"), "download failed, kept the old list:", error, flush=True)
    time.sleep(EVERY)

Point your software at the saved file, and it always reads a list no older than 10 minutes. Keep the interval at 10 minutes: we throttle more frequent requests.

Protocols, prices and test

Each port of a rotating package takes HTTP, HTTPS, SOCKS4 and SOCKS5 alike, see SOCKS5 proxies and HTTP proxies. Sizes and periods sit on the price list, and support gives a free hour of any package on the contact page.