VPS Web Scraping

Serious scraping doesn't happen on a laptop

A collection script running on your laptop stops when you close the lid, shares your personal IP with your targets, and dies at the first Wi-Fi drop. Professional scraping demands a machine on 24/7 with a stable dedicated IP: that's the definition of a VPS.

The Titan is our recommendation: 8 Ryzen 9 vCores at 5.7 GHz to run several Playwright workers in parallel, 16 GB of RAM because headless Chromium eats 400 MB per tab. If your pipeline is pure HTTP (Scrapy, httpx), an Apex is plenty.

The dedicated IP: your most precious asset

With shared hosts, you inherit a shared IP whose history you don't know — sometimes already blacklisted by Cloudflare. Here, every VPS gets a clean dedicated IPv4, and our abuse team checks its reputation before assignment. For respectful scraping, a stable IP with configured reverse DNS gets through far better than rotating shady IPs.

Skeleton of a robust pipeline

# minimal docker-compose.yml
services:
  scraper:
    build: .
    restart: unless-stopped
    environment:
      - REDIS_URL=redis://redis:6379
  redis:
    image: redis:7
  postgres:
    image: postgres:16
    volumes:
      - pgdata:/var/lib/postgresql/data

Architecture that holds up: Redis queue for URLs to visit, stateless workers pulling from it, PostgreSQL for results, and a scheduler (cron or APScheduler) feeding the queue. Each worker in its own container: a Chromium crash doesn't kill the pipeline.

Ethics and legality: the section competitors avoid

Scraping public data is legal in most European jurisdictions, under three conditions: respect robots.txt, don't circumvent technical protection measures, and don't collect personal data beyond what's necessary (GDPR). Our terms of service require it too: scraping that ignores robots.txt or saturates a target site leads to account suspension. Set your DOWNLOAD_DELAY — Scrapy does it natively with AutoThrottle.

Monitoring: know when it breaks

A scraper dies silently: the target changes its HTML, adds a captcha, or rate-limits you. Set up metrics (items collected/hour) and an alert when throughput drops 50%. Our article on VPS monitoring covers the Prometheus + Grafana stack that fits on the same Titan.