Scrapy

How Scrapy works

A Scrapy project defines one or more spiders. Each spider yields requests, processes responses in callback methods and extracts fields with CSS selectors or XPath. It can also follow matching links, which lets one spider act as a web crawler.

Scrapy schedules and downloads many pages concurrently without starting a browser for each one. Extracted items can pass through pipelines that validate values, remove duplicates and write results to JSON, CSV or a database.

Sending Scrapy through a proxy

Scrapy’s built-in proxy middleware reads the proxy value from each request’s metadata. This example sends the request through an authenticated proxy and extracts the page title:

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={"proxy": "http://USERNAME:PASSWORD@HOST:PORT"},
        )

    def parse(self, response):
        yield {"title": response.css("title::text").get()}

Keep credentials outside source code in production. For a proxy pool, middleware can select a different proxy for each independent request. Keep the same proxy and cookies when several requests belong to one session. A proxy changes the source IP, but it doesn’t correct unrealistic headers or excessive request rates.

Scrapy vs Beautiful Soup vs Selenium

These tools solve different parts of data collection.

Beautiful Soup can be used inside Scrapy, though Scrapy’s selectors often cover the same parsing tasks. Selenium uses more memory because it controls a browser, but it can handle pages that require clicks or JavaScript execution.

Limits and responsible crawling

Scrapy doesn’t execute page JavaScript by default. If the data loads later, inspect the site’s public API or use a browser integration when its terms permit automation.

Control load with CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY and AUTOTHROTTLE_ENABLED. If a site returns 429 Too Many Requests, slow down instead of rotating IPs and retrying immediately. Respect robots.txt, access controls and the site’s terms.

FAQ

Is Scrapy better than Beautiful Soup?

It depends. Scrapy is better suited to complete crawling projects, while Beautiful Soup is simpler when you only need to parse HTML that another tool fetched.

Can Scrapy scrape JavaScript websites?

No, not by itself. Scrapy downloads HTTP responses but doesn’t run JavaScript, so dynamic pages may require an underlying JSON endpoint or a browser automation tool.

Does Scrapy support authenticated proxies?

Yes. Put http://USERNAME:PASSWORD@HOST:PORT in the request’s proxy metadata. Your proxy sends 407 Proxy Authentication Required when the request lacks valid proxy credentials.