What Beautiful Soup does
Beautiful Soup 4, commonly called BS4, turns a markup document into a tree of Python objects. You can navigate parent and child elements, search by tag or attribute and extract text, links and other values.
It is a data parsing library, not a downloader or crawler. Another tool must fetch each page before Beautiful Soup can process it. It can also modify markup, though extraction and cleanup are more common in web scraping.
How Beautiful Soup parses HTML
You choose a parser when creating the BeautifulSoup object. Python’s built-in html.parser has no extra dependency, while parsers such as lxml can offer different performance and XML support. Results can differ when the source contains malformed markup.
After parsing, methods such as find(), find_all() and select() locate elements. select() accepts a CSS selector, such as .product .price or article > h2.
Using Beautiful Soup with a proxy
Beautiful Soup has no network or proxy settings. Configure the HTTP client that downloads the page to use a proxy server, then pass the returned HTML to Beautiful Soup.
import requests
from bs4 import BeautifulSoup
proxy = "http://USERNAME:PASSWORD@HOST:PORT"
response = requests.get(
"https://example.com",
proxies={"http": proxy, "https": proxy},
timeout=20,
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
print(soup.select_one("h1").get_text(strip=True))Set timeouts and handle retries in the HTTP client. A proxy changes the source IP seen by the site, but it doesn’t replace responsible request pacing. Follow the site’s terms, robots.txt rules and rate limits.
Beautiful Soup vs other scraping tools
| Tool | Main function | Best fit |
|---|---|---|
| Beautiful Soup | Parses existing HTML or XML | Small extraction scripts |
| Scrapy | Crawls, schedules and parses | Multi-page crawling |
| Playwright | Controls a browser | Interactive or dynamic pages |
Beautiful Soup doesn’t run scripts or perform JavaScript rendering. If data appears only after scripts execute, use a browser tool and pass its rendered HTML to Beautiful Soup. Choose Scrapy when you need built-in request queues, concurrency and retry handling across many pages.
FAQ
Is Beautiful Soup a web scraper?
Not by itself. Beautiful Soup parses and searches markup, while an HTTP client or crawling framework fetches the pages.
Can Beautiful Soup scrape JavaScript websites?
Only if the needed data is present in the initial HTML. For content created by JavaScript, render the page with a browser tool such as Playwright or find the site API that supplies the data.
Can Beautiful Soup use a proxy?
Yes, but the proxy is configured in the downloading tool, not in Beautiful Soup. Pass the downloaded HTML to Beautiful Soup after the request succeeds.