What Selenium is used for
Selenium WebDriver is the component most scripts use. It sends commands to Chrome, Firefox, Edge or Safari through each browser’s driver. Selenium IDE records simple browser actions, while Selenium Grid runs sessions across several machines.
Common uses include:
- Testing forms, navigation and other browser workflows.
- Collecting content that needs JavaScript rendering.
- Taking screenshots or checking pages across browsers.
Because Selenium controls a full browser, it can click buttons, enter text, manage cookies and inspect the rendered document.
How Selenium WebDriver works
Language bindings translate Python, Java, C# or JavaScript calls into WebDriver commands. The browser performs each action and returns element data, page state or errors. Modern Selenium versions can usually locate or download a compatible browser driver automatically.
This Python example opens a page and waits for an element found with a CSS selector:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://example.com")
heading = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.CSS_SELECTOR, "h1"))
)
print(heading.text)
driver.quit()Explicit waits are safer than fixed sleeps because they continue as soon as the expected condition is met.
Sending Selenium through a proxy
For a proxy that doesn’t require credentials, pass its host and port to Chrome before creating the WebDriver session:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=http://HOST:PORT")
driver = webdriver.Chrome(options=options)Start a new browser session when you need different proxy settings. Firefox uses its own preference-based configuration rather than Chrome’s command-line switch.
An authenticated endpoint is commonly written as http://USERNAME:PASSWORD@HOST:PORT. WebDriver’s standard proxy settings have no credential fields for HTTP proxies, and Chrome often won’t authenticate when credentials are placed in its proxy switch. Use a browser extension, configured profile or local proxy adapter that handles proxy authentication. An HTTP proxy usually returns 407 Proxy Authentication Required when credentials are missing or rejected.
Selenium vs Playwright vs Scrapy
| Tool | Best fit | Main trade-off |
|---|---|---|
| Selenium | Cross-browser UI testing | Browser setup and overhead |
| Playwright | Modern browser automation | Different API and tooling |
| Scrapy | Large HTTP crawls | No JavaScript rendering by default |
Selenium uses more memory and CPU than a direct HTTP client. Use it when a page requires browser rendering or interaction. Keep request rates within the site’s limits and follow its robots.txt rules and terms.
FAQ
Is Selenium good for web scraping?
Yes, especially when content appears after JavaScript runs or requires clicks and scrolling. A direct HTTP client is usually faster for static HTML or APIs.
Can Selenium use an authenticated proxy?
Yes, but authentication often needs a browser extension, configured profile or local adapter. Standard WebDriver proxy settings define no credential fields for HTTP proxies.
Does headless Selenium avoid bot detection?
No. Headless mode hides the browser window, but sites can still examine WebDriver signals, fingerprints and behavior. A proxy changes the IP address the site sees, not those other signals.