Puppeteer

What Puppeteer does

Puppeteer is a Node.js API for automating Chrome and Firefox. It can run a browser without a visible window or show the window while you debug. Unlike a basic HTTP client, it executes page scripts and exposes the rendered DOM.

Common uses include:

• Extracting content that appears after JavaScript rendering.
• Clicking buttons, typing into forms and following redirects.
• Running browser-based tests and taking screenshots or PDFs.
• Reading network responses, cookies and performance timings.

How Puppeteer works

A script launches or connects to a browser, opens a page and sends navigation or input commands. It can then read elements with a CSS selector after the required content appears.

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://HOST:PORT']
});
const page = await browser.newPage();
await page.authenticate({ username: 'USERNAME', password: 'PASSWORD' });
await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
console.log(await page.$eval('h1', el => el.textContent));
await browser.close();

This script opens example.com and prints its main heading. For client-rendered data, wait for a specific selector or network response instead of relying on a fixed delay.

Using a proxy with Puppeteer

An authenticated proxy is commonly written as http://USERNAME:PASSWORD@HOST:PORT. Chromium’s launch argument receives http://HOST:PORT, while page.authenticate() supplies the username and password when the proxy challenges the request. Call it before page.goto() or the proxy may return 407 Proxy Authentication Required.

A launch-level proxy usually applies to every page in that browser process. Changing the proxy may therefore require a separate browser process, depending on your Puppeteer version and configuration.

Limits and alternatives

A proxy changes the IP address that a website sees. It doesn’t hide automation or correct an unusual browser fingerprint. Keep request rates within the site’s limits, respect its robots.txt and terms and back off after 429 responses.

FAQ

Is Puppeteer a headless browser?

No. Puppeteer is a library that controls a browser. Headless describes running that browser without a visible interface.

Can Puppeteer use an authenticated proxy?

Yes. Set the proxy host and port when launching the browser, then provide its username and password with page.authenticate() before navigation.

Is Puppeteer good for web scraping?

Yes, especially when a page requires JavaScript, scrolling or clicks. For static HTML or an available API, a direct HTTP client usually uses fewer resources.