cURL

What cURL is used for

cURL sends network requests from a terminal or script without opening a browser. Common uses include:

• Calling an API with GET, POST and other HTTP methods.
• Inspecting response headers, redirects and status codes.
• Downloading files or static HTML for simple data collection.
• Testing connectivity, TLS certificates and proxy settings.

The command-line program is named curl. Its underlying transfer library is called libcurl and is also used by many applications.

How cURL sends a request

A URL by itself makes a GET request:

curl -i -L \
  -H "Accept: text/html" \
  https://example.com/

The -i option includes response headers and -L follows redirects. Each -H adds an HTTP header. By default, cURL writes the response body to standard output, so you can redirect it to a file or pipe it into another program.

An HTTP 404 or 500 doesn’t always produce a failing shell exit code. Use --fail-with-body when a script should treat HTTP errors as failed commands while preserving the response body.

How to use cURL with a proxy

Pass the proxy with -x or --proxy:

curl -x http://USERNAME:PASSWORD@HOST:PORT \
  -i https://example.com/

cURL connects to the proxy, which forwards the request to the destination. For an HTTPS URL through an HTTP proxy, cURL usually creates a CONNECT tunnel and then establishes TLS with the destination.

A 407 Proxy Authentication Required response means the proxy rejected or didn’t receive valid credentials. Use -v to inspect connection details, but don’t share verbose logs without checking them for credentials, cookies and authorization headers.

Limits for web scraping

• No JavaScript execution: cURL downloads the server response but doesn’t perform JavaScript rendering. Use Playwright for browser-rendered pages.
• Manual session handling: use -c cookies.txt to save cookies and -b cookies.txt to send them later.
• No browser interaction: cURL can’t click controls, complete browser flows or work with a rendered DOM.
• No built-in crawl policy: your script must control request rates, respect robots.txt and follow the site’s terms.

Changing the User-Agent alone doesn’t make cURL behave like a browser. Websites can also evaluate TLS details, headers, cookies and request patterns.

FAQ

Can cURL scrape a website?

Yes, for static HTML, files and APIs. It won’t render client-side JavaScript, so dynamic pages often require a browser automation tool.

Does cURL use a proxy automatically?

Usually not. You must provide -x, configure a cURL settings file or set a supported proxy environment variable such as HTTPS_PROXY.

How can I tell if cURL connected through the proxy?

Run the command with -v and check the connection lines for the proxy host and port. For HTTPS destinations, the output often shows the CONNECT request used to create the tunnel.