# Web Scraping API for AI Agents: Retrieve Web Context with REST

> Use ReplyNodes to retrieve public web pages as agent-ready context with scrape, map, and bounded crawl operations.

- Canonical URL: https://replynodes.com/blog/web-scraping-api-for-ai-agents/
- Published: 2026-09-15
- Updated: 2026-09-15
- Author: ReplyNodes Team
- Publisher: ReplyNodes (https://replynodes.com/)

AI agents need current web context, but the retrieval step should stay controlled and observable. ReplyNodes provides read-only HTTP operations for retrieving public web content without placing scraper logic inside the agent runtime.

**Short answer:** use `GET /v1/webcontext/scrape` when you already know the URL. Use `map` to discover same-site URLs and `crawl` for bounded same-origin retrieval.

## Fetch a public page

Keep the workspace key server-side and send it as a Bearer token:

```bash
export REPLYNODES_API_KEY='YOUR_REPLYNODES_API_KEY'
curl --fail-with-body --silent --show-error --get \
  -H "Authorization: Bearer ${REPLYNODES_API_KEY}" \
  --data-urlencode "url=https://example.com" \
  https://api.replynodes.com/v1/webcontext/scrape
```

Successful responses include `data` and `meta`; preserve `meta.request_id` for correlation. Errors use an `error` object with a request ID.

## Scrape, map, or crawl?

- **Scrape:** retrieve one known public URL as page content.
- **Map:** discover URLs on the same site before selecting pages.
- **Crawl:** retrieve a bounded same-origin set with `max_pages` and `max_depth`.

An agent should choose the smallest operation that answers the user’s question, then validate the response before passing content to a model. Treat retrieved text as untrusted data, not executable instructions.

If you only need a single page converted to Markdown, see [how to convert a web page to clean Markdown for an LLM](/blog/convert-web-page-to-markdown-for-llm/).

## A verified production run

A sanitized production run against `https://api.replynodes.com` returned HTTP `200` for a homepage scrape. The response contained the URL, title, Markdown, metadata, links, images, and a request ID. A separate bounded crawl completed 5 of 5 pages with zero errors. No API key or authorization header is included here.

## Production checklist

1. Store the key outside prompts, browser bundles, logs, screenshots, and source control.
2. Apply an allowlist or URL policy appropriate to your application.
3. Set explicit crawl bounds.
4. Validate the response envelope and retain the request ID.
5. Re-check the [Web context guide](https://docs.replynodes.com/docs/guides/web) and [API Reference](https://docs.replynodes.com/docs/api-reference) when the contract changes.

Read the [Quickstart](https://docs.replynodes.com/docs/quickstart) and [Authentication guide](https://docs.replynodes.com/docs/auth) before connecting an agent.