# How AI Agents Should Fetch Markdown: Content Negotiation, llms.txt, Then a Converter

> A practical fetch ladder for AI agents: request Markdown, check published site files, then convert HTML while preserving provenance.

- Canonical URL: https://replynodes.com/blog/agent-markdown-content-negotiation/
- Published: 2026-10-31
- Updated: 2026-10-31
- Author: ReplyNodes Team
- Publisher: ReplyNodes (https://replynodes.com/)

When an agent needs a readable page, converting HTML should not be the first step. Ask the source for Markdown, check whether the site publishes useful Markdown or an `llms.txt` file, and convert the original page only when those options do not provide what you need.

This order can avoid unnecessary conversion while keeping the fetched content tied to its source URL. It is a retrieval strategy—not a guarantee that a site supports Markdown or that Markdown will be more complete.

## A three-step fetch ladder

1. **Negotiate with the page.** Request the original URL with `Accept: text/markdown`. Check the status and `Content-Type`; a successful response can still be HTML.
2. **Inspect what the site publishes.** If the page came back as HTML, check the site root for `llms.txt`. Follow a listed `.md` link only when its same-origin path corresponds to the requested page (for example, `/guide.html` → `/guide.md` or `/guide/` → `/guide/index.md`). Do not substitute an unrelated link from the site's guide. The file may be absent, or may not list a matching Markdown page.
3. **Convert only if needed.** If neither response gives you usable Markdown, pass the original URL to a converter you trust. Keep the original URL as the source, and record the converter separately as the transformation step.

Cloudflare documents an opt-in `Accept: text/markdown` approach for eligible responses. A small probe on October 31, 2026 found a Cloudflare documentation page responding with `text/markdown`, while `example.com` and `wikipedia.org` returned `text/html`. The same day's probes found `cloudflare.com/llms.txt` and `cloudflare.com/llms-full.txt`, but not `example.com/llms.txt` or `wikipedia.org/llms.txt`. These are point-in-time checks of a few URLs—not evidence of broad support or adoption.

`llms.txt` is a proposal, not a guarantee that a site has published the file or that it contains a page-specific Markdown alternative. Check the actual response rather than assuming the file exists.

## A runnable TypeScript example

The example below tries Markdown negotiation, checks the site's `llms.txt`, and then optionally calls a configured converter. It uses Node.js 20+ built-in `fetch`. The converter URL is a caller-supplied template containing `{url}`; it must accept a URL-encoded source URL and return Markdown. For example, configure an endpoint only after checking its current contract; ReplyNodes documents a Markdown API at [Markdown API](/markdown-api/). The example rejects converter redirects to a different origin. Without a configured converter, it reports that conversion is needed instead of guessing an API.

```ts
type FetchResult = {
  markdown: string;
  provenance: {
    requestedUrl: string;
    finalUrl: string;
    contentType: string;
    vary: string | null;
    method: string;
    converterUrl?: string;
  };
};

const TIMEOUT_MS = 10_000;
const MAX_REDIRECTS = 5;

function checkedHttpUrl(value: string): URL {
  const url = new URL(value);
  if (url.protocol !== "https:" && url.protocol !== "http:") {
    throw new Error(`Unsupported URL scheme: ${url.protocol}`);
  }
  return url;
}

async function fetchChecked(
  input: string,
  accept: string,
): Promise<{ response: Response; finalUrl: string }> {
  let current = checkedHttpUrl(input);

  for (let redirects = 0; ; redirects++) {
    const response = await fetch(current, {
      headers: { Accept: accept },
      redirect: "manual",
      signal: AbortSignal.timeout(TIMEOUT_MS),
    });

    if (![301, 302, 303, 307, 308].includes(response.status)) {
      return { response, finalUrl: current.href };
    }

    if (redirects >= MAX_REDIRECTS) {
      throw new Error("Too many redirects");
    }

    const location = response.headers.get("location");
    if (!location) throw new Error("Redirect has no Location header");

    current = checkedHttpUrl(new URL(location, current).href);
  }
}

function contentType(response: Response): string {
  return response.headers.get("content-type")?.toLowerCase() ?? "";
}

function isMarkdown(type: string): boolean {
  return type.includes("text/markdown");
}

function isHtml(type: string): boolean {
  return type.includes("text/html");
}

async function readText(response: Response): Promise<string> {
  if (!response.ok) {
    throw new Error(`HTTP ${response.status}`);
  }
  return response.text();
}

function matchingMarkdownLinks(text: string, base: URL, page: URL): URL[] {
  const pagePath = page.pathname.replace(/\.html?$/i, "");
  const normalized = pagePath.endsWith("/") ? pagePath.slice(0, -1) : pagePath;
  const expectedPaths = new Set([
    `${normalized}.md`,
    `${pagePath.endsWith("/") ? pagePath : `${pagePath}/`}index.md`,
  ]);
  const links: URL[] = [];

  for (const match of text.matchAll(/\]\(([^)]+)\)/g)) {
    try {
      const candidate = new URL(match[1], base);
      if (
        (candidate.protocol === "https:" || candidate.protocol === "http:") &&
        candidate.origin === page.origin &&
        expectedPaths.has(candidate.pathname)
      ) {
        links.push(candidate);
      }
    } catch {
      // Ignore malformed or unsupported links in the site-provided file.
    }
  }
  return links;
}

export async function fetchPageAsMarkdown(
  input: string,
  converterTemplate = process.env.MARKDOWN_CONVERTER_TEMPLATE,
): Promise<FetchResult> {
  const requestedUrl = checkedHttpUrl(input).href;

  const direct = await fetchChecked(requestedUrl, "text/markdown, text/html;q=0.9");
  const directType = contentType(direct.response);

  if (direct.response.ok && isMarkdown(directType)) {
    return {
      markdown: await readText(direct.response),
      provenance: {
        requestedUrl,
        finalUrl: direct.finalUrl,
        contentType: directType,
        vary: direct.response.headers.get("vary"),
        method: "Accept: text/markdown",
      },
    };
  }

  if (!direct.response.ok || !isHtml(directType)) {
    throw new Error(
      `Expected Markdown or HTML; got HTTP ${direct.response.status} ${directType || "(no Content-Type)"}`,
    );
  }

  const pageUrl = new URL(direct.finalUrl);
  const llmsUrl = new URL("/llms.txt", pageUrl);
  const guide = await fetchChecked(llmsUrl.href, "text/plain, text/markdown;q=0.9");
  const guideType = contentType(guide.response);
  let guideText = "";

  if (guide.response.ok && (guideType.includes("text/plain") || isMarkdown(guideType))) {
    guideText = await readText(guide.response);
  }

  for (const candidate of matchingMarkdownLinks(guideText, new URL(guide.finalUrl), pageUrl)) {
    const alternative = await fetchChecked(candidate.href, "text/markdown");
    const alternativeType = contentType(alternative.response);

    if (alternative.response.ok && isMarkdown(alternativeType)) {
      return {
        markdown: await readText(alternative.response),
        provenance: {
          requestedUrl,
          finalUrl: alternative.finalUrl,
          contentType: alternativeType,
          vary: alternative.response.headers.get("vary"),
          method: `Site-listed Markdown (${llmsUrl.href})`,
        },
      };
    }
  }

  if (!converterTemplate) {
    throw new Error(
      `HTML received and no listed Markdown page found. Configure MARKDOWN_CONVERTER_TEMPLATE to enable conversion. Original URL: ${direct.finalUrl}`,
    );
  }

  const converterUrl = converterTemplate.replace(
    "{url}",
    encodeURIComponent(direct.finalUrl),
  );
  const configuredConverter = checkedHttpUrl(converterUrl);
  const converted = await fetchChecked(configuredConverter.href, "text/markdown");
  if (new URL(converted.finalUrl).origin !== configuredConverter.origin) {
    throw new Error("Converter redirected to a different origin");
  }
  const convertedType = contentType(converted.response);

  if (!converted.response.ok || !isMarkdown(convertedType)) {
    throw new Error(
      `Converter must return Markdown; got HTTP ${converted.response.status} ${convertedType || "(no Content-Type)"}`,
    );
  }

  return {
    markdown: await readText(converted.response),
    provenance: {
      requestedUrl,
      finalUrl: direct.finalUrl,
      contentType: convertedType,
      vary: converted.response.headers.get("vary"),
      method: "Configured converter",
      converterUrl: converted.finalUrl,
    },
  };
}

const inputUrl = process.argv[2];
if (!inputUrl) throw new Error("Usage: node fetch-markdown.js <https-url>");
const result = await fetchPageAsMarkdown(inputUrl);
console.log(JSON.stringify({
  characters: result.markdown.length,
  provenance: result.provenance,
}, null, 2));
```

To run it, save the snippet as `fetch-markdown.mts` and compile it with the repository's TypeScript compiler: `npx tsc --target ES2022 --module NodeNext --moduleResolution NodeNext --lib ES2022,DOM --types node fetch-markdown.mts`. This emits `fetch-markdown.mjs`; run `node fetch-markdown.mjs https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/`. Node.js 20+ supplies built-in `fetch` and `AbortSignal.timeout`. A source that returns HTML needs a `MARKDOWN_CONVERTER_TEMPLATE` configured for a converter contract you have verified. The example permits redirects only when they stay on the configured converter's origin.

The example limits redirects and accepts only HTTP(S) URLs, but it is not a complete defense for fetching arbitrary user-supplied URLs. In a server-side agent, also apply your environment's SSRF protections—such as destination allowlists or network-level blocking—and cap response size for your workload.

### Keep the representation and provenance together

Store the requested URL, final source URL, response `Content-Type`, retrieval method, and—if conversion happened—the converter URL alongside the Markdown. That helps downstream code distinguish source-provided content from transformed content.

Pay attention to `Vary: Accept`. If a response includes it, the representation may depend on the `Accept` request header. A cache that keys only on the URL can hand back the wrong representation; preserve the relevant variation when caching, or bypass that cache when testing negotiation.

## When to convert

Convert when the source returned HTML and its published files do not provide a suitable Markdown alternative. A converter is a fallback, not proof that the result preserves every detail of the original page. Keep the original URL as provenance and treat converted output as a representation of that source, not as a new source.

For the conversion implementation itself, see [how to convert a web page to Markdown for an LLM](/blog/convert-web-page-to-markdown-for-llm/). For a separate comparison of converter options, see [Jina Reader alternatives for AI agents](/blog/jina-reader-alternatives-for-ai-agents/). ReplyNodes also has a [Markdown API](/markdown-api/); check its documentation for the current interface before wiring it into an agent.

## A practical rule

Ask the source first, inspect its own published guidance second, and convert only when needed. At every step, verify the returned media type and retain provenance. Do not infer support from a successful HTTP status alone.