Content negotiation for the agentic web

Agents are reading your site. What do they see?

AI agents don't render your site. They read it. When they request your pages with Accept: text/markdown, most sites dump raw HTML. Nav bars, cookie banners, and all. We audit what agents actually get back.

Most sites score below 30

What agents actually receive

When an AI agent visits your site, it sends Accept: text/markdown to ask for clean, structured content. Most servers ignore this header and return raw HTML instead.

GEThttps://acme.com
200 OKContent-Type: text/html; charset=utf-8
<!DOCTYPE html>
<html lang="en">
<head>
  <script src="/analytics.js"></script>
  <link rel="stylesheet" href="/styles.css">
</head>
<body>
  <nav class="navbar">
    <a href="/">Home</a>
    <a href="/about">About</a>
    <a href="/pricing">Pricing</a>
  </nav>
  <div class="cookie-banner">
    We use cookies...
    <button>Accept</button>
  </div>
  <main>
    <div class="hero">
      <h1>Welcome to Acme</h1>
      <p>We help teams ship faster.</p>
    </div>
  </main>
  <footer>
    <div class="footer-links">...</div>
    <script src="/chat-widget.js"></script>
  </footer>
</body>
</html>

~2.4 KB of noise. Nav bars, scripts, cookie banners, and chat widgets all land in the agent's context window, burning tokens and burying the actual content.

Why this matters

You optimize for Google. You optimize for mobile. The next wave of traffic comes from AI agents, and they need clean, structured content, not raw HTML.

Content Negotiation Test

text/markdownthe header agents send

We send Accept: text/markdown, the same header AI agents use. If your server ignores it and returns HTML, agents are forced to parse DOM soup, bloating context and burning tokens.

Context Bloat Detection

~70%of most responses is noise

Nav bars, footers, JavaScript, cookie banners. None of it helps an agent. We measure how much noise your pages carry versus clean, structured markdown an LLM can actually use.

Agent Discovery Audit

Sitemap + Frontmatterhow agents navigate your site

Can agents find your pages? We check for sitemaps, clean link hierarchies, and frontmatter metadata that lets agents understand your content without scraping blind.

How it works

Three steps. Results in under a minute.

1

Enter your URL

We start from your homepage, discover linked pages, and crawl up to 10, just like an agent exploring your site.

2

We fetch like an agent

Each page is requested twice: once as HTML, once with Accept: text/markdown. We compare what humans see versus what agents get.

Accept:text/html200
Accept:text/markdown200
3

See what to fix

Get a per-page score with specific failures like missing content negotiation, context bloat, and no frontmatter. Know exactly what to improve.

/pricing92
/docs/api61
/blog/hello18

The 7 checks

Each page is scored 0–100 across these weighted criteria, based on real-world patterns for optimizing content for agents.

FailingWarningPassing

Markdown Response

Fail

Does the server honor Accept: text/markdown and return actual markdown instead of HTML?

20pts

Valid Markdown

Fail

Is the response well-formed markdown, not HTML dumped as plain text?

20pts

Navigation Stripped

Warn

Are nav bars, headers, footers, and other browser chrome removed from agent-facing content?

15pts

Frontmatter Present

Warn

Does the markdown include structured metadata (title, description) so agents understand the page without parsing?

15pts

Sitemap / Index

Pass

Can agents discover your pages programmatically via sitemap or a structured index?

10pts

Link Quality

Pass

Are internal links clean and resolvable? No broken refs, no JavaScript-only navigation.

10pts

Size Delta

Pass

Is the markdown meaningfully smaller than HTML? A good delta means less context bloat and real tokenization savings.

10pts

Example above: 30 / 100 — passing sitemap, links, and size delta, but failing content negotiation and markdown quality.