How to make your website agent ready.
From a wall of HTML to a clean, readable representation: six steps in the order that actually matters, drawn from 1,904 page scans across 104 domains. Every one of them is something you can verify with a single request.
The short answer
Your site is agent ready when an AI agent asks for a page and gets a clean, purpose-built version of it instead of your whole browser document. In practice: honor Accept: text/markdown with real Markdown, strip the navigation and chrome, and include enough metadata that the agent doesn’t have to guess what it’s looking at.
It’s not a ranking tactic and it’s not a meta tag you drop in the head. It’s a second rendering path for your content, and it either exists across your site or it doesn’t.
What the data shows
Most sites send the browser page again.
Across the pages in each domain's most recent completed audit, the median “Markdown” response body was 99.8% the size of the HTML body. On 43% of pages it came back the same size or larger. The format preference changed. The payload didn’t.
- 31%
- returned a distinct response with the right content type
- 31%
- changed the response but still sent HTML
- 38%
- showed no Markdown support at all
Three patterns account for almost all of it, and none of them look broken from the outside:
- The header goes unread. Nothing on the server looks at Accept, so an agent gets the exact document a browser gets — nav, cookie banner, and all.
- Something changes, but it’s still HTML. A route responds differently and the response is still markup, so the agent is back to guessing where the content starts.
- A converter runs over the finished page. You get Markdown syntax wrapped around every browser concern that was already in the DOM. Same payload, new punctuation.
The result: an agent spends its context window on your mega-menu and reads your pricing table last.
Here’s the encouraging part. Once that first step is real, the rest tends to follow. Of the pages that did return a distinct Markdown response, 97% had navigation stripped and 99% cut the payload below half the HTML size. Nobody ships content negotiation as one isolated header change, so the teams who build the path usually build all of it. Read the full research.
The six steps
In the order that matters.
These are layers, not a checklist you can work in parallel. If the representation never changes, frontmatter and link quality have nothing to sit on.
- 01
Return a genuinely different response
Read the Accept header. When a client prefers text/markdown, send Markdown — and label it accurately.
Everything else depends on this one, and it’s where most sites stop. Inspect the request’s Accept header, and when a client prefers text/markdown, return a Markdown body with Content-Type: text/markdown.
Send Vary: Accept alongside it. Skip that header and a shared cache will happily store your Markdown response and hand it to the next browser that asks for the same URL — or store the HTML and serve it to every agent that shows up after.
The exchange to aim for GET /pricing HTTP/1.1 Accept: text/markdown, text/html, */* HTTP/1.1 200 OK Content-Type: text/markdown; charset=utf-8 Vary: Accept
- 02
Render from your content, not from the finished page
Generate both representations from the same structured source instead of converting a completed DOM.
Running finished HTML through a converter gives you Markdown syntax, not agent-ready content. You’re starting with every browser concern already baked into the document, then trying to subtract exactly the right ones.
If DOM conversion is genuinely your only option, define the main-content boundary yourself instead of asking a generic converter to make editorial calls. No converter can tell a related-products carousel from the product description.
- 03
Strip the navigation and chrome
The mega-menu, cookie controls, footer columns, social icons, and every read-more link — all of it goes.
An agent has no use for your mobile menu or your newsletter modal, but those elements eat its context window exactly like real content does. Easy test: if a human would skip past it to find the answer, it doesn’t belong in the agent response.
Do this in a shared renderer, not per template. Chrome removal that works on the blog but not on product pages gives you a site that looks ready right up until an agent crosses a template boundary.
- 04
Add identity with frontmatter
Answer the obvious questions in YAML so an agent never has to infer them from the body.
Frontmatter hands an agent the title, description, canonical URL, and date without making it parse prose to find them. It costs almost nothing to emit and it removes a whole category of guesswork.
Keep the canonical URL absolute. An agent that got your Markdown through a proxy or a cache may have no reliable way to reconstruct where the content came from.
Frontmatter that answers the obvious questions --- title: "Pricing" description: "Plans, limits, and what each tier includes." url: "https://example.com/pricing" date: "2026-08-30" --- # Pricing
- 05
Make the homepage a real index
Give agents a short, organized set of descriptive links instead of a flattened version of your mega-menu.
An agent that lands on your homepage is deciding where to go next. A concise list of labeled destinations answers that. A flattened reproduction of your navigation design doesn’t.
Write link text that describes where the link goes. “Click here” and “read more” carry no information once the surrounding layout is gone, which is exactly the situation an agent is in.
- 06
Test across template boundaries, not one showcase URL
Pick a representative route from every rendering system you run, and check all of them.
Partial coverage is easy to miss and it’s everywhere. A homepage negotiates correctly while product pages fall through to HTML. A blog on one stack works while the docs on another don’t.
Test the homepage plus one route from each system you operate: product, docs, blog, pricing, anything hosted separately. Compare headers and bodies, check payload sizes, follow the links. If support disappears at a template boundary, you’re not done.
Where llms.txt fits
A discovery aid, not a substitute.
llms.txt tells agents which pages on your site are worth reading. It doesn’t change what those pages return. Publish one before you serve a real representation and you’ve pointed agents at the same browser document more efficiently — which is why it belongs after the six steps, not instead of them.
Once the representation layer exists, llms.txt is genuinely useful and takes minutes to produce. How the two signals differ.
Verify it yourself
One request tells you where you stand.
Ask for a page the way an agent would, then compare it to the browser response. Same content type, same body, same size? You’re not negotiating yet.
curl -sS -D - -o /dev/null \ -H 'Accept: text/markdown' \ https://example.com/pricing
To check a whole site instead of one URL, AgentReady runs that same comparison across up to 10 pages, applies seven checks to each, and shows you the raw response behind every result. You can also preview any page side by side to see what an agent has to read.
Run a free auditQuestions and objections
Straight answers, no folklore.
What does it mean for a website to be agent ready?
An agent-ready site returns a clean, purpose-built version of a page when an AI agent asks for one, instead of the full browser document. In practice that means honoring the Accept: text/markdown request header with real Markdown, stripping navigation and other browser chrome, and including enough metadata that an agent knows what the page is without guessing.
Is agent readiness the same thing as SEO?
No. SEO is about how a page gets discovered and ranked. Agent readiness is about what a machine receives once it arrives. A page can rank beautifully and still hand an agent 500KB of markup with the useful part buried somewhere inside it.
Do I need llms.txt to be agent ready?
llms.txt helps agents find the pages that matter, but it doesn’t change what those pages return. If a listed page still serves the full browser document, llms.txt has just pointed an agent at the same problem faster. Serve a real representation first, then publish llms.txt as a discovery aid.
Will making my site agent ready hurt my rankings?
No. Content negotiation serves browsers exactly what they asked for, so the HTML a search crawler sees doesn’t change. The one thing to get right is Vary: Accept, so a shared cache never hands a Markdown body to a browser or a crawler.
How do I know whether my site is already agent ready?
Request one of your pages with an Accept: text/markdown header and compare it to the HTML version. If the content type, the body, and the size are all effectively identical, your site isn’t negotiating. AgentReady runs that comparison across up to 10 pages and shows the response evidence behind every check.
“Our pages are already clean. Do we really need a second path?”
Maybe not, and it’s worth checking before you build anything. Some content-first sites already ship lean HTML with a clear main-content boundary, and agents do fine with that. But “clean by our standards” and “clean once the nav, cookie controls, and footer are counted” are different measurements. Send one request and read the actual bytes before you decide.
“We don’t have engineering time for a second rendering path.”
Then don’t build all six steps. Step one on your highest-traffic template is most of the value, and on most stacks it’s a header check plus a Markdown serializer you already have somewhere in the build. The teams who stall are the ones who try to convert the whole site at once instead of shipping one route and expanding from there.
“Agents read our HTML fine today.”
They do, and they’ll keep managing. The cost isn’t failure, it’s the context window. An agent spends its budget parsing your mega-menu and consent banner before it reaches the paragraph that answers the question, which shows up as slower, vaguer, more often wrong answers about your product. Nothing errors, so it’s easy to miss.