How we serve markdown on AgentReady.dev

We built AgentReady.dev to test whether websites serve clean markdown to AI agents. It felt wrong to ship a tool that audits other sites and not pass our own checks. So we implemented content negotiation on agentready.dev itself.

Here's exactly how it works.

The pattern

The goal is simple: when a request includes Accept: text/markdown, return a clean markdown version of the page instead of HTML. Same URL, different response based on what the client asked for.

We needed this for three routes: / (home), /blog (post index), and /blog/[slug] (individual posts).

Step 1: Middleware intercepts the request

Next.js middleware runs before any route handler. We use it to detect the Accept header and rewrite matching requests to a dedicated API handler.

// proxy.ts
const MARKDOWN_PATHS = new Set(["/", "/blog"]);

function isMarkdownPath(path: string): boolean {
  const normalized = path.replace(/\/$/, "") || "/";
  if (MARKDOWN_PATHS.has(normalized)) return true;
  if (normalized.startsWith("/blog/") && normalized.split("/").length === 3) return true;
  return false;
}

export function proxy(request: NextRequest) {
  const accept = request.headers.get("accept") || "";

  if (!accept.includes("text/markdown")) {
    return NextResponse.next();
  }

  const path = request.nextUrl.pathname;

  if (!isMarkdownPath(path)) {
    return NextResponse.next();
  }

  const url = request.nextUrl.clone();
  url.pathname = "/api/md";

  return NextResponse.rewrite(url, {
    request: {
      headers: new Headers({
        ...Object.fromEntries(request.headers),
        "x-original-path": path,
      }),
    },
  });
}

export const config = {
  matcher: ["/", "/blog", "/blog/:slug*"],
};

The key detail: we forward the original path in an x-original-path header. The rewrite changes the URL to /api/md, but we need the handler to know which page was actually requested.

Step 2: The API handler generates the markdown

/api/md is a single route handler that reads the forwarded path and calls the appropriate generator function.

// app/api/md/route.ts
export async function GET(request: NextRequest) {
  const originalPath = request.headers.get("x-original-path") || "/";
  const path = originalPath.replace(/\/$/, "") || "/";

  let markdown: string | null = null;

  if (path === "/") {
    markdown = homeMarkdown();
  } else if (path === "/blog") {
    markdown = blogIndexMarkdown();
  } else if (path.startsWith("/blog/")) {
    const slug = path.replace("/blog/", "");
    markdown = blogPostMarkdown(slug);
  }

  if (!markdown) {
    return NextResponse.json({ error: "Not found" }, { status: 404 });
  }

  return new NextResponse(markdown, {
    status: 200,
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Cache-Control": "public, s-maxage=3600, stale-while-revalidate=86400",
    },
  });
}

Step 3: Generating the markdown content

Each page type has its own generator in lib/markdown-pages.ts. They all follow the same structure: YAML frontmatter first, then clean content with no nav or wrapper markup.

The home page generates a sitemap-style link index, which is what earns the Sitemap/Index check in our own auditor:

export function homeMarkdown(): string {
  const posts = getAllPosts();
  const postLinks = posts
    .map((p) => `- [${p.title}](${BASE_URL}/blog/${p.slug})`)
    .join("\n");

  return `---
title: "AgentReady.dev — Is Your Website Agent Ready?"
description: "Audit how well websites handle AI agent requests."
url: "${BASE_URL}"
---

# AgentReady.dev

...

## Site Pages

- [Home](${BASE_URL}/)
- [Blog](${BASE_URL}/blog)
${postLinks}
`;
}

Blog posts are even simpler. We just strip the MDX content out and wrap it in frontmatter:

export function blogPostMarkdown(slug: string): string | null {
  const post = getPostBySlug(slug);
  if (!post) return null;

  return `---
title: "${post.meta.title}"
description: "${post.meta.description}"
date: "${post.meta.date}"
url: "${BASE_URL}/blog/${slug}"
---

# ${post.meta.title}

${post.content}
`;
}

Testing it

You can verify it works with curl:

curl -H "Accept: text/markdown" https://agentready.dev/
curl -H "Accept: text/markdown" https://agentready.dev/blog/how-we-serve-markdown

Or run AgentReady.dev on itself. We do — we'd know if we broke it.

What this approach gets right

A few things worth calling out:

No nav, no wrapper markup. The markdown generators only emit content. No navigation links, no headers/footers. That's what the Navigation Stripped check tests for.

Frontmatter on every response. Title, description, date, and canonical URL. Structured metadata that agents can read directly.

The home page is a link directory. An agent hitting the root URL gets a structured list of every major page on the site, not a marketing hero section it can't parse.

Cache-friendly. The markdown responses get a one-hour CDN cache with stale-while-revalidate. Same as our HTML pages.