Content Negotiation for AI: Serving Markdown Without Cloaking

Content Negotiation for AI: Serving Markdown Without Cloaking

A single blog post consumes 16,180 tokens as HTML but only 3,150 tokens as Markdown — an 80% reduction. That gap is why content negotiation matters for AI visibility.

The Problem with HTML for AI

When an AI crawler visits your page, it receives the same HTML served to browsers: navigation bars, cookie banners, footer links, JavaScript, CSS classes, and thousands of tokens of structural noise. The actual content might be 20% of the payload.

AI systems have finite context windows. Every wasted token on HTML chrome is a token not spent understanding your content. The more efficient the format, the more of your page fits into the retrieval context — and the more likely it gets cited.

What Is Content Negotiation?

HTTP content negotiation is a standard mechanism (RFC 9110 §12) where a client indicates what format it prefers via the Accept header, and the server responds accordingly.

# Browser request
GET /blog/my-article HTTP/1.1
Accept: text/html

# AI agent request
GET /blog/my-article HTTP/1.1
Accept: text/markdown

Same URL. Same content semantically. Different format based on what the client asked for. This is the same mechanism used for JSON APIs since the late 1990s.

Why This Isn't Cloaking

Cloaking serves different content to search engines and users — typically by checking User-Agent strings and returning keyword-stuffed pages to crawlers. That's a Google penalty.

Content negotiation serves the same semantic content in the format the client requested. The key differences:

Cloaking (Bad) Content Negotiation (Safe)
Different content per visitor ✅ ❌
Based on User-Agent ✅ ❌
Based on Accept header ❌ ✅
Same URL Maybe ✅
Semantic parity ❌ ✅
Standards-based ❌ ✅ (RFC 9110)

Cloudflare ships this feature ("Markdown for Agents", Feb 2026). Vercel documents it as a best practice. It's the pattern major platforms endorse.

How It Works in Practice

When a request comes in, your server checks the Accept header:

  1. Accept: text/html → Serve normal HTML (browsers, Googlebot)
  2. Accept: text/markdown → Serve Markdown representation (AI agents)
  3. Accept: */* → Serve HTML (default — this is what Googlebot sends)

The response includes Vary: Accept so caches and CDNs store both variants correctly.

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
X-Content-Format: markdown

Which AI Agents Request Markdown?

As of mid-2026, these agents send Accept: text/markdown when their browse/fetch tools hit a URL:

  • Claude (via browse tool) — requests text/markdown when available
  • ChatGPT (browse mode) — checks for Markdown variants
  • Perplexity — uses content negotiation for source extraction
  • Google AI — evaluates Markdown via the Gemini pipeline

Even agents that don't explicitly request Markdown benefit from it when llms.txt points them to Markdown-formatted content.

WordPress Implementation

The Zitably plugin handles this automatically:

  1. Intercepts requests at the WordPress template_redirect hook
  2. Checks the Accept header for text/markdown
  3. Converts the post content to clean, structured Markdown
  4. Serves it with proper Content-Type and Vary headers
  5. Logs the request for your bot traffic dashboard

No configuration needed. Install the plugin, and Markdown serving is active for any client that requests it.

For sites not using Zitably, you can implement basic content negotiation with a WordPress filter:

add_action('template_redirect', function() {
    if (strpos($_SERVER['HTTP_ACCEPT'] ?? '', 'text/markdown') === false) {
        return;
    }
    
    $post = get_queried_object();
    if (!$post || !is_singular()) return;
    
    header('Content-Type: text/markdown; charset=utf-8');
    header('Vary: Accept');
    
    $title = get_the_title($post);
    $content = apply_filters('the_content', $post->post_content);
    // Strip HTML to basic Markdown (simplified)
    $markdown = "# {$title}\n\n" . strip_tags($content);
    
    echo $markdown;
    exit;
});

(Note: This is a simplified example. Production implementations need proper HTML-to-Markdown conversion that preserves headings, lists, tables, and code blocks.)

Performance Considerations

  • Markdown responses are 70–80% smaller than HTML equivalents
  • Server-side conversion adds <5ms per request
  • Add Cache-Control headers for CDN caching of both variants
  • Use <link rel="alternate" type="text/markdown"> in your HTML <head> for discovery

The Token Economics

Format Tokens (Typical Blog Post) Context Usage
Raw HTML ~16,000 80% noise
Cleaned HTML ~8,000 40% noise
Markdown ~3,200 5% noise

When an AI system retrieves 10 pages for context, the format difference means fitting 5× more actual content into the same context window. More content in context = better answers = higher citation probability.

Zitably implements production-grade content negotiation for WordPress out of the box. See all features →

Sources