The Third Epoch of the Web

Since 2024, more than half of all web requests come from machines—for the first time in a decade, more than from humans. And they're no longer just search engine crawlers scraping your pages. They're AI agents: software that reads websites, understands them, and acts autonomously.

Your website has gained a new audience. Uninvited. And this audience has very different needs.

A Brief History of the Web in Three Acts

Act 1: Humans. The early web was a document format. HTML described text for human eyes, and websites were found via link directories—Yahoo!, DMOZ, word of mouth.

Act 2: Search Engines. In 1998, Google arrived, and suddenly there was an intermediary between content and reader: the crawler. An entire industry emerged—SEO—whose sole purpose was to make websites readable for an algorithm. Structured data, meta tags, Schema.org: tools to help machines understand what humans had long seen.

Act 3: Agents. Now there's a new intermediary, and it's hungrier than any crawler. AI agents don't index—they consume, reason, and act. Gartner predicts that by 2028, 90% of B2B purchasing will run through AI agents. McKinsey estimates 3 to 5 trillion dollars in agent-driven revenue in consumer goods alone by 2030.

The lesson from SEO was simple: if an algorithm mediates access, the content must work for that algorithm. The same lesson applies again—except the new algorithm doesn't just index. It understands, decides, and buys.

Markdown, Content Negotiation, and the Hype

In February 2026, Cloudflare launched "Markdown for Agents"—a feature that automatically converts HTML pages into Markdown when an AI agent requests them. Other companies—including us at effektiv.ch—have built similar solutions at the application level. The idea is the same everywhere. But to put it in context, two brief explanations are needed.

Markdown is a simple text markup language. The same content, without the overhead a browser needs for rendering. For AI systems that work internally with text rather than layout, Markdown is the natural format. Switch between the tabs—the difference speaks for itself:

<h2 class="section-title">
  About Us
</h2>
<p class="company-description
   text-base leading-relaxed">
  We develop AI solutions
  for Swiss SMEs.
</p>

Content Negotiation is a mechanism that has existed since the early days of the web—HTTP/1.0 from 1996 already defined it. The client tells the server which format it prefers, and the server delivers the matching variant. This happens constantly today:

The last case is new. The principle is thirty years old. And it's not cloaking—not deceptively serving different content—but the same principle by which your browser has been receiving smaller image formats for years.

Can't AI Agents Just Read HTML?

Yes. And this is where the discussion needs to get honest.

Google's John Mueller called Markdown for bots a "stupid idea" on Bluesky. His argument: LLMs were trained on trillions of HTML pages. They can read HTML. Why bother with an extra format?

Research partially supports him. HtmlRAG (WWW 2025) shows that HTML actually works better than plain text for structured data—because LLMs know the semantics of HTML tags from training. "Table Meets LLM" (WSDM 2024) confirms this for tables: HTML outperforms Markdown on several tasks significantly. And the larger the model, the less sensitive it is to format differences overall.

Mueller has a point. LLMs don't fail at HTML. The problem lies elsewhere.

The Big Noise

Open the source code of any corporate website. The actual text often accounts for just 15 to 70% of the source code. The rest? Navigation, footer, cookie banners, CSS classes, JavaScript, tracking pixels, ad banners. Invisible to humans—the browser filters it out. For a language model that must process every token, it's noise.

And this noise has measurable consequences. Researchers have demonstrated that 30,000 irrelevant tokens in the context window can push a language model's accuracy from 96% down to 11%. Not because the model can't find the information—but because the sheer volume of noise dilutes its attention. Imagine searching for a sentence in a book, but someone has glued three pages of phone directory between every page. Theoretically findable. Practically lost.

JetBrains Research confirmed this in an experiment with 500 programming tasks: removing irrelevant context reduced costs by 52%—while simultaneously improving success rates.

The value of Markdown doesn't lie in ## being better than <h2>. It lies in the noise disappearing. It's not about format. It's about signal-to-noise.

What This Means in Practice

Anyone who lived through SEO recognizes the pattern. First, companies ignored search engines. Then they panicked. Then they learned that good content matters more than technical tricks. The agent web is following the same trajectory.

What's Worth Doing

• Structure content cleanly. Semantic HTML, clear headings, descriptive alt texts—not new advice, but with new weight. What's good for accessibility is also good for AI agents. Both use the same semantic tree.
• Provide an /llms.txt file. The llms.txt standard by Jeremy Howard is a compact Markdown file that describes what your organization does and links to its most important pages. Over 844,000 websites have adopted it, including Anthropic, Stripe, and Shopify. Whether AI providers actively use the file remains an open question. But the effort is minimal.
  → View our /llms.txt
• Offer machine-readable variants. On effektiv.ch, we've implemented this: every page exists as clean Markdown, including a token estimate. AI agents that send Accept: text/markdown are automatically redirected to it. In the footer, you can switch between the "Human" and "Machine" view—try it out.
  → View this article as Markdown

Equally important is avoiding common mistakes:

• Don't build a separate Markdown web. Markdown endpoints deliver the same content, not different content. Showing AI agents different information than humans is cloaking—and destroys trust.
• Don't believe one format solves everything. For tables, HTML is better. For prose, Markdown is more efficient. For APIs, JSON is standard. The answer isn't "everything in Markdown" but the right format for the context.
• Don't forget the basics. The most elegant Markdown conversion is useless if the content is thin. AI agents recognize substance—just like search engines have since the Panda update in 2011.

Clarity Wins

The web has always adapted to new audiences. With search engines, the winners weren't the websites with the most meta keywords but those with the best content. The agent web will be no different.

The web is indeed learning a new language. But that language isn't Markdown. It's clarity.