Beyond llms.txt: The Four-Layer Machine-Readable Content Architecture

Adapted from Duane Forrester's analysis on Duane Forrester Decodes. Aurora extends his framing for marketing teams running AI-search visibility programs.

llms.txt was a reasonable starting point. It is also visibly running out of road. A flat markdown manifest at the site root cannot express that Product A belongs to Product Family B, that Feature X was deprecated in Version 3.2 and replaced by Feature Y, or that this pricing tier applies only to a specific region between specific dates. When an AI agent runs a comparison query and has to resolve contradictions across multiple sources, a flat list with no provenance metadata is the input most likely to produce confident-sounding but wrong outputs about your brand.

Adoption tells the same story. Server-log analyses across hundreds of thousands of domains show major AI crawlers do not routinely request /llms.txt. Google's John Mueller compared it to the deprecated meta keywords tag. Gary Illyes confirmed at Search Central Live in July 2025 that Google does not support and does not plan to support it. The platforms that would have to consume the file did not.

What will survive is the principle underneath: content must be structured for machine understanding while staying valuable for humans. That principle outlasts any specific protocol. The brands that build for it now will define the architecture standards form around — the same way the 2012 Schema.org early adopters shaped how Google consumed structured data for the following decade.

Forrester proposes a four-layer architecture as the practical version of that principle.

Layer 1: Structured fact sheets in JSON-LD

JSON-LD is the foundational machine-facing fact layer. Treat it not as a rich-snippet enhancement but as the authoritative machine-readable description of your products, services, pricing, features, and relationships. Pages with valid structured data are 2.3× more likely to appear in Google AI Overviews than equivalent pages without it. Princeton's GEO research found content with clear structural signals saw up to 40% higher visibility in AI-generated responses. This is the floor of the architecture.

Layer 2: Entity relationship mapping

Where Layer 1 describes individual entities, Layer 2 expresses the graph that connects them. Implementation ranges from lightweight @id graph extensions in JSON-LD to dedicated relationship endpoints in a headless CMS. The point is to let an AI system traverse your content the way a well-organized human analyst would review a product catalog — with relationship context preserved at every step. Pricing tier belongs to plan. Integration belongs to solution category. Case study belongs to industry. The graph is what turns a list of facts into a model the AI can reason over.

Layer 3: Content API endpoints

Moving from passive markup to active infrastructure. Programmatic endpoints provide timestamped, attributed responses that signal authority to AI agents. The Model Context Protocol — introduced by Anthropic in late 2024, subsequently adopted by OpenAI, Google DeepMind, and the Linux Foundation — provides a standardized framework for exactly this kind of integration. You do not have to implement MCP today. You do have to recognize that the trajectory of brand-to-AI data exchange is heading toward structured, authenticated, real-time interfaces, and the brands shipping API-grade content layers now will not have to retrofit them later.

Layer 4: Verification and provenance metadata

Timestamps, authorship, update history, and source chains transform content from inferred information into verifiable facts. When a RAG system has to choose between several conflicting facts to surface in a response, provenance metadata is the tiebreaker. A pricing page with no last-updated timestamp loses to a competitor's pricing page that has one, even if both are accurate.

What this looks like in practice

Forrester illustrates with a mid-market SaaS example: a $50M ARR project management platform with three pricing tiers and 150 integrations. Without machine-readable architecture, the pricing renders in JavaScript and is opaque to AI; feature comparisons are locked in PDFs that resist reliable parsing; case studies lack structured attribution. When an AI agent evaluates them against a competitor for a procurement comparison, it works from whatever it can infer from crawled text — which means it is probably wrong on pricing and probably wrong on which features are enterprise-only.

With the four-layer architecture: JSON-LD describes pricing tiers programmatically; entity relationships cluster integrations into solution categories; a content API serves structured comparison data with provenance metadata timestamping every fact. The AI does not hallucinate pricing. It correctly represents enterprise features. It surfaces the right integrations because the entity graph connected them to the right solution category. The compounding outcome: clean structured data also produces sharper RAG embeddings, which improves the brand's presence at the retrieval layer.

Build vs. wait

Standards remain unsettled. Historical precedent favors early implementation anyway. The brands that implemented Schema.org in 2012 shaped how Google consumed structured data for the following decade. They did not wait for a guarantee. They built to the principle and let the standard form around their use case.

A minimum viable implementation this quarter

Three concrete steps achievable in 90 days:

1. JSON-LD audit and upgrade of core commercial pages — Organization, Product, Service, FAQPage — with proper @id graph linking so entities reference each other rather than living in isolation.
2. A single structured content endpoint for frequently compared information (pricing, core features). Generate it programmatically from your CMS so it stays current without manual sync.
3. Provenance metadata on every public-facing fact: timestamp, attributed author or team, version reference.

This is not an llms.txt. It is not a markdown copy of your website. It is durable infrastructure that serves both current AI retrieval systems and whatever standard formalizes next.

Where Aurora helps

Aurora measures whether the architecture is doing its job. Per-prompt citation tracking tells you whether your product pages are showing up in AI answers about your category. Brand visibility scoring catches the cases where the model uses your content without citing — the paraphrase signal that often precedes a citation as trust builds. Source intelligence shows which competitor or third-party domains the engines reach for in your category, which is the directional signal for whether the architecture is moving you onto the playing field or leaving you off it. If you ship the four layers and the per-prompt visibility moves, you have your answer. If it does not move, you have the diagnostic that tells you to look elsewhere.

The brands asking should we build this? are already behind the ones asking how do we scale it? Start with the minimum. Ship something this quarter you can measure. The architecture will tell you where to go next.