llms.txt Is Not a Standard. Here's What B2B Marketers Should Actually Do About It.

A file placed at your website root will not make large language models treat your content better. Not yet. But the underlying problem it addresses is real and growing.

You may have seen llms.txt mentioned in SEO newsletters, developer Slack channels, or a LinkedIn post claiming it's the robots.txt of the AI era. Some vendors have started generating it automatically. Agencies are beginning to include it in audits.

Before you add it to a Q2 content initiative, understand what it actually is.

What llms.txt Is

In September 2024, Jeremy Howard of Answer.AI proposed placing a Markdown file at /llms.txt on any website. The file is meant to give large language models a curated summary of what a site contains: a plain-language description, links to important pages, API documentation, and structured content that is easier for a model to parse than JavaScript-heavy HTML.

The premise is reasonable. LLMs work with limited context windows. When a model retrieves a page to answer a question, it gets navigation clutter, cookie banners, and JavaScript that renders nothing useful. A clean Markdown file bypasses that noise. Think of it as a curated sitemap written for inference rather than for indexing.

Some well-known technology companies have published one. Anthropic has one. Stripe, Vercel, and Cloudflare have examples. Documentation tooling platforms like Mintlify now generate it automatically. By late 2025, trackers reported over 800,000 sites with the file in place (vendor-supplied figure, unaudited).

Why It Is Not a Standard

robots.txt has RFC 9309 behind it. Major crawlers honor it by convention and formal commitment. llms.txt has none of that. It is an informal proposal posted at llmstxt.org. There is no W3C process, no published RFC, no schema with enforcement.

More importantly: the platforms that matter have not committed to using it. Google's John Mueller put it plainly in a Reddit discussion on the topic: none of the AI services have said they're using llms.txt, and server logs confirm they are not checking for it. His comparison was direct — he called it comparable to the keywords meta tag, a signal that site owners supply about themselves but that search engines stopped trusting because it was too easy to game and too unreliable to verify.

The underlying logic holds. A crawler that retrieves an llms.txt file still has to verify the actual site content to guard against spam. At that point, why not just read the site directly? The file creates an obvious surface for cloaking: show AI agents one set of content, show users and search engines another. That is exactly why the keywords meta tag was abandoned, and it is the same structural problem here.

Practitioners who have run server log analysis confirm the picture. One host managing over 20,000 domains reported that no AI agents are retrieving the file — only niche technical bots are hitting that path. Anthropic publishes its own llms.txt but has not confirmed that its crawlers rely on it for training or retrieval decisions. That distinction matters.

Adoption surveys show mixed results. Some large-domain surveys put implementation at around 10 percent. Studies of top-ranked sites show figures near zero. The developer and documentation community has moved faster than the broader web, which tracks — the file is most obviously useful for technical reference content and API documentation.

The problem llms.txt is trying to solve is real. The solution just isn't functional yet at most sites.

What Actually Helps Your Content Surface in AI Answers Today

The underlying challenge llms.txt points at is legitimate: when an LLM retrieves your content, it often gets a degraded version of it. Noise reduces the probability that your actual expertise comes through. That problem exists with or without the file.

What reliably helps right now:

• Clean, semantic HTML. Proper heading hierarchy (H1, H2, H3 in sequence), descriptive anchor text, and minimal JavaScript required to render body content all make your pages more parseable. This was true for traditional search engines. It is still true for AI retrieval.
• Direct answers near the top of the page. Models tend to cite content that states a position or answers a question in the first 200 words. If your content buries the point under context-setting, it loses out in AI-generated summaries.
• Factual specificity. Named executives, dates, product names, and concrete outcomes give a model something retrievable. Generic claims ("we help companies grow") are invisible to AI answers.
• Consistent entity presence. If your company, product, or executive is referenced across credible external sources, models are more likely to include your content as a corroborating source. This is where earned media and third-party mentions matter more than ever.
• Structured data where it applies. Schema.org markup for articles, FAQs, and products provides explicit signals that remain well-supported across retrieval systems.

Where llms.txt Could Become Worth Your Attention

There are two scenarios where the file has legitimate near-term value.

The first is technical documentation. If your product has developer documentation, API references, or integration guides, a well-constructed llms.txt helps AI coding tools and developer-facing agents navigate your content more accurately. This is where adoption has been highest and where the ROI is clearest.

The second is forward positioning. If a major platform announces formal support for the convention, companies that already have the file benefit immediately. Implementation takes less than a day for most sites. The cost of maintaining it is near zero. Some organizations treat it as low-cost insurance against a shift that could happen in 2026 or beyond.

Watch for announcements from OpenAI, Anthropic, and Google. If any of them confirms that their retrieval or crawling infrastructure honors the file, move it up your priority list. Until then, it is optional.

What to Do Monday

If someone on your team or your agency is pushing llms.txt as a priority initiative, redirect the conversation. The time is better spent on content fundamentals that are working now.

Run a sample of your 10 most important pages through a plain-text extractor or a reader mode view to see what a model actually receives when it retrieves that page. Three ways to do this without any tools:

• Firefox Reader View. Open the page in Firefox. If the reader icon appears in the address bar, click it. What renders is roughly what a retrieval system sees: headline, body copy, images with alt text. If the icon does not appear at all, Firefox cannot detect a clean article structure — that is a signal worth investigating.
• Google Cache text-only view. In Google Search, find your page in results, click the three dots next to the result, and select Cached. Switch to the text-only version. This strips CSS and JavaScript and shows raw content order. If your main message appears below a hundred lines of navigation and footer links, models are reading that noise before they reach your point.
• Check whether your body content exists in page source. In Chrome or Firefox, right-click any important page and select View Page Source. Use Ctrl+F to search for a sentence from your main body copy. If it does not appear in the source, the content is JavaScript-rendered — meaning many AI crawlers never see it at all. This is the fastest way to identify a structural problem that no amount of llms.txt will fix.

If the extracted output is thin, fragmented, or dominated by calls to action and navigation text, the fix is structural: tighten your HTML semantics, move your main argument higher on the page, and ensure body copy is rendered in the HTML source rather than injected by JavaScript after load.

If you run a documentation-heavy site or a developer-facing product, implement llms.txt now. The format is simple Markdown: a site summary at the top, followed by categorized links to your most important content. Visit llmstxt.org for the proposed structure and review how Stripe or Vercel have formatted theirs.

For everyone else: set a quarterly reminder to check whether major platforms have updated their position on the file. Add the check to whatever process you use to monitor AI search developments. When the support is there, the implementation is fast. There is no reason to make it a project today.

---