Is llms.txt worth it? An honest look, and how to build a good one

If you have spent any time reading about AI and search this past year, you have run into llms.txt. It gets pitched as the robots.txt for the AI era, the file that finally makes your site readable to ChatGPT, Claude and the rest. Some of that is true. A lot of it is wishful thinking. This post gives you the honest picture of where llms.txt stands today, where the real value is, and how to build a file that actually earns its place, using the LLMs.txt Generator plugin for Joomla to do it.

What llms.txt actually is

llms.txt is a proposal from Jeremy Howard of Answer.AI, first published in September 2024. The idea is simple. Large language models work with a limited context window, and a full website with its navigation, scripts and markup is noisy and expensive to read. So you place a single curated Markdown file at /llms.txt that lists your most important pages as clean links with short descriptions, plus a title and a summary explaining what the site is about. Think of it as a hand-picked table of contents written for a machine instead of a crawler that has to guess.

It is a sensible idea. The question is whether anything on the receiving end actually reads it.

How it is really being used

This is the part most articles skip, so let's be direct about it.

As an SEO or ranking tactic, llms.txt does not work. Google has said so plainly. Gary Illyes confirmed in mid-2025 that Google does not support the file and has no plans to, and John Mueller compared it to the old keywords meta tag, the one search engines learned to ignore because it was too easy to game. Google's own 2026 guidance for AI features lists llms.txt by name as something that does not help. Mueller's point is worth understanding: if a model has already downloaded your real pages, why would it trust a separate file that could say anything? And what stops someone from putting polished content in llms.txt while showing users something else? That is cloaking, and it is exactly why crawlers do not lean on it.

The server logs back this up. A SE Ranking study of 300,000 domains found roughly one in ten sites now publishes an llms.txt file, so adoption is real. But an Ahrefs analysis in May 2026 found that 97% of those files received zero AI requests. Across hundreds of millions of AI bot visits, only a tiny fraction ever touched llms.txt. GPTBot fetches it occasionally, and the crawlers behind Google, Claude and Perplexity effectively do not. No major provider, not OpenAI, Anthropic, Google, Meta or Mistral, has committed to using it as a signal in their live answer surfaces.

So if the promise was better rankings in AI search, the honest answer today is no.

Where the value actually is

Here is the other side, and it is genuine.

The tools that reliably do read llms.txt are AI coding assistants and agents. Cursor, Claude Code, GitHub Copilot, Windsurf, Cline and Aider all look for /llms.txt when you point them at a site, and they use it to navigate documentation efficiently. This is why so many developer platforms, including Anthropic and Perplexity's own documentation, publish the file. It is built for the case where an agent is told to use your site as a source and needs to find the right page without wading through your entire HTML.

That reframes who llms.txt is for. If your site is documentation, a knowledge base, an API reference, a product catalogue or anything an AI agent might be asked to look things up in, the file has a real job to do. If your goal is purely to rank higher in ChatGPT answers, it will not get you there.

There is also a low-cost, forward-looking argument. The file is cheap to generate and keep updated, it does no harm, and the ecosystem is still forming. Publishing a clean, honest llms.txt now means you are ready if consumption grows, without betting anything on it. Just go in with correct expectations rather than the hype.

Best practice: building a file that earns its place

A bad llms.txt is a dump of every URL on your site. A good one is curated, current and honestly labelled. Here is how to build that, and how each choice maps to the plugin settings.

Start with a clear header. The Title, Summary and Intro fields at the top of the file tell a model what your site is in a sentence or two before it looks at a single link. Write these as if you were briefing someone who has never heard of you. This is the cheapest context you can give and the most often skipped.

Curate, do not mirror. The fastest way to a useless file is to include everything. Build your sections from your main menu or a hand-picked set of pages rather than your whole article base. In the plugin, a Menu section with "Only these items" lets you list exactly the pages you want, and "Exclude items" drops the noise. Set a sensible Max items so a section stays focused. If you only want top-level pages, turn off "Include submenu items".

Give every link a real description. A link with no context is close to useless to a model. The Descriptions setting controls this: leave it on Automatic to use each page's meta description and fall back to the intro text, or force Meta description only if you keep those tidy. This is a good reason to actually write meta descriptions. If you genuinely only want a link list, set it to none, but that is rarely the better choice.

Structure it into honest sections. Split your file into clear H2 sections by theme, for example Documentation, Products and Guides, each one its own section in the plugin with its own heading and source. The spec has a convention worth using: name a section Optional and agents may skip it when they are short on context, so put secondary material there.

Keep it fresh, automatically. A stale llms.txt is worse than none. Because this is a scheduled task plugin, it rebuilds the file on a schedule you set, so it tracks your site without you touching it. For a blog or news section, use "Maximum age (days)" to include only recent articles and set the order to Newest first, so the file always reflects what is current rather than what you published three years ago.

Only expose what is public. You never want an AI file leaking pages that sit behind a login. The plugin handles this for you: every item is filtered against the public and guest access levels, so a section can only ever contain links a visitor without an account could already reach. That means you can point a section at a menu or category without auditing each item by hand.

Handle multilingual cleanly. On a multilingual site, add a Language filter per section so each one stays in a single language, and enable "Remove URL language code" so links come out as /page instead of /en/page where the prefix adds nothing.

The short version

llms.txt will not lift your search rankings, and the big AI crawlers largely ignore it today. What it does do, reliably, is help AI agents and developer tools use your site as a clean, curated source. If that fits your site, it is worth doing well, and doing it well means a small, current, honestly described file rather than a mirror of your sitemap. The LLMs.txt Generator plugin builds exactly that from your existing Joomla menus and content, and keeps it up to date on a schedule so you set it once and leave it running.