llms.txt
A single markdown file, placed at the root of your domain at /llms.txt, that hands AI systems a clean, curated map of your most important content in plain text. It does for language models roughly what robots.txt and sitemap.xml did for search crawlers: it offers a direct, machine-readable invitation to understand your site on your terms rather than leaving models to guess from tangled HTML.
llms.txt is a proposed standard that lets you write the table of contents an AI reads first, curating what matters instead of hoping a model correctly parses your entire site.
What llms.txt Actually Is
llms.txt is a proposed web standard, introduced by Jeremy Howard of Answer.AI in September 2024. The idea is disarmingly simple: publish a plain markdown file at your domain root, reachable at /llms.txt, that describes what your site is and points to the pages that matter most. It sits beside the two files the web already agreed on years ago. robots.txt tells crawlers where they may and may not go, and sitemap.xml lists every URL for indexing. llms.txt is different in intent: it is not a list of permissions or an exhaustive URL dump, it is a human-curated briefing written specifically to be read by a large language model.
The format is deliberately lightweight. A conventional file opens with an H1 carrying the site or project name, an optional blockquote giving a one-sentence summary, then H2 sections that group links by category: documentation, guides, products, policies, key articles. Each list item is a markdown link, ideally followed by a colon and a short description of what the reader will find there. Because it is markdown, it is trivial for a model to parse and cheap for it to consume. There is no schema to validate, no rendering engine required, no JavaScript to execute.
The proposal also defines a companion file, llms-full.txt, and the distinction is worth holding onto. llms.txt is the curated index: a short map that points outward to your real pages. llms-full.txt is the curated content: the actual text of those key pages concatenated into one long markdown document, so a model can ingest your substance in a single fetch without crawling link by link. Think of llms.txt as the contents page and llms-full.txt as the whole book pressed flat.
Watch: LLMs.txt: What It Is and Do You Need One? by Edward Sturm, an honest walkthrough of the proposal, its promise, and the real state of adoption. Source: YouTube.
Why It Matters for Whether AI Can Find and Cite You
Language models do not read the web the way a browser does. A live page is a thicket of navigation menus, cookie banners, sidebars, share buttons, tracking scripts, and layout markup, all wrapped around the few paragraphs that actually carry meaning. When a model or an AI-crawler ingests that page, it has to work to separate signal from chrome, and it does not always succeed. Complex tables, content injected by JavaScript, and dynamic widgets can be misread or missed entirely, which is how a model ends up confidently stating the wrong price or the wrong hours for a business.
llms.txt attacks that problem at the source. By handing the model a clean markdown version of your priorities, you remove the ambiguity. You establish a ground truth: this is who we are, these are the pages that matter, this is what each one says. You also reduce the computational cost of understanding you, and a model that can grasp your site cheaply and unambiguously is a model more likely to retrieve you accurately and cite you correctly. It pairs naturally with the work of structured data, which encodes machine-readable facts, and with keeping your content legible to AI crawlers in the first place. Where structured data labels the facts, llms.txt curates the map.
The Honest State of Adoption
This is where discipline matters, because the topic attracts hype. llms.txt is a proposal, not a ratified standard, and we no longer have to argue about its adoption, because the server logs settle it. An Ahrefs study published in June 2026, drawing on log data from 137,210 domains, found that 97% of all published llms.txt files received no requests at all during May 2026: no bots, no humans, nothing. Among the small remainder that saw any traffic, AI training crawlers accounted for 5.3% of requests, AI assistants 2.5%, and AI retrieval bots 1.1%. Tools built to study llms.txt itself, GEO and AEO checkers plus file validators, accounted for 9.4%: more than every AI training crawler, assistant, and retrieval bot combined. A separate sweep of more than 500 million AI bot visits over a 90-day window found 408 that touched an llms.txt.
The vendors have been equally plain. John Mueller stated in June 2025 that no AI system currently uses llms.txt. On 15 June 2026 Google added a note to its own AI optimization guidance confirming that llms.txt files are not required for Google Search. Neither OpenAI's nor Anthropic's crawler documentation instructs site owners to publish one: both point to robots.txt for crawler management. Meanwhile roughly one site in ten now publishes the file. That gap, widespread publishing against almost no reading, is the entire story.
So the honest case is smaller than the pitch, and it is worth stating exactly. The file is cheap to author and does no harm. Documentation platforms and developer-tool sites, whose audience includes coding agents that do fetch it deliberately, get real use from it. If you have an afternoon spare, publish one. What you must not do is treat it as an AI visibility strategy, count it as work delivered, or pay anyone for it as a service. Anyone selling llms.txt as the fix for how AI sees your business is selling a file that, on the evidence, almost nothing opens.
Why It Stalled: The Meta Keywords Lesson
The adoption numbers are usually blamed on inertia, as though the engines simply have not got around to it yet. That reading is wrong, and the reason matters more than the file does. Set the three root-level files side by side and the difference is structural.
robots.txt survived because it is a directive, and compliance is verifiable by watching what a crawler actually does. sitemap.xml survived because it is a list of URLs, and every entry can be checked by fetching it. llms.txt is neither. It is unverifiable prose about yourself, published in a channel no human visitor ever sees.
The web has run this experiment already. The meta keywords tag was invisible to users and uncheckable against the page, and within a few years it was pure spam and every engine had stopped reading it. A site can write "the leading provider in the region" into its llms.txt today with nothing on the site supporting the claim, and no mechanism exists to catch it. Any channel that is both invisible to users and uncorroborated by the rendered page decays into a spam vector, and engines learn to discount it. The engines paid for that lesson once. The rule they took from it is the one that should shape your work: AI systems weight what they can corroborate. A claim in your llms.txt is worth nothing. The same claim rendered in your page text, mirrored in your structured data, and confirmed by a third party is worth everything.
How to Publish One
Standing up an llms.txt file is an afternoon of work, not a project. The mechanics fall into three plain steps:
Curate, do not dump
List only the pages that genuinely represent you: your core services, your clearest explainers, your pricing and policy pages, your best proof. A short, sharp map beats an exhaustive one. This is an editorial act, not an export.
Write it in the conventional shape
Open with an H1 for the name, add a one-line summary, then group links under H2 headings with a colon-and-description after each. Save it as markdown and place it at your domain root so it resolves at /llms.txt.
Keep it true, and keep it current
The file is only as useful as it is accurate. When your pages, prices, or offers change, update it. A stale map is worse than none, because it teaches a model a fact that is no longer true.
llms.txt does not make you findable. It makes you legible. It cannot force any engine to read it, but for the ones that do, it replaces guesswork with a clean, curated account of who you are, in your own words.
How AIOInsights Reads This Signal
llms.txt sits squarely inside the AI Discoverability pillar: the set of signals that determine whether AI systems can even reach and parse your content before any question of trust or citation arises. AIOInsights checks for the presence and shape of a root-level /llms.txt file as one observable, verifiable signal among several, alongside how your site treats AI crawlers and how much of your meaning is exposed in plain, retrievable text rather than locked in scripts.
That check is real and deterministic, and we want to be exact about what it does. We report llms.txt. We do not score it. Publishing one will not raise your AI Discoverability number, and lacking one will not lower it. It appears in your findings as a neutral note, never as a pass and never as a problem, because on the evidence above we cannot honestly claim it changes how AI systems see you.
The weight it used to carry now sits with the signals that can be corroborated: structured data that mirrors what your page actually says, and your own facts written as plain text a model can read and quote without running JavaScript. We would rather score fewer things accurately than more things generously. Every evaluation reflects what is actually there, nothing guessed and nothing inflated.
Check How AI-Legible Your Site IsKeep reading the lexicon: Structured Data, AI Crawlers, and robots.txt.