llms.txt
A single markdown file, placed at the root of your domain at /llms.txt, that hands AI systems a clean, curated map of your most important content in plain text. It does for language models roughly what robots.txt and sitemap.xml did for search crawlers: it offers a direct, machine-readable invitation to understand your site on your terms rather than leaving models to guess from tangled HTML.
llms.txt is a proposed standard that lets you write the table of contents an AI reads first, curating what matters instead of hoping a model correctly parses your entire site.
What llms.txt Actually Is
llms.txt is a proposed web standard, introduced by Jeremy Howard of Answer.AI in September 2024. The idea is disarmingly simple: publish a plain markdown file at your domain root, reachable at /llms.txt, that describes what your site is and points to the pages that matter most. It sits beside the two files the web already agreed on years ago. robots.txt tells crawlers where they may and may not go, and sitemap.xml lists every URL for indexing. llms.txt is different in intent: it is not a list of permissions or an exhaustive URL dump, it is a human-curated briefing written specifically to be read by a large language model.
The format is deliberately lightweight. A conventional file opens with an H1 carrying the site or project name, an optional blockquote giving a one-sentence summary, then H2 sections that group links by category: documentation, guides, products, policies, key articles. Each list item is a markdown link, ideally followed by a colon and a short description of what the reader will find there. Because it is markdown, it is trivial for a model to parse and cheap for it to consume. There is no schema to validate, no rendering engine required, no JavaScript to execute.
The proposal also defines a companion file, llms-full.txt, and the distinction is worth holding onto. llms.txt is the curated index: a short map that points outward to your real pages. llms-full.txt is the curated content: the actual text of those key pages concatenated into one long markdown document, so a model can ingest your substance in a single fetch without crawling link by link. Think of llms.txt as the contents page and llms-full.txt as the whole book pressed flat.
Watch: LLMs.txt - What It Is and Do You Need One? by Edward Sturm, an honest walkthrough of the proposal, its promise, and the real state of adoption. Source: YouTube.
Why It Matters for Whether AI Can Find and Cite You
Language models do not read the web the way a browser does. A live page is a thicket of navigation menus, cookie banners, sidebars, share buttons, tracking scripts, and layout markup, all wrapped around the few paragraphs that actually carry meaning. When a model or an AI-crawler ingests that page, it has to work to separate signal from chrome, and it does not always succeed. Complex tables, content injected by JavaScript, and dynamic widgets can be misread or missed entirely, which is how a model ends up confidently stating the wrong price or the wrong hours for a business.
llms.txt attacks that problem at the source. By handing the model a clean markdown version of your priorities, you remove the ambiguity. You establish a ground truth: this is who we are, these are the pages that matter, this is what each one says. You also reduce the computational cost of understanding you, and a model that can grasp your site cheaply and unambiguously is a model more likely to retrieve you accurately and cite you correctly. It pairs naturally with the work of structured data, which encodes machine-readable facts, and with keeping your content legible to AI crawlers in the first place. Where structured data labels the facts, llms.txt curates the map.
The Honest State of Adoption
This is where discipline matters, because the topic attracts hype. llms.txt is a proposal, not a ratified standard, and adoption by the major AI engines is still partial. As of now there is no public confirmation that Google, OpenAI, or Anthropic systematically fetch and privilege /llms.txt during retrieval, and some search voices have argued it duplicates work that existing crawlers already do. It is genuinely emerging: real, growing, and not yet universally consumed. Anyone promising that an llms.txt file will vault you up the AI rankings tomorrow is selling certainty that does not exist.
The honest case for adopting it anyway is a cost-benefit one. The file is cheap to author, it does no harm, it is trivially easy for any model or agent to consume if it chooses to, and adoption among documentation platforms and developer tools is already visible. It is a low-cost, forward-leaning bet: a direct invitation that some systems accept today and more may accept tomorrow. You are not betting the business on it. You are leaving the door open and putting a clean welcome mat in front of it.
How to Publish One
Standing up an llms.txt file is an afternoon of work, not a project. The mechanics fall into three plain steps:
Curate, do not dump
List only the pages that genuinely represent you: your core services, your clearest explainers, your pricing and policy pages, your best proof. A short, sharp map beats an exhaustive one. This is an editorial act, not an export.
Write it in the conventional shape
Open with an H1 for the name, add a one-line summary, then group links under H2 headings with a colon-and-description after each. Save it as markdown and place it at your domain root so it resolves at /llms.txt.
Keep it true, and keep it current
The file is only as useful as it is accurate. When your pages, prices, or offers change, update it. A stale map is worse than none, because it teaches a model a fact that is no longer true.
llms.txt does not make you findable. It makes you legible. It cannot force any engine to read it, but for the ones that do, it replaces guesswork with a clean, curated account of who you are, in your own words.
How AIOInsights Reads This Signal
llms.txt sits squarely inside the AI Discoverability pillar: the set of signals that determine whether AI systems can even reach and parse your content before any question of trust or citation arises. AIOInsights checks for the presence and shape of a root-level /llms.txt file as one observable, verifiable signal among several, alongside how your site treats AI crawlers and how much of your meaning is exposed in plain, retrievable text rather than locked in scripts.
That check is real and deterministic. We fetch the file, or record its absence, and we do not invent a score or reward you for a standard the major engines have not yet universally adopted. We report the observable truth: whether you have offered AI a clean map, and whether the rest of your site backs it up. Every evaluation reflects what is actually there, nothing guessed and nothing inflated.
Check How AI-Legible Your Site IsKeep reading the lexicon: Structured Data, AI Crawlers, and robots.txt.