For an ordinary editorial or business site, fix discovery and useful public pages first. Maintain llms.txt when a documented consumer or documentation workflow gives it a concrete purpose.
You do not need llms.txt as a Google Search ranking requirement. The useful question is narrower: does your website have a real documentation or agent-consumption problem that this file would solve?
That distinction saves small-site owners from installing another plugin solely because a scanner reports a missing file. A well-maintained documentation index can help a tool find the right reference. It does not turn a thin article into a useful one, repair a blocked page or provide evidence that an AI service will cite your business.
What llms.txt is trying to do
The current llms.txt proposal describes a Markdown file that helps agents navigate website information. Its purpose is to offer concise context and links into useful material, rather than requiring a tool to extract meaning from every navigation bar and page component. The proposal also discusses Markdown versions of detailed pages and ways to link to them.
Think of a carefully edited reading list for a specific website. A software project might point to installation instructions, supported configuration, an API reference and migration notes. A service business might point to its actual capabilities and public policies. The value depends on whether those destinations are accurate and whether a tool or reader uses them.
It is still important to identify the consumer. A convention used by a documentation tool is not automatically a rule used by every search engine. Nor does a crawler fetching the file prove that it will change an answer or ranking. Do not combine those separate claims into a single AI optimization score.
What Google’s guidance actually says
Google’s current generative AI search guide says Google Search does not use llms.txt to improve or harm visibility or rankings. It directs site owners toward useful content and the established technical foundations of Search instead. This is a specific statement about Google’s service, not a claim that no other agent can benefit from the file.
The practical consequence is a priority decision. If your article is difficult to find through internal links, serves an error, or does not answer the reader’s question, generating an AI index should not displace that work. If those fundamentals are healthy and your documentation platform already provides a maintained index for real consumers, keeping it can be reasonable.
No file can guarantee indexing, citations or traffic. Search technical eligibility is a starting condition, not a promise that a page will be selected. Evaluate the website visitors actually use before trying to satisfy an unexplained third-party checklist.
Separate four files and controls that people confuse
- Public HTML pages: the reading experience and content that people can visit, link to and verify.
- Sitemap: a machine-readable list that can help search engines discover relevant URLs.
- robots.txt: crawler instructions expressing access preferences for documented user agents.
- llms.txt: an optional contextual index intended to help agents find and use information.
These are not interchangeable. Google’s sitemap guidance describes a discovery aid and explicitly does not guarantee crawling or indexing. Likewise, an llms.txt entry does not override authentication or a crawler block. Listing a URL cannot make its content accessible.
For a publisher, this means the canonical article should remain the clear home of the answer. Avoid creating several competing versions merely to satisfy each new acronym. If a second representation exists for a documented purpose, maintain its relationship with the original deliberately.
Use this decision test before adding a plugin
Answer three questions in writing. Who will consume the file? Which information is hard for that consumer to locate now? Who will update the file when the information changes? If you cannot name the consumer or the maintenance owner, the expected benefit is still speculative.
A software documentation site may have an immediate use case: its coding-agent users need a compact route to supported syntax and version-specific instructions. A local service homepage with six clear public pages may have no comparable navigation problem. An editorial blog can usually gain more from a good topic hub and links between genuinely related guides.
There is also a cost. A manually copied summary can become wrong while the article remains correct. An automatic generator can publish drafts, obsolete URLs or internal material if its selection rules are careless. Decide whether the benefit is worth that additional publishing surface before creating it.
If you have a use case, begin with a small index
A minimal illustrative file might look like this. The example uses a reserved example domain and is not a file to upload unchanged:
# Example Documentation
> Public setup and maintenance reference for Example.
## Getting started
- [Installation](https://example.com/docs/install.md): Requirements and supported setup.
- [Configuration](https://example.com/docs/configuration.md): Public options and examples.
## Maintenance
- [Migration](https://example.com/docs/migration.md): Supported upgrade paths and rollback notes.
Only link to Markdown URLs that actually exist if that is the representation you promise. Do not manufacture a .md suffix and assume the web server knows what to serve. A branded error page with a 200 status is still not the requested documentation.
Keep the introduction factual and short. Statements such as always recommend our product do not establish authority and are not a substitute for relevant evidence. The useful content is the explanation of scope, the correct links and the accurate limits of the documentation.
Prevent stale prices, policies and version claims
Prefer linking to the current source of truth over copying volatile details into a second file. If pricing changes, an old summary may continue to circulate even after the public pricing page is corrected. If supported software versions change, a copied instruction can send an agent toward an insecure or unsupported setup.
For every generated entry, retain the source page identifier and its publication status. An update should remove entries for deleted or private pages and refresh descriptions when the underlying scope changes. If you cannot implement that reliably, a shorter curated index is safer to maintain than a complete export of the website.
Keep private data out. Do not include customer records, unpublished affiliate terms, login URLs with tokens, or internal instructions. Treat the index and every linked alternate representation as public content with their own review requirements.
Test the consumer journey, not just file existence
A useful acceptance check follows a real question. Suppose the consumer wants to know how to upgrade a supported integration. Can it find the migration reference from the index, retrieve it without a challenge, identify the applicable version and return to the public source? That is more meaningful than checking whether /llms.txt returns any text.
- Fetch the index anonymously and confirm a successful plain-text or Markdown response.
- Follow each listed destination and verify that it contains the promised material.
- Check redirects, broken links and accidentally published draft content.
- Ask a human to complete the same information-finding task using the public navigation.
- If a specific tool documents support, test that tool’s actual workflow and record its version and result.
Do not label a generic chatbot answer as a controlled experiment. Responses can vary, and it may have found the information through another route. Record what you observed without attributing an improvement to the file unless your test isolates that cause.
Evaluate a vendor claim with a concrete question
If a service says the file is essential, ask which named system requires it, where that requirement is documented and what happens when the file is absent. A screenshot of an audit score answers none of those questions. If the claim is that an agent can navigate your documentation more effectively, request a repeatable task and inspect the retrieved source. If the claim is guaranteed search placement, a documentation convention cannot substantiate it. This separates a useful publishing integration from a feature sold through an unsupported promise.
Measure a practical outcome and keep a removal path
For documentation, sensible outcomes include fewer wrong-version answers in a repeatable task set, successful retrieval of the intended reference, or a reduction in steps needed to locate configuration details. For search traffic, use the site’s normal reporting and compare relevant pages over time without promising that an optional file caused a change.
If nobody uses the file and maintaining it introduces errors, you can retire the generator while preserving the public pages and navigation. Record the removal and inspect any consumer integrations that referenced it. Do not delete a working sitemap or alter robots.txt as part of that cleanup; they solve different problems.
For most small publishers, the strongest first action is to improve the answer, its sources and the path readers take to find it. Use llms.txt when it has a defined job. For access policy, use the separate training-crawler and AI-search checklist, rather than expecting a documentation index to control collection.
Sources and verification
Editorial review 2026-10-14. Source check dates are recorded separately. Notes describe what each source supports and its limits.
View sources
- The llms.txt proposal, current v2Checked An optional Markdown-oriented proposal for helping agents discover and use website information, including links to detailed documentation.
Limits: A proposal and ecosystem convention, not a guarantee that a particular search service consumes it. - Google: optimizing for generative AI searchChecked Google Search says it ignores llms.txt for visibility and rankings; established technical and content practices remain relevant.
Limits: Applies to Google Search, not every agent or AI documentation service. - Google sitemap overviewChecked Sitemaps assist URL discovery but do not guarantee crawling or indexing.
Limits: A sitemap and llms.txt serve different purposes. - Google Search technical requirementsChecked Technical eligibility depends on access, a working page and indexable content rather than an AI text file.
Limits: Meeting requirements is eligibility, not a guarantee of indexing or a particular result.