Beyond SEO: Why Independent Publishers Are Using llms.txt
What if your next reader is not human?
Independent publishers have spent decades learning how to publish for two audiences at once. We write and design pages for people, while quietly maintaining another layer for search engines: metadata, canonical links, XML sitemaps, structured data and robots.txt. Most readers never see any of it, yet it helps machines understand what we have published.
This is also a continuation of an idea I explored earlier in Adapting to the Age of AI: independent publishers should not treat AI simply as something happening around them, but as a change in the publishing environment that is worth actively experimenting with. llms.txt gives that argument a surprisingly practical new direction.
Now there is another machine audience. AI assistants and agents increasingly find websites, read articles, synthesise information and recommend sources to users. That raises an interesting question for independent publishers: if an AI is going to read my website anyway, why should I leave it to guess what the publication is about?
We already publish for machines
Publishing information specifically for machines is not a strange new concession to artificial intelligence. We have been doing it for years. A sitemap says where our pages are. A canonical link says which version should be treated as the original. robots.txt provides instructions to web crawlers.
These mechanisms are useful, but mostly structural. They can tell a machine that an article exists without explaining why that article matters, how it relates to the rest of the publication or what sort of publication it belongs to.
This is where llms.txt becomes interesting.
Originally proposed by Jeremy Howard in 2024 and revised in 2026, llms.txt is a simple Markdown file that can be placed on a website to provide AI agents with concise information and links to important material. It remains an emerging convention rather than a replacement for established web standards, and support across AI systems is still developing.
The technical idea is almost disarmingly simple. But for an independent publisher, the more interesting question is not how to create the file. It is what you choose to say in it.
Why let AI guess what your publication is?
Consider what happens when an AI encounters a small independent website for the first time. It has to infer whether the site is a personal blog, a magazine, an archive, a specialist reference work or something else entirely. It must work out which articles are important, which pages are merely navigational and whether several apparently separate articles actually form part of a larger collection.
Large publishers have enormous external footprints helping machines make those judgements. Smaller publications often do not.
For an independent publisher, leaving that interpretation entirely to machines begins to look like an unnecessary gamble.
When I added llms.txt to my own publications, I quickly found that merely listing pages seemed like a wasted opportunity. I already have a sitemap for that. Instead, I started treating the file as a guided introduction to each publication.
Give the machine a guided tour
For PhilReichert.org, I wanted an AI reader to understand that it is an independent digital publication covering interconnected interests such as computing, music technology, making and experimentation, rather than automatically reducing a domain carrying my name to the category of āpersonal blogā.
Phillaneum required something different. Its llms.txt explains that Victoriana means more than famous Victorians and major historical events. It directs attention towards everyday life, technology, folklore, patents, artefacts, primary sources and forgotten ideas. It also identifies cornerstone articles and encourages an AI researcher to follow related reading rather than treating every page as an isolated search result.
This also extends the thinking behind The Three Link Rule, where I argued that articles gain value when they are deliberately connected into reader pathways rather than left as isolated pages. An llms.txt file potentially exposes some of that same editorial structure to AI, telling a machine not only what exists, but which relationships are worth following.
Kurtz Vintage Vibes needed another interpretation again. There, the emphasis is on the artefact itself. A piece of historical sheet music is not merely a document containing notes. It can also reveal illustration, publishing, popular taste, domestic entertainment and cultural history. The machine-readable guidance reflects that editorial philosophy.
That led me to a much more useful way of thinking about llms.txt:
It can be more than a description of your website. It can be a guided tour written specifically for machines.
Is that marketing?
I think it can be.
Marketing does not have to mean advertising. At its most basic, it is about deliberately presenting what you offer to an audience rather than leaving that audience to define you for itself.
For an independent publisher, the desirable sequence is straightforward: an AI discovers the publication, understands what it specialises in, selects an appropriate article, preserves the provenance of distinctive work and, where useful, directs the human reader back to the original source.
llms.txt does not guarantee any of that. There is no evidence that publishing one will suddenly improve search rankings or produce a flood of AI referrals, and support varies between systems. It should not be treated as magical āAI SEOā.
But that is not the same as saying it has no value. A well-written llms.txt gives a machine useful editorial context at negligible cost. For an independent publisher, that is enough to make experimentation worthwhile.
Do not write for AI. Publish something for AI.
There is also an important lesson here from the history of search optimisation.
Once publishers discovered that search engines could drive enormous audiences, many began changing the way they wrote for humans in order to satisfy machines. The result was frequently bloated articles, repetitive phrasing and pages designed around presumed algorithmic preferences rather than readers.
There is no reason to repeat that mistake with AI.
Do not make your articles sound like an LLM. Make it easier for an LLM to understand your articles.
Keep the article written for the person who chose to read it. Put machine-oriented guidance in the machine-readable layer around it. Tell AI what the publication is, where its strongest material lives, how collections relate to one another and how original work should be attributed.
That separation is particularly attractive to independent publishers because it preserves editorial voice while improving machine legibility.
The independent publisher's advantage
Nobody yet knows how important llms.txt will ultimately become. It is still a proposal, the ecosystem is developing quickly and different AI systems will treat these files differently.
That uncertainty is not necessarily a disadvantage for an independent publisher.
Large organisations may need policies, committees, platform changes and implementation projects before experimenting with a new publishing convention. An independent publisher can create a small text file this afternoon, publish it, refine it next week and learn while the conventions are still forming.
For decades, machines have told publishers how they would like websites structured. llms.txt offers an intriguing reversal: publishers can begin telling machines how they would like their publications understood.
Tell them who you are. Tell them what matters. Show them where to start.
There is a larger experiment behind all of this: what happens when a publication is deliberately built so that each new piece strengthens what already exists? That is the question I explore in Build Your Own Idea Mesh, and it is a useful place to continue if you are thinking beyond individual articles towards the long-term value of your own publishing archive.
Writer's Notes
Writing this article shifted my view of llms.txt from a small technical convention to something much more interesting: a new publishing surface. I began with the assumption that it was simply another machine-readable file, perhaps analogous to robots.txt or a sitemap. The realisation came when I started writing one for my own publications. I was no longer just listing pages; I was explaining what the publication was, which material mattered, how articles related to one another, and how I wanted that body of work to be understood.
That is where the idea became much more compelling to me. Independent publishers have spent years adapting to systems that classify, rank and interpret our work from the outside. llms.txt suggests the possibility of participating more directly in that interpretation. I can describe the editorial intent of a site, identify cornerstone material, distinguish original analysis from source material, and give an AI agent a much better starting point than a collection of disconnected URLs. Whether every system will act on that guidance is still uncertain, but the underlying publishing idea is strong enough to be worth exploring.
I also see this as an evolving field rather than a finished technique. AI discovery, agents, machine-readable publishing and attribution are all changing quickly, and the useful conventions may look quite different a year from now. That makes this an unusually fertile area for continued experimentation. I expect to keep refining these files, testing stronger editorial guidance, improving the relationships between articles and watching how AI systems respond. The interesting question is no longer simply how to make a website readable by machines, but how much of a publication's structure, intent and editorial character we can deliberately communicate to them.
Reader Guide
The following material expands on the terminology, historical context, technical concepts, and related reading connected to this article.
Glossary
Some of the terms used in this article have specialised, historical or technical meanings. This glossary provides additional context for selected terms and ideas.
- llms.txt
- A Markdown file placed at a website root that gives AI agents concise editorial guidance and links to important material; originally proposed by Jeremy Howard in 2024 and revised in 2026, it is an emerging convention (not a formal web standard) intended to supply machines with editorial context that sitemaps and structural files do not.
- robots.txt
- A plainātext file at a site's root that tells automated web crawlers which areas of the site they are allowed or asked not to crawl; it is an advisory protocol used to manage crawler behaviour and discovery, but it does not enforce access control and should not be relied on for security or for communicating editorial intent.
- canonical link
- An HTML link element (commonly rel="canonical") that marks a preferred or āoriginalā URL among duplicate or similar pages, helping search engines and agents consolidate indexing, avoid splitting authority between copies and attribute the correct source when multiple versions exist.
- XML sitemap
- A machineāreadable XML document that lists a site's URLs (often with optional metadata such as last modified dates and change frequency) to help crawlers discover and index pages; it is primarily structural and does not explain editorial priorities or relationships between articles.
- LLM
- Abbreviation of 'large language model' ā a statistical AI trained on vast amounts of text to predict and generate natural language; LLMs power many assistants and summarisation agents, but they infer meaning probabilistically and can misclassify or lose provenance unless given explicit, machineāoriented guidance.
Frequently asked questions
Curious about something youāve just read? These frequently asked questions explore some of the key ideas, details and questions surrounding the topic.
What is llms.txt?
llms.txt is a simple Markdown file that can be placed on a website to provide AI agents with concise information and links to important material; it was proposed by Jeremy Howard in 2024 and revised in 2026 and is an emerging convention adopted by some major documentation platforms and technology companies.
Why should an independent publisher create an llms.txt file?
An llms.txt helps machines understand what a publication is, highlights cornerstone articles and collections, preserves provenance, and guides AI readers toward relevant material; it is low-cost to publish and allows independent publishers to experiment quickly while conventions are still forming.
What kinds of content or guidance belong in an llms.txt?
It can act as a guided tour for machines by describing the publicationās scope and editorial philosophy, directing attention to important or related articles, explaining how collections relate, and indicating how original work should be attributed.
Will publishing an llms.txt guarantee better search rankings or lots of AI referrals?
No ā there is no evidence that an llms.txt will automatically improve search rankings or produce a flood of AI referrals and support varies between systems, though a well-written file can still provide useful editorial context at negligible cost.
Related articles…
References
The following primary reference provides the technical background and current proposal for the llms.txt convention discussed in this article.
- Howard, Jeremy. The /llms.txt File, v2. llms.txt proposal and specification, originally published 3 September 2024 and updated 10 August 2026.
Disclosure
llms.txt is an emerging convention, not a universally supported web standard. AI systems may ignore, interpret or use these files differently, and no claims are made that implementing one will improve search rankings, AI citations, recommendations or website traffic.