Written by: Mariana Fonseca, Editorial Team, AI Growth Agent
Key Takeaways for llms.txt Implementation
- llms.txt is a plain Markdown file at the domain root that gives AI agents a curated, noise-free map of a site’s most important pages. llms-full.txt inlines full page content for agents that prefer a single large fetch.
- The file is not a standalone ranking signal. Its value comes from using it inside a broader agentic technical SEO stack that also includes schema, bot tracking, and incremental visibility reporting.
- Successful implementation requires domain-root hosting with correct text/plain headers, explicit Allow rules in robots.txt for AI crawlers, and verification that every listed URL returns HTTP 200.
- Common pitfalls include incorrect MIME types, redirect chains, stale caches, and missing blockquotes after the H1. These issues prevent AI agents from reading the file correctly.
- When you want llms.txt handled for you, AI Growth Agent sets up your agentic SEO stack, then ships your first AI-ready article within a week.
What You Need Before You Create llms.txt
Confirm a few technical and content prerequisites before you write any Markdown for llms.txt.
- Control of the domain and the ability to place a file at the root path, so
https://yourdomain.com/llms.txtis reachable without authentication. - A current robots.txt that explicitly allows the AI crawlers you want to reach the file: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, and Applebot-Extended. This prevents silent blocking.
- A list of 10 to 50 canonical public pages that best represent the brand, its products, and its documentation. Exclude login-walled, paginated, or duplicate URLs so agents see only stable, public content.
- Brand guidelines and a one-to-three sentence summary of what the site is and who it serves. This blockquote is the single highest-leverage element in the file because models often quote it directly.
- Access to server configuration, a CDN dashboard, or a platform file manager so you can set the correct Content-Type header.
- For llms-full.txt, the full Markdown body of each priority page. Keep the combined file under 200,000 tokens (roughly 150,000 words) so models can ingest it in one pass.
Existing schema markup and a clean sitemap.xml are optional for shipping llms.txt but required for a complete agentic technical SEO stack. llms.txt performs best when the pages it highlights are already structured for AI retrieval.
From Draft to Deployment: Process Overview
The implementation workflow follows three phases: create the file, host it at the root with correct headers, and verify behavior. Expanded into a production-ready sequence, those phases become seven concrete steps.
Delays usually appear at two handoff points. Content teams often finish the Markdown while engineering has not yet configured headers. After deployment, teams sometimes forget to verify Content-Type, so the file ships but returns HTML or a download prompt. The checklist in Step 7 prevents both problems.
Step-by-Step Guide
Step 1: Choose the Pages That Deserve Citations
Start in a spreadsheet and list every canonical public page worth citing. Treat this as a curated index, not a sitemap. Current best practice is 10 to 50 curated links organized into sections.
For a typical SaaS product, this usually includes core product pages, key documentation, comparison pages, and a small set of high-authority blog posts. Exclude paginated archives, session-bound URLs, and any page that redirects so agents do not waste context on unstable locations.
Confirm every selected URL returns HTTP 200 in an incognito window. Every listed link must return HTTP 200; a stale file actively misleads AI agents about site content.
Step 2: Draft the llms.txt File in Markdown
Open VS Code, Notepad++, or any plain-text editor and create a new file. Save it as exactly llms.txt (lowercase, no .md extension) with UTF-8 encoding. The llmstxt.org specification requires this structure:
# Brand Name > One to three sentences describing the site, its audience, and what > visitors can accomplish here. This blockquote is what LLMs lift > directly as the "what is this company" answer for citations. ## Core Pages - [Home](https://example.com/): Overview of the platform and its value proposition. - [Product](https://example.com/product/): Full feature set and use cases. - [Pricing](https://example.com/pricing/): Plan comparison and enterprise options. ## Documentation - [Getting Started](https://example.com/docs/getting-started/): Quickstart guide for new users. - [API Reference](https://example.com/docs/api/): Full endpoint documentation with examples. ## Optional - [Blog](https://example.com/blog/): Articles on industry trends and product updates. - [Changelog](https://example.com/changelog/): Release history and version notes. Follow these key rules from the specification.
- Use exactly one H1 containing the brand or project name.
- Place a blockquote immediately after the H1. Skipping this blockquote is the single highest-leverage error because LLMs lift it directly as the “what is this company” answer.
- Group links into H2 sections by topic or function so agents can scan quickly.
- Format each link as
- [Title](URL): one-line descriptionto keep parsing simple. - Add an optional
## Optionalsection at the bottom for lower-priority content that LLMs may skip when context is limited. - Use only absolute URLs with
https://, not relative paths. - Keep the file under 10 KB. By April 2026, best practices solidified around short files under 5 KB that lead with an authoritative blockquote.
Once you have a clean llms.txt that lists your priority pages, you can extend it with a companion file that exposes full content for single-fetch agents.
Step 3: Create llms-full.txt for Full-Content Fetches
llms-full.txt concatenates the full Markdown body of each page listed in llms.txt, separated by --- dividers, and is served at /llms-full.txt. Profound’s agent traffic analysis shows AI agents visit llms-full.txt files more than twice as frequently as standard llms.txt files.
# Brand Name - Full Documentation --- ## Home [Full Markdown content of the home page] --- ## Product [Full Markdown content of the product page] --- Keep llms-full.txt under 200,000 tokens so common models can load it in one context window. For larger sites, include only the highest-priority pages instead of mirroring the entire site.
When manual file creation becomes tedious, AI Growth Agent automates llms.txt and llms-full.txt generation, then tracks how agents actually use those files.
Step 4: Host llms.txt at the Domain Root With Correct Headers
Serve the file at exactly https://yourdomain.com/llms.txt and return HTTP 200 from that path. Recommended Content-Type headers are text/plain; charset=utf-8 or text/markdown; application/octet-stream is incorrect because it forces a download. Use a caching header such as Cache-Control: public, max-age=3600 to reduce load while keeping the file fresh.
Use these platform-specific patterns as a starting point.
WordPress (self-hosted): Upload llms.txt via SFTP directly to the root directory, the same folder that contains wp-config.php. Yoast SEO and Rank Math both added native llms.txt generation in 2025; once toggled on in the plugin dashboard, the file is generated and updated automatically with no FTP access required.
If you rely on a plugin, confirm the Content-Type is text/plain and not text/html. The Website LLMs.txt plugin can assist with proper file generation; verify that your permalink structure does not redirect the file.
Framer: Framer supports hosting llms.txt on Pro, Scale, and Enterprise plans by uploading the file through the Domains > Files section in the dashboard; after publishing the site, the file becomes available at https://example.com/llms.txt. Framer launched its Static Files feature on March 23, 2026, which consolidated well-known file management into the Files tab under Domains. Uploading requires a paid plan.
For dynamically generated content, create a route handler at src/app/llms.txt/route.ts:
export async function GET() { const content = await generateLlmsTxt(); // your generation logic return new Response(content, { headers: { 'Content-Type': 'text/plain; charset=utf-8', 'Cache-Control': 'public, max-age=3600', }, }); } To set explicit headers for the static file, add this to next.config.mjs:
async headers() { return [ { source: '/llms.txt', headers: [ { key: 'Content-Type', value: 'text/plain; charset=utf-8' }, { key: 'Cache-Control', value: 'public, max-age=3600' }, ], }, { source: '/llms-full.txt', headers: [ { key: 'Content-Type', value: 'text/plain; charset=utf-8' }, { key: 'Cache-Control', value: 'public, max-age=3600' }, ], }, ]; } Vercel (framework-agnostic): Place llms.txt in public/llms.txt; when not using a framework, drop the file in the project root and add a vercel.json rewrite rule:
{ "rewrites": [ { "source": "/llms.txt", "destination": "/llms.txt" } ], "headers": [ { "source": "/llms.txt", "headers": [ { "key": "Content-Type", "value": "text/plain; charset=utf-8" }, { "key": "Cache-Control", "value": "public, max-age=3600" } ] } ] } Cloudflare Pages: Save llms.txt as public/llms.txt; Pages automatically serves the file with text/plain content type.
For a Cloudflare Worker with no traditional host:
export default { async fetch(request) { const url = new URL(request.url); if (url.pathname === '/llms.txt') { return new Response(LLMS_TXT_CONTENT, { headers: { 'content-type': 'text/plain; charset=utf-8' }, }); } }, }; Nginx:
location = /llms.txt { default_type text/plain; add_header Content-Type "text/plain; charset=utf-8"; add_header Cache-Control "public, max-age=3600"; } Apache (.htaccess):
AddType text/plain .txt <Files "llms.txt"> Header set Cache-Control "public, max-age=3600" </Files> Step 5: Fix 301 Redirects and llm.txt Variants
Some tools and agents request /llm.txt (without the “s”) or /llms.md. Add 301 redirects from these variants to the canonical path so they resolve correctly.
For Nginx:
location = /llm.txt { return 301 /llms.txt; } location = /llms.md { return 301 /llms.txt; } For Apache (.htaccess):
Redirect 301 /llm.txt https://yourdomain.com/llms.txt Redirect 301 /llms.md https://yourdomain.com/llms.txt For Vercel (vercel.json):
{ "redirects": [ { "source": "/llm.txt", "destination": "/llms.txt", "permanent": true }, { "source": "/llms.md", "destination": "/llms.txt", "permanent": true } ] } If a subpath is technically required, implement a 301 redirect from /llms.txt to the actual location; the root placement remains the canonical convention. Never serve llms.txt through a redirect chain from the canonical path itself, because a redirect returns the wrong HTTP status code and prevents AI crawlers from accessing the file.
Step 6: Update robots.txt and Reference llms.txt
Confirm that robots.txt does not block the AI crawlers you want to reach llms.txt. Add explicit Allow directives for the file.
User-agent: GPTBot Allow: /llms.txt Allow: /llms-full.txt User-agent: ClaudeBot Allow: /llms.txt Allow: /llms-full.txt User-agent: PerplexityBot Allow: /llms.txt Allow: /llms-full.txt User-agent: OAI-SearchBot Allow: /llms.txt Allow: /llms-full.txt User-agent: Google-Extended Allow: /llms.txt Allow: /llms-full.txt User-agent: Applebot-Extended Allow: /llms.txt Allow: /llms-full.txt After placing llms.txt at the domain root, add a comment referencing the file inside robots.txt to aid future crawler discovery. Add a line such as # LLM index: https://yourdomain.com/llms.txt near the top of robots.txt.
Sites with robots.txt blocking AI crawlers showed zero citation improvement after adding llms.txt, so this step is mandatory.
When you want robots.txt rules, headers, and discovery files kept in sync, AI Growth Agent manages these technical details and reports back in plain language.
Step 7: Verify Accessibility, Headers, and Content
Run a short verification checklist before you consider the implementation complete.
Check HTTP status and headers:
curl -I https://yourdomain.com/llms.txt Expected output includes:
HTTP/2 200content-type: text/plain; charset=utf-8(ortext/markdown)cache-control: public, max-age=3600
Check the content body:
curl https://yourdomain.com/llms.txt Confirm the raw Markdown content is returned, not HTML. Open the URL in an incognito browser window and confirm the file renders as plain text, not a download prompt and not an HTML page.
Repeat both checks for /llms-full.txt and for the redirect variants /llm.txt and /llms.md. For Content-Type verification, you can reuse the same approach described in the WordPress and hosting sections.
Validate the file structure at llms-txt-validator.dev, which scores structure, link reachability, and best practices on a 0 to 100 scale.
Confirm every URL listed inside llms.txt returns HTTP 200 by running a link checker or manually spot-checking each one.
Common Mistakes and Troubleshooting
Incorrect MIME Type
MIME type errors are the single most common failure mode in plugin-generated llms.txt deployments, where servers incorrectly serve the file as text/html. Diagnose this by running curl -I https://yourdomain.com/llms.txt and checking the content-type header.
Fix the issue by overriding server configuration in .htaccess, nginx.conf, or the platform’s header settings. Serving llms.txt with a text/plain MIME type is acceptable and does not cause Lighthouse to fail, provided the file content uses proper Markdown link syntax.
Redirect Chains and 404 Errors
The canonical path must return HTTP 200 directly. Any redirect from /llms.txt to another path breaks AI crawler access and can cause missed citations.
If the file is hosted at a subpath for technical reasons, implement a 301 from /llms.txt to that location, not the reverse. After deployment, ensure robots.txt does not contain Disallow: /llms.txt; add Allow: /llms.txt above any Disallow rules if blocking is detected.
Stale Cache Serving Old Content
Fix caching issues by configuring caching plugins to exclude llms.txt from caching and setting Cloudflare Page Rules to Cache Level: Bypass for the llms.txt path. During active iteration, use Cache-Control: public, max-age=60 and switch to max-age=3600 once the file is stable.
Measurement Gaps
The absence of bot requests to /llms.txt does not always mean the file is broken. OtterlyAI’s 90-day experiment found that only 84 of 62,100+ total AI bot visits targeted /llms.txt directly, while agents overwhelmingly discovered content via standard web pages.
Measure effectiveness through three lenses: server log analysis filtered by AI crawler user agents, citation rate tracking across ChatGPT, Perplexity, and Claude before and after deployment, and coverage breadth testing using a broader query set.
| Error | Symptom | Fix |
|---|---|---|
| Wrong MIME type | curl shows content-type: text/html | Override in .htaccess, nginx.conf, or platform headers |
| File not at root | 404 at /llms.txt | Move file to domain root or add 301 from /llms.txt |
| Redirect chain on canonical path | 301 or 302 on /llms.txt itself | Serve 200 directly from /llms.txt |
| AI crawlers blocked in robots.txt | Zero bot requests in server logs | Add Allow: /llms.txt for each AI crawler user agent |
| Stale CDN cache | Old content returned after update | Purge CDN cache; set Cache-Control: max-age=60 during iteration |
| Broken links inside file | Validator reports link errors | Audit all URLs monthly; remove or update broken links |
| Non-Markdown link format | Lighthouse audit fails with “no links found” | Use – [Title](URL): description format for every link |
| Missing blockquote after H1 | Low validator score; weak citation context | See the blockquote requirement in Step 2 and add a two-sentence summary |
Verifying Outcomes and Measuring Results
Measurement of llms.txt effectiveness requires combining server log analysis with citation tracking across AI platforms. No single tool currently provides the complete picture.
Server logs act as the authoritative source for AI crawler access. Filter access logs by known AI crawler user agents including GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, and Applebot-Extended. Track requests to /llms.txt and /llms-full.txt separately, and monitor whether crawlers subsequently visit the pages listed inside the file.
Citation rate tracking compares how often the domain is cited in AI-generated answers before and after deployment. Use a fixed query set of 15 to 20 topic-relevant questions per platform and test across ChatGPT, Perplexity, and Claude on a weekly cadence. A 4 to 8 week lag is expected between deployment and measurable citation changes.
Google Search Console provides an independent audit of impressions and clicks for the pages listed in llms.txt. Bot analytics tools that track AI crawler visits per article, such as those built into AI Growth Agent’s WordPress plugin, show exactly when ChatGPT cites specific content and where.
Use a simple review cadence: verify headers and HTTP status on every deploy, audit all links inside the file monthly, and review the full file quarterly or whenever major new content is published.
If you want this monitoring handled continuously, AI Growth Agent tracks AI bot behavior and citation lift, then rolls the findings into your content roadmap.
Advanced Scenarios and Next Steps
Multiple domains and subdomains: Each subdomain requires its own separate llms.txt file; docs.example.com/llms.txt is distinct from example.com/llms.txt. Models do not inherit files from the apex domain.
For multilingual sites, serve one llms.txt per language root (for example, /en/llms.txt and /fr/llms.txt) plus a default at the root rather than mixing languages in a single file.
Dynamic generation in CI/CD: For sites with frequent content changes, generate llms.txt at build time from the same source of truth used for sitemap.xml. Include a link checker in the CI/CD pipeline that runs on every deploy to keep all referenced URLs valid. Ship llms.txt via CI/CD pipelines that include a link checker running on every deploy.
MCP integration: llms.txt is one component of a complete agentic technical SEO stack. Blog MCP exposes schema, manifest, discovery, and capability guidance to agents directly. OpenAI discovery and Agent Card guidance served via /.well-known/ complement llms.txt by providing structured agent onboarding.
Natural language query parameters via /?s={query} allow agents that pass a query straight into the URL to receive a tailored, internally linked response. These components work together: llms.txt provides the curated map, MCP provides the structured interface, and bot tracking closes the measurement loop.
Frequently Asked Questions
Does llms.txt actually improve AI citations?
The evidence is mixed and depends heavily on implementation quality and the AI platform in question. Multiple large-scale studies have found no statistically significant correlation between llms.txt presence and citation rates when controlling for domain authority.
Studies that focus on implementation quality rather than mere presence have found meaningful improvements, particularly for Perplexity and ChatGPT with Browse, and especially for sites with high-quality blockquote summaries and well-described page listings. The file is most effective when it supports a broader agentic SEO strategy that also includes schema markup, bot tracking, and regular content audits.
