Agent Enabled Sites Implementation: Production Checklist

July 5, 2026

Written by: Mariana Fonseca, Editorial Team, AI Growth Agent

7-Step Deployment Checklist

Validate domain ownership, TLS, sitemap.xml, and robots.txt baseline
Author and deploy llms.txt and llms-full.txt at the site root
Publish /.well-known/ discovery files (MCP Server Card, Agent Card, API Catalog)
Expose a production MCP endpoint with OAuth 2.1 and Streamable HTTP transport
Configure natural-language query parameters and Markdown delivery for agent crawlers
Integrate the four observability pillars: Search Intelligence, AI Analytics, Bot Tracking, and AI Ranking
Validate end-to-end with live agent probes and incremental visibility reporting

Key Takeaways for Agent-Ready Sites

Agent-enabled sites rely on a complete technical SEO stack so AI agents can discover, read, and cite content without human mediation. That stack includes llms.txt, MCP endpoints, and /.well-known/ discovery files.
The 2026 agentic surface expects OAuth 2.1 authentication, Streamable HTTP transport, and structured Markdown delivery instead of traditional crawler-only optimization.
Four observability pillars must be instrumented before deployment to measure incremental visibility and prove ROI in a zero-click environment: Search Intelligence, AI Analytics, Bot Tracking, and AI Ranking.
Production MCP servers need strict security controls such as PKCE, Dynamic Client Registration, rate limiting, and audit logging to prevent vulnerabilities and keep agent connections compliant.
AI Growth Agent provisions the complete agentic technical SEO stack automatically within the first week; see the full implementation live in a walkthrough.

Prerequisites and Starting Conditions

Confirm a solid technical foundation before you add any agentic layer. These conditions keep later steps predictable and testable.

Domain ownership and DNS control. You must be able to add records and configure path rewrites at the registrar or CDN level (Cloudflare, Vercel, or equivalent).
TLS on all endpoints. Production MCP implementations require TLS for all remote transports, and compliant agents reject unencrypted endpoints.
Existing sitemap.xml and robots.txt. These files define the baseline crawl surface. The agentic layer extends this surface instead of replacing it.
Access to the four pillars of observability. Search Intelligence, AI Analytics, Bot Tracking, and AI Ranking must be instrumented before go-live so validation in Step 7 has data to read.

Process Overview: From Discovery to Measured Impact

With those baseline conditions confirmed, the deployment follows a specific sequence that prevents blind spots and failed connections. Discovery files must exist before agents can find your MCP endpoint, and observability must be live before you test whether agents use what you built.

The deployment flow moves from discovery files outward to callable endpoints and then to full observability. Discovery files such as llms.txt and /.well-known/ establish the map agents follow. The MCP endpoint then exposes tools agents can invoke. Natural-language query parameters and Markdown delivery give agents structured, citable responses. Observability closes the loop by confirming that agents reach the site, call tools successfully, and surface the brand in AI answers with measurable frequency.

Step-by-Step Guide to Agent Enablement

Step 1: Validate Domain Baseline

Goal: Confirm that TLS, sitemap.xml, and robots.txt are production-ready before you add any agentic files.

Actions: Run an SSL certificate check on all subdomains and subdirectories that will serve agentic content. Confirm sitemap.xml returns HTTP 200 with Content-Type: application/xml. Confirm robots.txt returns HTTP 200 with Content-Type: text/plain.

Roles: DevOps or platform engineer.

Validation: All endpoints return correct status codes and Content-Type headers. The TLS certificate is valid and not expiring within 30 days.

robots.txt additions for agentic crawlers:

User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://yourdomain.com/sitemap.xml Sitemap: https://yourdomain.com/web-stories-sitemap.xml

Step 2: Author and Deploy llms.txt and llms-full.txt

Goal: Give AI agents a curated, token-efficient index of the site and a self-contained full export for ingestion pipelines.

Actions: Place llms.txt at the site root. The standard format opens with an H1 for the site name, a blockquote summarizing the product in one or two sentences, and H2 sections containing curated lists of links with descriptive text, which keeps token counts low for conversational agents. Place llms-full.txt at the same root level with complete page content embedded so agents that prefer one-pass ingestion can read everything without extra HTTP requests. llms.txt provides a compact overview suited to token-limited assistants, while llms-full.txt embeds complete descriptions for agents preferring self-contained context, so the two files serve different consumption patterns.

llms.txt example:

# Brand Name > One-sentence description of what the site covers and who it serves. ## Getting Started - [Product Overview](https://yourdomain.com/overview): What the product does and who it is for. - [Quickstart Guide](https://yourdomain.com/quickstart): Step-by-step setup for new users. ## Core Topics - [Topic A](https://yourdomain.com/topic-a): Authoritative guide covering X, Y, and Z. - [Topic B](https://yourdomain.com/topic-b): Comparison of approaches with validated data. ## API and Integration - [API Reference](https://yourdomain.com/api): Full endpoint documentation with authentication details.

Roles: Technical content lead and platform engineer.

Validation: Load llms.txt into Claude or a compatible agent and request a specific integration task. Confirm the agent returns accurate, on-brand responses without hallucination. Stale llms.txt files generate broken code and incorrect agent behavior, so version-control the file and set alerts for unauthorized changes.

Watch AI Growth Agent provision llms.txt and llms-full.txt automatically on day one.

Step 3: Publish /.well-known/ Discovery Files

Goal: Enable AI clients to auto-configure connections to the site’s MCP server, agent capabilities, and API catalog without prior integration.

Actions: Publish three files under /.well-known/ following RFC 8615 conventions.

/.well-known/mcp/server-card.json (per SEP-1649):

{ "serverInfo": { "name": "Brand MCP Server", "version": "1.0.0", "description": "Exposes brand content and tools to AI agents." }, "transport": { "type": "streamable-http", "endpoint": "https://yourdomain.com/mcp" }, "capabilities": { "tools": true, "resources": true, "prompts": false }, "authentication": { "type": "oauth2.1", "authorizationUrl": "https://yourdomain.com/oauth/authorize", "tokenUrl": "https://yourdomain.com/oauth/token" } }

/.well-known/agent-card.json (A2A protocol):

{ "name": "Brand Agent", "description": "Answers queries about brand products and content.", "endpoint": "https://yourdomain.com/agent", "capabilities": ["content-retrieval", "product-lookup"], "authentication": { "type": "oauth2.1" } }

/.well-known/api-catalog (RFC 9727):

{ "apis": [ { "title": "Brand Content API", "description": "Retrieves structured content for AI agent consumption.", "url": "https://yourdomain.com/api", "spec": "https://yourdomain.com/openapi.json" } ] }

All discovery endpoints must return security headers including Content-Type: application/json, X-Content-Type-Options: nosniff, Cache-Control: public max-age=3600, and Access-Control-Allow-Origin. AI clients including Claude Desktop, ChatGPT, and VS Code automatically probe /.well-known/mcp/server-card.json to auto-configure MCP server connections, so incorrect headers silently break discovery.

Roles: Platform engineer.

Validation: Fetch each endpoint with curl and confirm correct status codes, Content-Type headers, and JSON structure. Probe with a compatible AI client and confirm auto-configuration succeeds.

Step 4: Expose a Production MCP Endpoint

Goal: Provide a callable, authenticated MCP server that agents can invoke to retrieve content and execute tools.

Actions: Deploy an MCP server using Streamable HTTP transport. Remote MCP servers use OAuth 2.1 with PKCE and Dynamic Client Registration; agents initiate the auth flow, users approve access in-browser, and tokens are issued and refreshed automatically without API keys in config files.

MCP tool schema JSON example:

{ "jsonrpc": "2.0", "method": "tools/list", "result": { "tools": [ { "name": "get_article", "description": "Retrieves a brand article by topic slug and returns structured Markdown.", "inputSchema": { "type": "object", "properties": { "slug": { "type": "string", "description": "The URL slug of the article to retrieve." } }, "required": ["slug"] } }, { "name": "search_content", "description": "Performs a natural-language search across brand content and returns ranked results.", "inputSchema": { "type": "object", "properties": { "query": { "type": "string", "description": "The natural-language query to search." } }, "required": ["query"] } } ] } }

Production MCP implementations require strict tool permissions, scoped credentials, rate limiting, JSON Schema validation, audit logging, and least-privilege access controls. The MCP Transports specification requires servers to validate the Origin header on all incoming connections to prevent DNS rebinding attacks.

Roles: Backend engineer, security engineer.

Validation: Invoke each tool with a test agent client. Confirm JSON Schema validation rejects malformed inputs. Confirm OAuth 2.1 token refresh works without manual intervention. Confirm audit logs capture every tool call.

See how Blog MCP ships with schema, manifest, and discovery already configured.

Step 5: Configure Natural-Language Query Parameters and Markdown Delivery

Goal: Ensure agents that pass queries directly into URLs receive tailored, internally linked responses, and that agent crawlers receive Markdown instead of HTML.

Actions: Configure /?s={query} to auto-trigger a personalized, internally linked response page. Serve Markdown to agent crawlers by detecting User-Agent headers for known AI bots and returning Content-Type: text/markdown with the page content serialized as clean Markdown. Add the Link response header per RFC 8288 so agents can discover the API catalog without HTML parsing:

Link: </.well-known/api-catalog>; rel="api-catalog"

Natural-language query parameter example:

GET https://yourdomain.com/?s=best+adjustable+bed+for+back+pain

The server returns a structured page covering the query with internal links to related articles, product pages, and schema markup. The response works for both human readers and agent crawlers.

Roles: Platform engineer, technical content lead.

Validation: Send test queries via curl with a GPTBot User-Agent header. Confirm Markdown is returned. Confirm internal links resolve correctly. Confirm schema markup is present in the HTML version.

Step 6: Integrate the Four Observability Pillars

Goal: Instrument Search Intelligence, AI Analytics, Bot Tracking, and AI Ranking before go-live so every agent interaction is captured from day one.

Actions:

Search Intelligence: Connect real-time Google and ChatGPT data feeds to map the full query universe, including head terms and long-tail queries, refreshed weekly. This map defines which queries exist and which ones your content should target.
AI Analytics: With the query universe mapped, instrument brand value and consumer behavior tracking across external touchpoints including Google and AI-tool queries, content consumption, demographics, and sentiment. This layer connects query visibility to actual user behavior.
Bot Tracking: Deploy per-article bot tracking that logs every crawl, citation, and training sweep from both traditional crawlers and AI training agents. Confirm GPTBot, ClaudeBot, and PerplexityBot interactions are captured with timestamps and referencing URLs so you can tie behavior back to specific content.
AI Ranking: Configure order-of-mention and citation context tracking across ChatGPT, Perplexity, and Google AI Mode. AI answers have no static ordered list, so position within the answer and citation context replace the traditional rank number and complete the observability picture.

Roles: Analytics engineer, technical marketing lead.

Validation: Confirm bot tracking logs populate within 24 hours of go-live. Confirm Search Intelligence returns a universe snapshot covering at least the seed terms defined in the content topology. Confirm AI Ranking reports show citation context for at least one tracked query.

Review how the four pillars feed into incremental visibility reporting.

Step 7: Validate End-to-End with Live Agent Probes and Reporting

Goal: Confirm that the complete agentic stack is reachable, callable, and generating measurable incremental visibility.

Actions: Run live agent probes from at least two compatible AI clients such as Claude Desktop and a ChatGPT plugin or VS Code with GitHub Copilot. Confirm each client auto-discovers the MCP server via /.well-known/mcp/server-card.json, authenticates via OAuth 2.1, and successfully invokes at least one tool. Pull the first incremental visibility report from Bot Tracking and AI Ranking to establish a baseline. Cross-reference bot traffic against Google Search Console to confirm indexing is progressing.

Roles: Platform engineer, analytics engineer, technical marketing lead.

Validation checklist:

MCP server auto-discovered by two or more AI clients
OAuth 2.1 token flow completes without manual intervention
At least one tool call returns a valid JSON-RPC 2.0 response
llms.txt and llms-full.txt return HTTP 200 with correct Content-Type
/.well-known/ endpoints return correct security headers
Bot Tracking logs show at least one AI agent crawl
AI Ranking report shows at least one citation context entry
Google Search Console confirms new URLs are being indexed

Common Mistakes and Troubleshooting

Even with a full checklist, teams run into recurring issues during deployment. These mistakes appear in most first-time implementations, and catching them early prevents failed validation in Step 7.

Missing OAuth 2.1. As of April 2026, multiple AI agents support custom remote MCP servers with native OAuth 2.1, and agents that encounter non-compliant authentication refuse to connect. This gap is the most common deployment blocker in early 2026 implementations.

Incorrect Content-Type headers. Discovery endpoints must return Content-Type: application/json and X-Content-Type-Options: nosniff. Serving discovery files as text/html causes agent auto-configuration to fail silently.

Stale llms.txt. Stale llms.txt files generate broken code and incorrect agent behavior. Teams often treat llms.txt as a static asset instead of production code, which creates drift between the file and live content. The fix from Step 2, version control and change alerts, prevents this drift.

No Bot Tracking integration. Without per-article bot tracking, you cannot confirm whether AI agents reach the site, which tools they call, or whether citations appear. Bot Tracking is not optional observability, because it provides the primary signal that the agentic stack is working.

Security vulnerabilities in MCP tool descriptions. A comprehensive security analysis found that a significant percentage of public MCP servers have at least one vulnerability and some already have poisoned descriptions in the wild. Audit every tool description for prompt injection vectors before go-live.

Verifying Outcomes and Measuring Results

Verification spans all four observability pillars so you can tie technical work to business impact. Search Intelligence confirms the site appears in the traditional search landscape for the seed terms and long-tail queries defined in the content topology. AI Analytics tracks brand value and consumer behavior across external touchpoints, confirming that agent-driven traffic converts.

Bot Tracking logs every crawl, citation, and training sweep, providing ground truth for whether AI agents read and cite the site. AI Ranking tracks order of mention and citation context across ChatGPT, Perplexity, and Google AI Mode week over week.

Incremental visibility reporting then isolates exactly what the agentic stack generated, separate from visibility the brand already had. This separation proves ROI in a zero-click world, where citation frequency and citation context replace page views as the primary signal.

Advanced Scenarios and Next Steps

Multi-agent orchestration. Anthropic’s multi-agent research system using Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents outperformed a single-agent Claude Opus 4 baseline on internal research evaluations. For sites that need to serve multiple agent types simultaneously, a supervisor-subagent architecture with stateless subagents provides strong context isolation and parallel execution.

Stateless MCP via SEP-1442. SEP-1442 on the MCP roadmap will move the protocol from stateful sessions toward stateless requests, enabling horizontal HTTP scaling and easier deployment behind load balancers and CDNs. Plan infrastructure now so it can support stateless request handling as this specification matures.

A2A protocol integration. MCP standardizes agent-to-tool connectivity, while Google’s A2A protocol handles separate agent-to-agent communication in multi-agent architectures. A2A v1.0 was formally released in March 2026, marking the transition from draft to production standard. Implementing the Agent Card at /.well-known/agent-card.json positions the site for A2A-native agent orchestration as adoption grows.

Frequently Asked Questions

What is the difference between llms.txt and llms-full.txt, and do I need both?

llms.txt is a compact Markdown index at the site root that summarizes what the site covers and links its most important content pages with one-sentence descriptions. It suits token-limited conversational AI tools that need a curated map without fetching every page. llms-full.txt embeds complete page content, authentication details, and integration examples in a single self-contained file, suited for agents and IDE integrations that prefer to ingest everything in one pass without extra HTTP requests.

Most production sites benefit from both files. llms.txt supports chat-based AI surfaces, and llms-full.txt supports ingestion pipelines and developer tooling. The two files serve different consumption patterns and are not redundant.

Which authentication standard does a production MCP endpoint require in 2026?

Remote MCP servers require OAuth 2.1 with PKCE (Proof Key for Code Exchange) and Dynamic Client Registration. The agent initiates the authorization flow, the user approves access in-browser, and tokens are issued and refreshed automatically without API keys stored in configuration files.

This standard is enforced by multiple AI agents that support custom remote MCP servers with native OAuth 2.1 as of April 2026, including Claude Desktop, Claude Code, ChatGPT, VS Code with GitHub Copilot, and Cursor. Implementations that use custom token-passing or non-standard authentication are rejected by compliant agents. The 2026 MCP roadmap from Anthropic explicitly prioritizes OAuth 2.1 as the enterprise authentication specification.

How do /.well-known/ discovery files relate to the MCP Server Card and Agent Card?

The /.well-known/ directory is the standard location for machine-readable metadata files that agents probe automatically, following RFC 8615 conventions. The MCP Server Card lives at /.well-known/mcp/server-card.json and describes the MCP server’s transport type, capabilities, and authentication requirements so AI clients can auto-configure connections without prior integration.

The Agent Card lives at /.well-known/agent-card.json and describes the agent’s capabilities, inputs, outputs, and required authentication for A2A protocol communication between agents. The API Catalog lives at /.well-known/api-catalog per RFC 9727 and provides a machine-readable index of the site’s APIs and specifications.

All three files work together. The MCP Server Card handles tool connectivity, the Agent Card handles agent-to-agent communication, and the API Catalog handles API discovery. Omitting any one of them leaves a gap that compliant agents notice.

What observability is required to confirm an agent enabled site is working correctly?

Four categories of observability are required. Bot Tracking logs every crawl, citation, and training sweep from AI agents, confirming that agents reach the site and which content they read. AI Ranking tracks order of mention and citation context across AI surfaces week over week, replacing the traditional rank number with position within the answer and the claim the brand is cited for.

Search Intelligence maps the full query universe across traditional and AI search, confirming the site appears for the seed terms and long-tail queries it targets. AI Analytics tracks brand value and consumer behavior across external touchpoints, connecting agent-driven traffic to downstream conversion signals. Incremental visibility reporting then isolates what the agentic stack generated separately from pre-existing brand visibility, providing the evidence needed to prove ROI and guide content decisions.

Conclusion: Shipping a Complete Agentic Stack

Agent enabled sites implementation functions as a complete stack rather than a single file or endpoint. The stack includes llms.txt and llms-full.txt for agent-readable content maps, /.well-known/ discovery files for auto-configuration, a production MCP endpoint with OAuth 2.1 for callable tools, natural-language query parameters and Markdown delivery for agent crawlers, and four pillars of observability to confirm the stack works and generates measurable citations.

Shipping these components in isolation creates gaps that compliant agents encounter. Shipping them together, with traditional technical SEO underneath, makes a site genuinely agent-ready in 2026.

AI Growth Agent provisions every component of this stack automatically within the first week, with no engineering hours required on the client side beyond the reverse proxy rewrite. Bot Tracking, Blog MCP, llms.txt, llms-full.txt, /.well-known/ discovery, natural-language query parameters, Markdown for agents, and the full traditional technical SEO layer ship together in every package.

Traditional search tools show you where your brand stands. AI Growth Agent makes your brand the answer. Book a kickoff and see your first article live within a week.

Go live with the complete agentic technical SEO stack within the first week.