Quick answer: Preparing a Drupal site for AI agents and agentic search means making your content easy for AI systems to crawl, understand, trust, and cite. This involves three main layers: permitting AI crawlers in your robots.txt, serving a curated llms.txt file summarising your key content, and adding accurate schema.org structured data (JSON-LD) so AI systems can extract facts without misreading them. Drupal now has purpose-built modules for all three, making this more achievable than on many other platforms.
Search behaviour has genuinely changed. A growing share of people now get answers from ChatGPT, Perplexity, Google AI Overviews, and Gemini rather than clicking through a traditional list of results — and increasingly, AI agents themselves are fetching and acting on web content directly, not just summarising it for a human to read. This guide covers what that means for a Drupal site specifically, and the practical steps to prepare for it.
Why This Matters Beyond Just SEO
Traditional SEO optimises for ranking in a list a human scans. Agentic readiness is different — it’s about being the source an AI system trusts enough to cite, or the page an autonomous agent can successfully read and act on without a human present at all.
This distinction matters because a new category of AI agent is emerging that goes well beyond search: an agent booking an appointment, checking product availability, or pulling documentation to complete a task on someone’s behalf. Coding agents like Cursor, GitHub Copilot, and Claude Code already routinely fetch a site’s llms.txt file to understand a project’s documentation before doing anything else — this same pattern is extending into general-purpose AI assistants acting on behalf of everyday users.
The Three Layers of Agent Readiness
1. Crawler Permission — Can AI Agents Even Access Your Site?
Before an AI system can read anything, it needs permission. Your robots.txt file needs explicit rules allowing the AI crawlers relevant to your goals — this includes bots like OAI-SearchBot (ChatGPT Search), ClaudeBot, PerplexityBot, and Google-Extended. It’s worth deciding deliberately: some businesses want their content included in AI answers but excluded from AI model training data, which requires distinguishing between search-purpose crawlers and training-purpose crawlers rather than blocking or allowing everything indiscriminately.
2. llms.txt — A Curated Map for AI Systems
An llms.txt file sits at your site’s root (yourdomain.com/llms.txt) and gives AI systems a clean, Markdown-formatted summary of your site’s most important content — stripped of navigation menus, footers, and other noise that isn’t useful for an AI trying to understand what your organisation actually does.
For Drupal specifically, this is no longer something you need to build manually. Several purpose-built modules now exist:
- The llmstxt module lets administrators manage the content of their llms.txt file directly through a simple admin settings form, served dynamically at the /llms.txt route
- An AI-powered LLMs.txt Generator module parses your sitemap.xml and automatically produces structured output that follows the llms.txt specification, integrating with Drupal’s AI ecosystem to generate and let you review the content
- A content-type-based generator module automatically creates one llms.txt section per content type, listing published nodes as organised links, and regenerates automatically via cron to stay current
3. Structured Data — Giving AI Systems Verifiable Facts
Schema.org structured data (JSON-LD) explicitly tells AI systems what a page contains — a product, a service, a review, an FAQ — rather than forcing them to infer it from unstructured text. This matters because AI systems increasingly favour content they can extract with confidence over content that requires interpretation.
Drupal’s ecosystem has moved specifically toward this: purpose-built JSON-LD modules now generate structured data with strict honesty rules — emitting a fact only when it’s actually rendered on the visible page, and emitting nothing at all rather than risk inflated or inaccurate structured data. This “emit nothing rather than guess” approach matters, since AI systems that catch a site publishing inaccurate structured data are less likely to trust that source going forward.
What Actually Makes Content “Agent-Readable”
Beyond the technical infrastructure, the content itself needs to be structured in a way AI systems can extract cleanly:
- Use genuine HTML tables for comparisons, pricing, and specifications — AI agents parse structured tables far more reliably and accurately than the equivalent information written out in paragraph form
- Lead with direct answers before expanding into detail — a clear, concise answer near the top of a section gets extracted more reliably than one buried in the fourth paragraph
- Use FAQ structures with genuine Q&A pairs — but only where there’s real substance; a thin or padded FAQ section is often better left out entirely, since low-quality structured data can hurt trust more than help it
- Avoid heavy client-side rendering for key content. AI crawlers often struggle with JavaScript-heavy pages that require full hydration before content becomes visible — server-rendered content (which Drupal handles natively) has a real advantage here over JavaScript-framework-only sites
What This Doesn’t Mean
Agent-readiness isn’t a replacement for genuinely good content, and it isn’t a shortcut around Drupal’s existing SEO fundamentals — crawlability, clean page hierarchies, fast load times, and accurate content still form the foundation everything else sits on. AI systems consistently reward the same underlying qualities traditional search always has: clarity, accuracy, and genuine expertise — the delivery format has changed more than the substance that earns trust.
A Practical Starting Checklist for Drupal Sites
- Audit your current robots.txt and deliberately decide which AI crawlers to allow or block
- Install and configure an llms.txt module suited to your site’s structure (manually curated vs. auto-generated by content type)
- Review your structured data setup — implement JSON-LD schema for your key content types (services, products, FAQs) using Drupal’s current AI/SEO module ecosystem
- Reformat key comparison or pricing content into genuine HTML tables rather than prose
- Check whether critical content relies on heavy client-side JavaScript rendering that could block AI crawlers from reading it
- Re-test periodically — this space is moving quickly, and today’s best practice may shift again within months
Frequently Asked Questions
What is llms.txt and does my Drupal site need one?
llms.txt is a Markdown file at your site’s root that gives AI systems a curated summary of your most important content. It’s increasingly useful for AI agents and coding assistants that fetch it directly, and Drupal now has several modules that generate and manage it automatically.
Will preparing my site for AI agents hurt my traditional SEO?
No. The foundational work — crawlability, clean structure, accurate content, fast performance — benefits both traditional search rankings and AI agent readiness. They’re complementary, not competing, priorities.
Do I need custom development to add llms.txt or structured data to Drupal?
Not necessarily. Purpose-built contributed modules now handle both llms.txt generation and JSON-LD structured data for Drupal, though a developer can help configure them correctly for your specific content model and verify the output is accurate.
Should I block AI crawlers from my site entirely?
Generally not, if visibility in AI-generated answers matters to your business. Most organisations instead choose to allow search-purpose AI crawlers while deciding separately whether to permit AI training-specific crawlers, since these serve different purposes.
How do I know if my Drupal site is actually agent-ready?
Start by checking whether your robots.txt explicitly allows major AI crawlers, confirming an llms.txt file exists and is current, and verifying your structured data validates correctly without errors. Ongoing monitoring matters too, since AI crawler behaviour and standards are still evolving.
Final Thoughts
Preparing a Drupal site for AI agents and agentic search isn’t a separate discipline bolted onto existing SEO — it’s an extension of the same fundamentals Drupal has always been strong at: clean architecture, server-rendered content, and a mature module ecosystem that’s already building dedicated tools for this exact challenge. Businesses that get the groundwork right now — crawler permissions, llms.txt, and accurate structured data — will be far better positioned as AI agents take on a larger share of how people discover and interact with content online.