Perplexity, ChatGPT Browse, and Claude are crawling your website right now — and deciding whether your business is worth recommending. Here is how to make your site legible, trustworthy, and AI-ready.
A New Kind of Visitor Is Reading Your Website
Most website analytics only count humans. But today, a growing share of your most important visitors are not human at all — they are AI agents. Perplexity AI, ChatGPT with Browse, Google Gemini, Claude, and dozens of specialised research tools are constantly crawling the web, reading your pages, and deciding whether your business is worth citing when someone asks a relevant question. Optimising for these machine visitors is the next frontier of technical SEO, and it is already happening whether you are ready or not.
How AI Agents Read Your Site Differently
A human visitor scans a page in seconds, guided by visual hierarchy and layout. An AI agent reads the raw structure — headings, semantic HTML, linked data, and page metadata — and constructs a mental model of what your business does and how authoritative it is. The differences matter:
- Design is invisible. Your beautiful hero image and animated counters mean nothing to an AI scraper. What matters is the text, the structure, and the data behind it.
- Hierarchy is critical. Clear H1 → H2 → H3 heading structure lets an AI agent understand your content's architecture at a glance. Pages with flat, chaotic heading structures are summarised poorly or skipped.
- Schema markup is a direct signal. JSON-LD structured data for your
Organization,Service,FAQPage, andArticletypes tells AI agents exactly what your content is, who published it, and what it is about — in machine-readable language they process natively.
The llms.txt Standard — A robots.txt for AI
A new emerging standard is the llms.txt file — a plain-text document placed at the root of your website (e.g. yourdomain.com/llms.txt) that tells AI language models which parts of your site are most important, which pages to prioritise for summarisation, and which to ignore. Think of it as a robots.txt for the AI age. Early adopters who add this file are signalling to AI crawlers that their site is AI-ready — a distinction that is likely to carry increasing weight as these crawlers become more selective.
What to Put in Your llms.txt
Core Technical Checklist for AI-Readiness
- ✅ Semantic HTML: Use proper
<article>,<section>,<main>,<nav>tags — not just<div>everywhere - ✅ JSON-LD structured data on every page (Organization, LocalBusiness, Article, FAQPage)
- ✅ Clear, descriptive page titles and meta descriptions — these are often what AI agents use as their one-line summary of your page
- ✅ Fast, crawlable pages — AI agents time out on slow sites just like Googlebot does
- ✅ An updated sitemap.xml that includes all your content pages
- ✅ An llms.txt file at your domain root guiding AI agents to your best content
- ✅ robots.txt that allows AI crawlers — many businesses accidentally block GPTBot, ClaudeBot, PerplexityBot via blanket
Disallow: /rules
The Compounding Advantage
The businesses that appear in AI-generated answers consistently are not the biggest — they are the most legible. Legible to machines means structured, consistent, authoritative, and current. MARCOMS builds websites and content architectures that work for both human visitors and AI agents. Get in touch to find out how AI-ready your current site is.