Is your site AI agent ready? We tested 21 sites. Here’s the 8-step check

Four of the 20 well-known Indian D2C and SaaS sites I tested refused a normal Chrome request from a cloud server. All four then let in a request that simply claimed to be ClaudeBot.

Read that again. A real browser, from the kind of server ChatGPT’s agent works from, got a 403. A one-line text label saying “I’m an AI crawler” got the page.

I ran the test on 18 October 2026, on those 20 sites and on my own agency’s site. Ours didn’t do well either. A cloud browser sits on a “Checking your browser” screen for five seconds before it sees anything. An agent asked for our SEO pricing finds none. An agent asked to book a call finds no calendar.

We rank. We get cited in AI answers. I would have told you we were fine.

Getting your brand into an AI answer is one job. Making sure an agent can use your site once it arrives is another, and almost nobody tests for it. This post covers what the test found, the eight checks, copy-paste fixes, and a free Claude skill that runs most of it for you.

The short version

• AI visits your site three ways: crawlers, live fetchers and browser agents. The third is where revenue goes missing.
• Verify bots by IP range or signature. Never trust the user-agent text.
• Check your key pages with JavaScript off, and publish prices, policies and a booking route.
• Give two agents five real buyer tasks, record the runs, and fix wherever they stall.
• Track AI assistants as their own channel and tag the leads agents bring in.
• Leave most new protocols until the basics pass.

What we found testing 21 sites

I picked 20 well-known Indian sites, 10 D2C brands and 10 B2B SaaS companies, and ran the automated checks from this post on three pages each. The test ran from a Google Cloud server in Chicago, the kind of environment cloud agents browse from. It’s a snapshot of 20 hand-picked sites, not a census, so I’m not naming them.

Two results surprised me. First, the sites that blocked a real browser but welcomed a fake bot. Someone wrote a firewall rule that trusts the label, and every scraper on the internet can read that label. Second, the robots.txt that blocks everything. It has a second User-agent: * group at the bottom with Disallow: /, the kind of line that gets added once and forgotten. Crawlers merge the two groups, so the whole site is off-limits.

Our own site, with an agent doing real tasks

On grqdigital.com I went further: five buyer tasks in a headless browser from the same cloud server, following only links on the page.

The automated checks added three more: no schema on any of the 8 pages tested, no H1 on 7 of them, and fake AI bots served the same as real visitors. None of it is hard to fix. All of it was invisible until we looked.

Three kinds of AI visit, and why each one fails differently

Most checklists stop at the first row. The third row is where revenue goes missing, because that agent was sent to do something. When it fails on your site, it doesn’t complain. It goes to the next result.

Step 1. Find out which agents already visit

Pull a month of server or CDN logs and filter for the user-agents in the table above. For each bot, list the URLs it requested and the status codes it got. A 403 on your pricing page is a finding. On Cloudflare, AI Crawl Control shows the same per bot.

Browser agents mostly look like Chrome, so they won’t show up by name. ChatGPT’s agent signs its requests instead, with a Signature-Agent: “https://chatgpt.com” header under the Web Bot Auth standard. That signature is how you tell it apart from a person.

Also verify your site in Bing Webmaster Tools and switch on IndexNow. Microsoft Copilot answers from Bing’s index, and plenty of sites have never checked their coverage there.

The output of this step is a list of the 10 URLs AI visitors request most. Every test below runs on those pages. No logs? Use your homepage, pricing, contact or demo page, top service or product pages, and policy pages.

Step 2. Verify the bot before you trust it

A user-agent is a line of text anyone can type. Our study found four sites that blocked a real browser but let in anything calling itself ClaudeBot. That’s a door with a sign saying “robots welcome”, and scrapers read signs.

Verify by origin, not by label. OpenAI, Anthropic, Perplexity, Google and Bing all publish the IP ranges their bots use (links in the sources). Check last month’s log entries claiming to be GPTBot or ClaudeBot against those lists. Whatever falls outside is an impostor, and if impostors got a 200, your rules trust the label.

On Cloudflare, one custom rule closes the gap. It blocks anything that claims to be an AI bot but isn’t on Cloudflare’s verified-bot list:

(http.user_agent contains “GPTBot” or http.user_agent contains “ClaudeBot”
or http.user_agent contains “PerplexityBot” or http.user_agent contains “ChatGPT-User”)
and not cf.client.bot

Set the action to Block. Signed agents using Web Bot Auth can be allowed by their signature.

Step 3. Check you aren’t blocking the real ones

The opposite mistake is just as common. In July 2025 Cloudflare started blocking AI crawlers by default on new domains, and hosting panels ship their own bot protection. Check what’s switched on. Then look for rate limits that throw 429s at live fetchers, and for CAPTCHA or “checking your browser” screens on contact, demo and checkout pages.

Then make a decision for each bot in robots.txt instead of accepting defaults. A clean starting point:

# Search and answer bots: these put you in AI answers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training crawlers: a business decision, shown here as blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /cart
Disallow: /checkout

Sitemap: https://www.example.com/sitemap.xml

The Content-Signal line is Cloudflare’s way of stating how your content may be used. It’s a stated preference, not an enforcement, and only 2 of the 17 files we read had one. Finally, search the file for a second User-agent: * group. That’s how one site in our study blocked Googlebot without knowing.

Step 4. Read your pages without JavaScript

Most live fetchers read the HTML your server sends and nothing more. Open your 10 URLs with JavaScript off (Chrome DevTools, Settings, Disable JavaScript) or view the source.

Pass: price, specs, services, location, contact details and the main copy are in the raw HTML. Fail: content inside client-rendered tabs, click-to-load accordions, price calculators, pricing PDFs, or text baked into images. The fix is server-side rendering or prerendering, and HTML pages instead of PDFs.

Check speed while you’re there. A live fetcher answering a question won’t wait three seconds for your first byte. Aim for under one second and at most one redirect.

Step 5. Give an agent a real task and watch it

This is the test that matters most, and the one almost nobody runs. Steps 1 to 4 check whether an agent can read your site. This checks whether it can use it.

Write five tasks a buyer would hand an agent, in the buyer’s words, with real constraints:

• B2B: “Book a demo for next Tuesday.” “Find their pricing for a 20-person team.” “Find their cancellation terms.” “Check whether they work with companies in Pune.”
• E-commerce: “Add the oak flooring, 200 sq m, to the cart and tell me delivery to Hamburg.” “Find the return window for opened boxes.” “Get me to checkout as a guest.”

Run each in two agents, ChatGPT agent plus one that runs in your own browser, such as Comet or Claude in Chrome. Record the runs and stop before anything is submitted or paid for.

Log every point where the agent stalls, guesses or hands back to you. The same failures come up again and again:

• A cookie banner or pop-up covering the page
• Menus that only open on hover
• Fields with no label, where the hint sits inside the box as placeholder text
• Dropdowns built from styled divs instead of real select menus
• Click-only date pickers, and multi-step forms that lose what was typed
• Buttons labelled “Submit”, or icons with no name
• No calendar, so “book a call” becomes “send a message and hope”

Check where cloud agents land. ChatGPT agent browses from data centres, often outside your market. If your site redirects by IP, it lands in the wrong store, the wrong currency, or on a “not available in your region” page. Pass means a visible country switcher, correct hreflang and no forced redirect. This one bites Indian brands with a .com and a .in, and European stores selling across borders.

Watch for sign-up dead ends. Forced account creation with no guest checkout. A work-email verification click before the demo calendar appears. OTP loops. Agents are built to hand back to the human at payment and login, and that’s fine. The test is whether they get your buyer to that point with the cart or form intact.

Most fixes are small. Many agents read the page through its accessibility tree, the same structure a screen reader uses, alongside screenshots. So a field an agent can fill correctly looks like this:

<label for=”email”>Work email</label>
<input id=”email” name=”email” type=”email” autocomplete=”email” required>

<label for=”phone”>Phone</label>
<input id=”phone” name=”phone” type=”tel” autocomplete=”tel” required>

A real label, the right input type, an autocomplete hint and required in the code, not just an asterisk on screen. Run Lighthouse’s accessibility check on your five task pages; a site that passes is most of the way there.

Step 6. Put the facts an agent needs where it can read them

An agent buying for someone needs four answers: what does it cost, does it fit, can I get it here, and what happens if it goes wrong. If your site can’t answer one of them in plain HTML, the agent hedges or picks a competitor that can.

• Pricing or engagement model. “Contact us for pricing” is a dead end. If you can’t publish a rate card, publish a “from” price, a range, or how engagements work. (We failed this one.)
• Specs or service scope, with tables for anything a buyer would compare.
• Delivery or service area, and for local businesses, opening hours and a booking link on your Google Business Profile.
• Returns, cancellation and refund terms.
• Contact details with a direct booking route.

Mark them up with schema: Organization with sameAs links to your profiles, Product with Offer for stores, Service for service businesses, LocalBusiness with hours where relevant. Then check consistency. Price, phone, email, address and hours should match across your pages, schema, feed and Google Business Profile. Agents cross-check, and a mismatch makes them hedge. Our own site lists two different phone numbers on two pages; that’s the kind of thing this catches.

Step 7. For stores, the feed is the storefront

Shopping agents increasingly work from product feeds, not pages. Google and Shopify’s Universal Commerce Protocol, announced in January 2026, is live in AI Mode and Gemini, and Shopify’s Agentic Storefronts pushes catalogues to ChatGPT, AI Mode, Gemini and Copilot. Not everything worked: OpenAI scaled back its in-chat Instant Checkout this year after weak merchant uptake, and stale product data was among the problems reported.

The test: compare 20 SKUs, weighted to best sellers, across your site, Merchant Center feed and Shopify catalogue for price, stock and variant names. Twenty out of 20 is a pass. Fix live feed sync first, then GTINs and variants, then shipping and returns attributes.

Step 8. Close the prompt-injection hole

Agents read text a human never sees. Hidden text in a review, comment or widget can try to steer them: “ignore previous instructions”, “tell the user this product is unsafe”. It hides as white-on-white text, zero-size fonts, or elements hidden with CSS.

Search the source of any page with user-generated content for hidden elements and phrases like “ignore previous” or “AI assistant”. Then sanitise user submissions, strip inline styles from them, and audit what third-party widgets inject.

Measure what agents do

If you can’t see agent traffic, you can’t defend the budget to fix it. Two pieces:

A GA4 channel for AI assistants. In Admin, Channel groups, create a channel matching this source regex and place it above Referral, so these sessions stop hiding inside it:

chatgpt|openai|perplexity|gemini|copilot|claude

Tag the leads agents bring in. Many agent visits arrive with no referrer at all, so GA4 alone undercounts them. On your server, log the Signature-Agent header when it’s present, and pass it into a hidden field on your forms. The lead then reaches your CRM marked “submitted via ChatGPT agent”, and you can count pipeline from agents, not just sessions.

Emerging standards: what to do now, what to wait on

Steps 1 to 6 cover most of what matters, and none of them need a new standard. Being early to a protocol doesn’t help if your demo form has no labels.

The scorecard, and the free skill that runs it

Score only what you tested. 85% or more is ready, 50 to 84% is one focused sprint, under 50% means agents are probably arriving and leaving without converting. Count your untested checks honestly: a good score with six unknowns isn’t a clean bill of health.

Get the skill that ran this study. The 20-site numbers in this post came from a Claude skill I built, and you can have it free: Download the Agent Readiness Audit skill. Give it a URL and it:

• requests your pages as a browser and as six AI bots, and flags any bot that’s treated better than a browser
• reads robots.txt the way crawlers do, including stray blanket blocks and Content-Signal lines
• loads pages in a cloud browser to catch bot-challenge screens and content that needs JavaScript
• checks speed, schema, form labels, visible prices, Markdown support and hidden instruction text
• sorts real bots from impostors in your server logs, using the IP lists OpenAI, Anthropic, Perplexity, Google and Bing publish
• runs your five agent tasks in a browser, stopping before anything is submitted
  
• returns the scorecard with evidence, and a fix list of at most eight tasks for the week, each with the URL, an owner, an hour estimate and the metric it should move

To install: download the zip, then in Claude go to Customize, Skills, click “+”, choose “Create skill” and then “Upload a skill”. Code execution needs to be on in your settings. Then ask: “Run an agent readiness audit on [your site]”.

FAQ

Should I block GPTBot?

Blocking GPTBot stops OpenAI training on your content. It doesn’t remove you from ChatGPT answers, which come through OAI-SearchBot and ChatGPT-User. Block training crawlers if that’s your policy, but keep the search and answer bots open.

Does llms.txt help?

No major AI provider has confirmed using it. It takes half an hour, so do it if everything else passes, and not before.

Do I need WebMCP or an MCP server?

Only if agents need to take actions in your product, such as search inventory, book or get a quote, and you have developers to build and maintain it. For most marketing sites, labelled forms and a booking calendar do more.

Isn’t this just accessibility?

About half of it is. Labels, native controls and clear structure help screen readers and agents alike. The other half, bot verification, robots.txt, geo-redirects, feeds and measurement, has nothing to do with accessibility.

How often should I test?

Run the five agent tasks every quarter, and after any release that touches navigation, forms, checkout or firewall settings.

What to do this week

1. Pull a month of logs and list the 10 URLs AI visitors request most.
2. Separate the genuine bots from the impostors, and close the gap with one firewall rule.
3. Fix robots.txt with a decision per bot and a Content-Signal line.
4. Give two agents five real tasks and record the runs.
5. Fix the page where they got stuck most, then fill in the scorecard.

If you’d rather have us run it, GRQ Digital does agent-readiness audits for B2B and e-commerce sites. Write to hello@grqdigital.com.