| Industry Cohort | Sample | Custom /llms.txt | Auto Boilerplate | Custom /agents.md | Base JSON-LD | FAQPage Schema | WAF HTTP 403 |
|---|---|---|---|---|---|---|---|
| B2B SaaS & Developer Tools | 10 | 10 / 10 ( 100 %) | 0 / 10 ( 0 %) | 2 / 10 ( 20 %) | 9 / 10 ( 90 %) | 0 / 10 ( 0 %) | 0 / 10 ( 0 %) |
| Shopify & D2C Commerce | 10 | 1 / 10 ( 10 %) | 6 / 10 ( 60 %) | 0 / 10 ( 0 %) | 7 / 10 ( 70 %) | 0 / 10 ( 0 %) | 1 / 10 ( 10 %) |
| SEO, Content & Digital Agencies | 10 | 4 / 10 ( 40 %) | 1 / 10 ( 10 %) | 0 / 10 ( 0 %) | 8 / 10 ( 80 %) | 2 / 10 ( 20 %) | 0 / 10 ( 0 %) |
| Multi-Location Healthcare, Legal & Home Services | 10 | 5 / 10 ( 50 %) | 0 / 10 ( 0 %) | 0 / 10 ( 0 %) | 9 / 10 ( 90 %) | 1 / 10 ( 10 %) | 1 / 10 ( 10 %) |
| All Cohorts Combined | 40 | 20/40 (50.0%) | 7/40 (17.5%) | 2/40 (5.0%) | 33/40 (82.5%) | 3/40 (7.5%) | 2/40 (5.0%) |
Four protocol gaps separating cited brands from invisible domains
The 80-Point Shopify Agent Truth Gap
Eight of ten probed D2C commerce brands (80.0%) expose active Shopify /api/ucp/mcp commerce endpoints, yet 0 of 10 (0.0%) have customized /agents.md and only 1 of 10 (10.0%) serves a custom /llms.txt. Six of ten serve unedited stock Shopify Shop-skill boilerplate lacking free-shipping thresholds, return windows, and bundle exclusions.
Open /robots.txt Files Mask Silent CDN 403 Blocks
Zero of the 40 probed domains (0.0%) disallow GPTBot or PerplexityBot at root inside /robots.txt, yet 2 of 40 (5.0% — ridge.com and westerndental.com) return HTTP 403 Forbidden when probed with real GPTBot or PerplexityBot User-Agent headers at the CDN edge.
One-Third of HTTP 200 /llms.txt Responses Are Uncurated or HTML Soft-404s
Thirty domains returned HTTP 200 on GET /llms.txt, but Content-Type and body inspection revealed 3 HTML SPA soft-404 pages (7.5% of the cohort) and 7 auto-generated Shopify or Rank Math boilerplate dumps (17.5%). Developer SaaS leads genuine curation at 100% (10/10), compared with 40% of SEO agencies and 10% of D2C brands.
Base Organization Schema Is Common; Answer-First FAQPage Schema Is Rare
While 33 of 40 domains (82.5%) render at least one application/ld+json block on their homepage, only 3 of 40 (7.5%) include FAQPage structured Q&A pairs—leaving ChatGPT Search, Gemini, and Perplexity to infer pricing tiers, service areas, and SLAs from third-party directories.
| Domain | Vertical | /llms.txt | /agents.md | Homepage JSON-LD | GPTBot / PerplexityBot | Shopify UCP | Live Check |
|---|---|---|---|---|---|---|---|
| stripe.com | B2B SaaS | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| vercel.com | B2B SaaS | CUSTOM CURATED | CUSTOM ENRICHED | 3 node s | HTTP 200 OPEN | — | Re-Probe → |
| supabase.com | B2B SaaS | CUSTOM CURATED | MISSING | 4 node s | HTTP 200 OPEN | — | Re-Probe → |
| resend.com | B2B SaaS | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| linear.app | B2B SaaS | CUSTOM CURATED | CUSTOM ENRICHED | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| cloudflare.com | B2B SaaS | CUSTOM CURATED | MISSING | 3 node s | HTTP 200 OPEN | — | Re-Probe → |
| posthog.com | B2B SaaS | CUSTOM CURATED | MISSING | 2 node s | HTTP 200 OPEN | — | Re-Probe → |
| clerk.com | B2B SaaS | CUSTOM CURATED | MISSING | 2 node s | HTTP 200 OPEN | — | Re-Probe → |
| dub.co | B2B SaaS | CUSTOM CURATED | MISSING | 2 node s | HTTP 200 OPEN | — | Re-Probe → |
| cal.com | B2B SaaS | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| allbirds.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 0 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| gymshark.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 0 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| brooklinen.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 4 node s | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| chubbies.com | Shopify D2C | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| ridge.com | Shopify D2C | AUTO BOILERPLATE | MISSING | 0 nodes | BOTH WAF 403 | UNAVAILABLE | Re-Probe → |
| cutsclothing.com | Shopify D2C | MISSING | MISSING | 2 node s | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| ruggable.com | Shopify D2C | HTML SOFT-404 | MISSING | 1 node | HTTP 200 OPEN | UNAVAILABLE | Re-Probe → |
| trueclassictees.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 3 node s | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| carawayhome.com | Shopify D2C | MISSING | MISSING | 2 node s | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| beardbrand.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 2 node s | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| siegemedia.com | SEO Agency | MISSING | MISSING | 2 node s | HTTP 200 OPEN | — | Re-Probe → |
| animalz.co | SEO Agency | MISSING | MISSING | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| omniscientdigital.com | SEO Agency | MISSING | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| singlegrain.com | SEO Agency | CUSTOM CURATED | MISSING | 3 node s · FAQPage | HTTP 200 OPEN | — | Re-Probe → |
| directiveconsulting.com | SEO Agency | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| victorious.com | SEO Agency | MISSING | MISSING | 2 node s · FAQPage | HTTP 200 OPEN | — | Re-Probe → |
| webmechanix.com | SEO Agency | AUTO BOILERPLATE | MISSING | 5 node s | HTTP 200 OPEN | — | Re-Probe → |
| klientboost.com | SEO Agency | HTML SOFT-404 | MISSING | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| powerdigitalmarketing.com | SEO Agency | CUSTOM CURATED | MISSING | 2 node s | HTTP 200 OPEN | — | Re-Probe → |
| webfx.com | SEO Agency | CUSTOM CURATED | MISSING | 3 node s | HTTP 200 OPEN | — | Re-Probe → |
How every domain was probed and classified
Each of the 40 public domains was probed directly over HTTPS using the same deterministic parser powering Verisource's production scanner: (1) GET /robots.txt parsed for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended directives; (2) homepage requests sent with explicit GPTBot/1.2 and PerplexityBot/1.0 User-Agent headers to catch silent CDN HTTP 403 / 429 blocks; (3) GET /llms.txt and GET /agents.md inspected for text/html SPA soft-404s and stock Shopify (shop.app/SKILL.md) or SEO plugin boilerplate; (4) homepage application/ld+json script nodes parsed for FAQPage entities; and (5) JSON-RPC tools/list handshakes tested against /api/ucp/mcp and /api/mcp on D2C commerce stores.
90% of DevTools ship /llms.txt. Only 20% ship /agents.md.
In a follow-up live HTTP probe across 10 developer infrastructure and B2B SaaS domains (trigger.dev, inngest.com, turso.tech, tinybird.co, knock.app, workos.com, stytch.com, upstash.com, highlight.io, and infisical.com), 9 of 10 (90.0%) served a valid /llms.txt directory, yet only 2 of 10 (20.0% — tinybird.co and infisical.com) published a custom /agents.md pricing and limits guardrail file, and only 1 of 10 (10.0% — upstash.com) shipped all 3 JSON-LD pillars.
| Protocol / Schema Signal | DevTools Cohort (N = 10) | Commercial Impact |
|---|---|---|
| /llms.txt (HTTP 200) | 90.0% (9 / 10) | High documentation-link adoption across modern DevTools |
| Custom /agents.md | 20.0% (2 / 10) | 80% leave pricing tiers & rate limits unguarded against AI hallucination |
| Full 3-Pillar JSON-LD | 10.0% (1 / 10) | 90% omit FAQPage or Offer pricing nodes on root pages |
Common questions about the 2026 AI Search Readiness dataset
In our live HTTP readiness probe across 40 public domains spanning B2B SaaS, Shopify D2C, SEO agencies, and multi-location services, 50.0% (20/40) serve a custom-curated /llms.txt file, another 17.5% (7/40) expose auto-generated Shopify or SEO-plugin boilerplate, 7.5% (3/40) return HTML soft-404s with HTTP 200 status, and only 5.0% (2/40 — vercel.com and linear.app) publish a custom /agents.md specification.
Shopify rolled out platform-level /api/ucp/mcp endpoints and stock /agents.md files pointing to https://shop.app/SKILL.md across modern storefronts, which explains why 8 of 10 probed D2C stores respond on /api/ucp/mcp. However, stock Shopify files omit brand-specific free-shipping thresholds, return windows, and bundle exclusions unless the merchant overrides them via templates/agents.md.liquid or /pages/ai-facts.
Yes. Zero of the 40 probed domains disallowed GPTBot or PerplexityBot at root in /robots.txt, yet 5.0% (2/40 — ridge.com and westerndental.com) returned HTTP 403 Forbidden at the CDN/WAF layer when requested with GPTBot or PerplexityBot User-Agent headers. Testing both /robots.txt and simulated crawler User-Agent HTTP status codes is required.