Newshot

40-domain study: 80% of Shopify stores expose /API/ucp/MCP, 0% /agents.md

Original text(via Hacker News)
Industry CohortSampleCustom /llms.txtAuto BoilerplateCustom /agents.mdBase JSON-LDFAQPage SchemaWAF HTTP 403
B2B SaaS & Developer Tools1010 / 10 ( 100 %)0 / 10 ( 0 %)2 / 10 ( 20 %)9 / 10 ( 90 %)0 / 10 ( 0 %)0 / 10 ( 0 %)
Shopify & D2C Commerce101 / 10 ( 10 %)6 / 10 ( 60 %)0 / 10 ( 0 %)7 / 10 ( 70 %)0 / 10 ( 0 %)1 / 10 ( 10 %)
SEO, Content & Digital Agencies104 / 10 ( 40 %)1 / 10 ( 10 %)0 / 10 ( 0 %)8 / 10 ( 80 %)2 / 10 ( 20 %)0 / 10 ( 0 %)
Multi-Location Healthcare, Legal & Home Services105 / 10 ( 50 %)0 / 10 ( 0 %)0 / 10 ( 0 %)9 / 10 ( 90 %)1 / 10 ( 10 %)1 / 10 ( 10 %)
All Cohorts Combined4020/40 (50.0%)7/40 (17.5%)2/40 (5.0%)33/40 (82.5%)3/40 (7.5%)2/40 (5.0%)

Four protocol gaps separating cited brands from invisible domains

The 80-Point Shopify Agent Truth Gap

Eight of ten probed D2C commerce brands (80.0%) expose active Shopify /api/ucp/mcp commerce endpoints, yet 0 of 10 (0.0%) have customized /agents.md and only 1 of 10 (10.0%) serves a custom /llms.txt. Six of ten serve unedited stock Shopify Shop-skill boilerplate lacking free-shipping thresholds, return windows, and bundle exclusions.

Open /robots.txt Files Mask Silent CDN 403 Blocks

Zero of the 40 probed domains (0.0%) disallow GPTBot or PerplexityBot at root inside /robots.txt, yet 2 of 40 (5.0% — ridge.com and westerndental.com) return HTTP 403 Forbidden when probed with real GPTBot or PerplexityBot User-Agent headers at the CDN edge.

One-Third of HTTP 200 /llms.txt Responses Are Uncurated or HTML Soft-404s

Thirty domains returned HTTP 200 on GET /llms.txt, but Content-Type and body inspection revealed 3 HTML SPA soft-404 pages (7.5% of the cohort) and 7 auto-generated Shopify or Rank Math boilerplate dumps (17.5%). Developer SaaS leads genuine curation at 100% (10/10), compared with 40% of SEO agencies and 10% of D2C brands.

Base Organization Schema Is Common; Answer-First FAQPage Schema Is Rare

While 33 of 40 domains (82.5%) render at least one application/ld+json block on their homepage, only 3 of 40 (7.5%) include FAQPage structured Q&A pairs—leaving ChatGPT Search, Gemini, and Perplexity to infer pricing tiers, service areas, and SLAs from third-party directories.

DomainVertical/llms.txt/agents.mdHomepage JSON-LDGPTBot / PerplexityBotShopify UCPLive Check
stripe.comB2B SaaSCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
vercel.comB2B SaaSCUSTOM CURATEDCUSTOM ENRICHED3 node sHTTP 200 OPEN—Re-Probe →
supabase.comB2B SaaSCUSTOM CURATEDMISSING4 node sHTTP 200 OPEN—Re-Probe →
resend.comB2B SaaSCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
linear.appB2B SaaSCUSTOM CURATEDCUSTOM ENRICHED0 nodesHTTP 200 OPEN—Re-Probe →
cloudflare.comB2B SaaSCUSTOM CURATEDMISSING3 node sHTTP 200 OPEN—Re-Probe →
posthog.comB2B SaaSCUSTOM CURATEDMISSING2 node sHTTP 200 OPEN—Re-Probe →
clerk.comB2B SaaSCUSTOM CURATEDMISSING2 node sHTTP 200 OPEN—Re-Probe →
dub.coB2B SaaSCUSTOM CURATEDMISSING2 node sHTTP 200 OPEN—Re-Probe →
cal.comB2B SaaSCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
allbirds.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY0 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
gymshark.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY0 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
brooklinen.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY4 node sHTTP 200 OPENUCP ACTIVERe-Probe →
chubbies.comShopify D2CCUSTOM CURATEDMISSING1 nodeHTTP 200 OPENUCP ACTIVERe-Probe →
ridge.comShopify D2CAUTO BOILERPLATEMISSING0 nodesBOTH WAF 403UNAVAILABLERe-Probe →
cutsclothing.comShopify D2CMISSINGMISSING2 node sHTTP 200 OPENUCP ACTIVERe-Probe →
ruggable.comShopify D2CHTML SOFT-404MISSING1 nodeHTTP 200 OPENUNAVAILABLERe-Probe →
trueclassictees.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY3 node sHTTP 200 OPENUCP ACTIVERe-Probe →
carawayhome.comShopify D2CMISSINGMISSING2 node sHTTP 200 OPENUCP ACTIVERe-Probe →
beardbrand.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY2 node sHTTP 200 OPENUCP ACTIVERe-Probe →
siegemedia.comSEO AgencyMISSINGMISSING2 node sHTTP 200 OPEN—Re-Probe →
animalz.coSEO AgencyMISSINGMISSING0 nodesHTTP 200 OPEN—Re-Probe →
omniscientdigital.comSEO AgencyMISSINGMISSING1 nodeHTTP 200 OPEN—Re-Probe →
singlegrain.comSEO AgencyCUSTOM CURATEDMISSING3 node s · FAQPageHTTP 200 OPEN—Re-Probe →
directiveconsulting.comSEO AgencyCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
victorious.comSEO AgencyMISSINGMISSING2 node s · FAQPageHTTP 200 OPEN—Re-Probe →
webmechanix.comSEO AgencyAUTO BOILERPLATEMISSING5 node sHTTP 200 OPEN—Re-Probe →
klientboost.comSEO AgencyHTML SOFT-404MISSING0 nodesHTTP 200 OPEN—Re-Probe →
powerdigitalmarketing.comSEO AgencyCUSTOM CURATEDMISSING2 node sHTTP 200 OPEN—Re-Probe →
webfx.comSEO AgencyCUSTOM CURATEDMISSING3 node sHTTP 200 OPEN—Re-Probe →

How every domain was probed and classified

Each of the 40 public domains was probed directly over HTTPS using the same deterministic parser powering Verisource's production scanner: (1) GET /robots.txt parsed for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended directives; (2) homepage requests sent with explicit GPTBot/1.2 and PerplexityBot/1.0 User-Agent headers to catch silent CDN HTTP 403 / 429 blocks; (3) GET /llms.txt and GET /agents.md inspected for text/html SPA soft-404s and stock Shopify (shop.app/SKILL.md) or SEO plugin boilerplate; (4) homepage application/ld+json script nodes parsed for FAQPage entities; and (5) JSON-RPC tools/list handshakes tested against /api/ucp/mcp and /api/mcp on D2C commerce stores.

90% of DevTools ship /llms.txt. Only 20% ship /agents.md.

In a follow-up live HTTP probe across 10 developer infrastructure and B2B SaaS domains (trigger.dev, inngest.com, turso.tech, tinybird.co, knock.app, workos.com, stytch.com, upstash.com, highlight.io, and infisical.com), 9 of 10 (90.0%) served a valid /llms.txt directory, yet only 2 of 10 (20.0% — tinybird.co and infisical.com) published a custom /agents.md pricing and limits guardrail file, and only 1 of 10 (10.0% — upstash.com) shipped all 3 JSON-LD pillars.

Protocol / Schema SignalDevTools Cohort (N = 10)Commercial Impact
/llms.txt (HTTP 200)90.0% (9 / 10)High documentation-link adoption across modern DevTools
Custom /agents.md20.0% (2 / 10)80% leave pricing tiers & rate limits unguarded against AI hallucination
Full 3-Pillar JSON-LD10.0% (1 / 10)90% omit FAQPage or Offer pricing nodes on root pages

Common questions about the 2026 AI Search Readiness dataset

In our live HTTP readiness probe across 40 public domains spanning B2B SaaS, Shopify D2C, SEO agencies, and multi-location services, 50.0% (20/40) serve a custom-curated /llms.txt file, another 17.5% (7/40) expose auto-generated Shopify or SEO-plugin boilerplate, 7.5% (3/40) return HTML soft-404s with HTTP 200 status, and only 5.0% (2/40 — vercel.com and linear.app) publish a custom /agents.md specification.

Shopify rolled out platform-level /api/ucp/mcp endpoints and stock /agents.md files pointing to https://shop.app/SKILL.md across modern storefronts, which explains why 8 of 10 probed D2C stores respond on /api/ucp/mcp. However, stock Shopify files omit brand-specific free-shipping thresholds, return windows, and bundle exclusions unless the merchant overrides them via templates/agents.md.liquid or /pages/ai-facts.

Yes. Zero of the 40 probed domains disallowed GPTBot or PerplexityBot at root in /robots.txt, yet 5.0% (2/40 — ridge.com and westerndental.com) returned HTTP 403 Forbidden at the CDN/WAF layer when requested with GPTBot or PerplexityBot User-Agent headers. Testing both /robots.txt and simulated crawler User-Agent HTTP status codes is required.

See how your website stacks up against the 40-domain 2026 readiness cohort.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by Hacker News
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗