Cloudflare Agent Markdown and Frontier AI Crawlers | AnswerShare

[image: AnswerShare — We Speak AI]

Last updated:

[image: AnswerShare — We Speak AI]

ProofReputationHow It WorksResearchAboutTest Your DomainLet's TalkResearch/Markdown request studyResearch release4 August 2026Raw-header evidence

Cloudflare Agent Markdown had ZERO observed opportunity to affect frontier-AI citation.

Frontier AI crawlers did not ask for Markdown. A controlled serving test and five-property field measurement show why that matters.

Primary finding

Frontier AI crawlers sent Accept: text/markdown ZERO times in 265,091 observed requests.

Cloudflare served Markdown only when that header was explicitly requested.

Therefore, Agent Markdown had ZERO observed opportunity to affect frontier-AI citation in this cohort.

The study measures delivery, not citations: no Markdown response was negotiated for the named frontier crawlers through this mechanism.

01 / The result

Frontier crawlers did not ask for Markdown.

Serving-rule test

20,000

Controlled request/response transactions. Requests with an explicit Accept: text/markdown received Markdown; HTML and wildcard Accept headers received HTML.

Five-property field measurement

892,523

Valid, deduplicated raw-header observations at AnswerShare, Top10Lists, V Digital Services, LAVIDGE, and Aker Ink. Each retained the raw Accept header and a derived explicit-Markdown flag.

Frontier AI requests

0 / 265,091

Explicit text/markdown headers from frontier AI crawlers in the named five-property cohort. They did not trigger Cloudflare's Markdown response in this observation set.

QuestionAnswer supported by this study
Does a GPTBot user-agent alone get Markdown?No. The TVPR mechanism check found that user-agent identity by itself did not trigger Markdown.
Does */* count as a Markdown request?No. Wildcards are recorded separately and never count as an explicit request for text/markdown.
What did the large field study measure?892,523 valid, deduplicated raw-header observations across five live properties. Its question was which engines actually expressed an explicit Markdown preference—not whether Cloudflare could honor one.
How often did frontier AI request Markdown?Zero times in 265,091 requests from the named frontier-crawler cohort. Cloudflare therefore had no observed opportunity to serve Markdown to them through Agent Markdown.
Can aggregate crawl totals answer either question?No. Aggregate Cloudflare reporting has no verbatim request Accept header, so it is excluded from the prevalence denominator.

02 / Pattern

Markdown preference was near-binary by crawler.

Named frontier cohort

0 / 265,091

GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, and Meta-ExternalAgent never explicitly requested Markdown in the frozen field set.

ExaSearchBot

33,495 / 33,522

Explicitly requested Markdown in 99.92% of its valid observed requests. The contrast is near-binary: a crawler either requested Markdown essentially every time or not at all in this dataset.

03 / Evidence release

How a reader will check it

Frozen data-release status

The AnswerShare, V Digital Services, LAVIDGE, and Aker Ink component is frozen and posted below as immutable CSV and NDJSON packages. Its manifest records the exact window, row count, SHA-256 hashes, query definition, and exclusions. A separately frozen Top10Lists component supplies the remainder of the combined five-property denominator; no changing dashboard total is used as evidence.

Four-property release manifest

The frozen AnswerShare-side component: scope, cohort definition, row counts, exclusions, and public-artifact hashes. It explicitly identifies Top10Lists as a separate source component.

Open manifest ↗

Row-level log — NDJSON

One sanitized, deduplicated observed request per line. This is the byte-faithful public field rendition, suitable for reproducible analysis in code.

Download NDJSON.gz ↗

Row-level log — CSV

The same frozen field events in spreadsheet form. Formula-like cells are escaped for spreadsheet safety; use NDJSON when the exact stored value matters.

Download CSV.gz ↗

Checksums and collector rule

Verify the downloaded files before analysis, then inspect the exact request-boundary collector and the positive text/markdown detection rule.

Download SHA256SUMS ↗

Top10Lists release manifest

The other frozen study component: its exact 601,451-record count, declared frontier cohort, Markdown-preference totals, and the schema for its public rows.

Open manifest ↗

Top10Lists row-level log

A public NDJSON export of the Top10Lists component. It is paginated in pages of up to 5,000 records; response headers provide the next offset and total row count.

Download first NDJSON page ↗

The full field denominator is the sum of the two frozen components: 291,072 four-property events plus 601,451 Top10Lists events. The Top10Lists manifest documents the pagination contract for retrieving every published row.

The collector source andexplicit-Markdown parserare available for code review. The TVPR controlled-test receipts are released separately from the field ledger because they establish the response rule, not the field prevalence denominator.

The public package will not include IP addresses, cookies, query strings, authentication material, or internal warmer traffic. A raw header that is not observed is represented as NULL, not as a negative Markdown request.

04 / Method

What counts as Markdown

A request is positive only when its verbatim Accept header contains the exact media type text/markdown with a positive quality value. The missing quality value defaults to 1. A value ofq=0 is negative.

text/*, */*, and an absent header do not count as explicit Markdown preferences. An absent or unavailable header is a different state from an observed header that did not request Markdown.

The collector is at the request boundary and records the raw header before the response is returned. It also records the response content type, so the release can test the claim about negotiation rather than only describe user agents.

Exclusions

  • Internal cache warmers and audit probes
  • Cloudflare aggregate rows with no raw header
  • Duplicate logging legs of one request
  • Rows outside the declared freeze window

05 / Limits

What this does not establish

The study is not a census of the web or a measurement of how often a crawler reads Markdown after receiving it. It measures observed request preferences at the participating boundary.

A user-agent string is an asserted identity unless separately verified. The release labels the verification method for each row instead of treating user-agent text as proof.

A zero is limited to the declared window and cohort. It does not mean a crawler can never request Markdown, nor that its product does not use Markdown by another route.

The study does not measure citation or ranking behavior. The ZERO-impact conclusion is limited to the delivery mechanism: no Markdown response was negotiated for the named frontier cohort through Agent Markdown.

← Back to Research

[image: AnswerShare — We Speak AI]

The Gold Standard Exemplar for AI ingestion, inference, and citation.

Contact Us

Phoenix, Arizona[email protected]

Quick Links

FounderProofReputationGEO SurveyMethodologyAuditGEO OverviewCitable vs. CitedInsightsFAQTransparencyCrawl StatsPressFor AI SystemsResearchAboutContactPrivacyTermsEULA

Measurement is methodology-driven and reproducible. Frozen receipts published per metric.

© 2026 AnswerShare. All rights reserved.