Glossary

Review the definitions for terms used across Cloudflare's AI Crawl Control documentation.

Term	Definition
AI crawler	A bot which scrapes content from websites in support of an AI model, including by scraping content for indexing, retrieval augmented generation, or training.
In-band pricing	A pricing model where pricing information is transmitted within the same communication channel as the content itself. In Pay Per Crawl, in-band pricing means the price is included in the HTTP response headers (crawler-price) alongside the actual content, rather than being retrieved from a separate system or API call. When custom pricing is enabled, Cloudflare sets the cf-pay-per-crawl header to pricing=in-band, indicating that the origin server or Worker can dynamically set prices by including a crawler-price header in the response.
Merchant of Record	The entity who facilitates "buying and selling". For pay per crawl, Cloudflare is the merchant of record.
robots.txt	A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can enforce robots.txt with Cloudflare WAF custom rules.