How to limit scraping of the catalogue, blog and documentation
Opselis
A scraper dumps product cards, articles and the API. robots.txt, cache, rate limit and Cloudflare AI Crawl Control. How content collection works and how to set limits on listing, search and docs.
Scraping is automated collection of HTML and JSON: product cards, prices, stock, blog posts, documentation files, REST endpoints. Tools on your side are robots.txt, full-page cache, Rate limiting and Bot Management in Cloudflare, plus AI Crawl Control for training bots. robots.txt is a contract with polite agents. A commercial scraper does not read it. Then the edge and a limit on origin count.
What content protection is for
You use it when the catalogue and guides are copied onto foreign domains, comparison sites or a model that quotes your description. Effect: origin traffic grows, conversion does not, PHP CPU goes to listings. Separately: crawlers from OpenAI, Anthropic, ByteDance. What belongs in Google does not have to go into training. A fake copy of the store (another domain, the same photos) is DNS and a legal process. Ops collects the log: User-Agent, ASN, paths, volume.
How collection and the limit work
A scraper walks the sitemap, page= on the listing, /graphql and /rest/V1/products. Rate limit counts requests per IP or TLS fingerprint in a window (e.g. 60 per minute on /catalogsearch). Over the threshold: 429 or a challenge. Cache (Varnish, Cloudflare Cache) serves the listing without PHP, so a dump is cheaper for you, but the content still leaves. That is why cache does not replace Block on a low bot score. AI Crawl Control in Cloudflare shows the crawler operator and whether it got a 200. Action: allow, block, robots.
How to set it up
At the site root, robots.txt: Allow Googlebot on the catalogue. Disallow on /checkout, /customer, /admin. For GPTBot, CCBot, anthropic-ai: Disallow / if the content must not go into training. That will not stop a hostile scraper, it sorts the polite ones.
Cloudflare, Security, Bots and WAF: on http.request.uri.path contains "/catalogsearch" or /blog, score lt 30 → Managed Challenge. Rate limiting rule: 30 req/min per IP on listing. Cache Rule: cache category HTML, Bypass on cart and account. AI Crawl Control: review operators, Block those you do not want.
On Magento origin: a limit on GraphQL (a module or nginx limit_req on the /graphql location). WordPress: turn xmlrpc off if unused. A 24 h log to compare before and after. Note on AI crawlers: AI threats. Edge: security.