Scraping is automated collection of pages and data: product cards, prices, stock, blog posts, documentation files, API endpoints. The effect: origin traffic grows, sales do not, and the copy lands on a foreign domain or in a model that quotes your text.
robots.txt is a contract with polite agents. A commercial scraper does not read it. Then the edge and a limit on the store server count.
Why protect the content
You use this when the catalogue and guides are copied onto foreign domains, comparison sites or model training. Separately: crawlers from OpenAI, Anthropic, ByteDance. What belongs in Google does not have to go into training.
A fake copy of the store, another domain and the same photos, is DNS and a legal process. Ops collects the log: from where, which path, how many requests.
How to set it up
In robots.txt: allow Googlebot on the catalogue. Block checkout, account and the panel. For training bots block everything if the content must not go into a model. That will not stop a hostile scraper. It sorts the polite ones.
In Cloudflare on listing and blog: a low bot score gets a challenge. A limit of a few dozen requests per minute per IP on listing. Category cache can serve HTML without PHP. The content still leaves, so cache does not replace a block.
On Magento, a limit on GraphQL. On WordPress, turn xmlrpc off if you do not use it. Compare a day’s log before and after. Note on AI crawlers: AI threats. Edge: security.
In short: polite bots are handled by robots.txt. The rest is cut by a limit and the edge. Cache takes load off the server. It does not protect the content.