Paste robots.txt to check AI crawler access (GPTBot, ClaudeBot, PerplexityBot, Google-Extended).
—
Plugsky is OpenAI-compatible — see docs and start on the free plan (2 free models, no card).
The Robots.txt Tester for AI Crawlers is a free page for checking how your robots.txt rules affect AI crawlers such as GPTBot, ClaudeBot and Google-Extended. It helps site owners see whether those agents are allowed or blocked, which matters for visibility in AI answers and for controlling how content is used. It suits SEOs, publishers and developers. No sign-up is required. Confirm behaviour against each crawler's official documentation.
That is a business decision. Blocking can protect content from being used in AI training or summaries, but it may also reduce how often your brand appears in AI answers. Many publishers allow answer engines while blocking training crawlers, and rules can differ per user-agent.
Add a User-agent group for the specific token, such as GPTBot or ClaudeBot, with Disallow rules, while leaving the wildcard group permissive for search engines. Test after each change, because a single syntax error or conflicting group can allow or block more than intended.
No. Robots.txt is a voluntary convention and only applies to crawlers that honour it. It does not control data already collected, content accessed by users, or scrapers that ignore the file. For stronger control, combine it with terms of service, rate limiting and access controls.
Canonical pricing and plans: plugsky.com/#sec-pricing · Terms · SLA · Docs