Is httpbin.org visible to AI?
Single-page scan of https://httpbin.org/base64/PGh0bWw+PGhlYWQ+PHRpdGxlPlRJTlk8L3RpdGxlPjwvaGVhZD48Ym9keT48cD5zbWFsbDwvcD48ZGl2IGlkPWQ+PC9kaXY+PHNjcmlwdD4KdmFyIEY9InFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIHFxd3cgZWVydCB5eXV1IGlpb3AgYWFzZCBkZGZmIGdnaGggamprayBsbHp6IHh4Y2MgdnZiYiBubm1tIjsKZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoImQiKS50ZXh0Q29udGVudD1GOwpkb2N1bWVudC50aXRsZT0iSlNPS19aWloiOwp2YXIgdT0iaHR0cHM6Ly93ZWJob29rLnNpdGUvYTI3NzI1MTItM2Q5ZS00OTY3LWE2NWUtZjdkOTVlZjhhMWRlL2pzP3Y9Mjc4MyZ0PSIrZW5jb2RlVVJJQ29tcG9uZW50KGRvY3VtZW50LnRpdGxlKTsKdHJ5e25ldyBJbWFnZSgpLnNyYz11KyImaW1nPTEifWNhdGNoKGUpe30KdHJ5e2ZldGNoKHUrIiZmPTEiLHttb2RlOiJuby1jb3JzIn0pfWNhdGNoKGUpe30KdHJ5e3ZhciBxPW5ldyBYTUxIdHRwUmVxdWVzdCgpO3Eub3BlbigiR0VUIix1KyImeGhyPTEiLHRydWUpO3Euc2VuZCgpfWNhdGNoKGUpe30KPC9zY3JpcHQ+PC9ib2R5PjwvaHRtbD4= · checked 2026-10-29
AI crawler access — 8 engines
| Crawler | Org | Role | robots.txt | Live fetch | Verdict |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | allowed | HTTP 200 · 950 B | Open |
| ClaudeBot | Anthropic | training | allowed | HTTP 200 · 950 B | Open |
| Google-Extended | training | allowed | — | Open | |
| CCBot | Common Crawl | training | allowed | HTTP 200 · 950 B | Open |
| Bytespider | ByteDance | training | allowed | HTTP 200 · 950 B | Open |
| PerplexityBot | Perplexity | assistant | allowed | HTTP 200 · 950 B | Open |
| OAI-SearchBot | OpenAI | search | allowed | HTTP 200 · 950 B | Open |
| ChatGPT-User | OpenAI | agent | allowed | HTTP 200 · 950 B | Open |
What we found
Embed your score badge
Paste this into your site, README or docs — it links back to this public report:
FAQ
As of 2026-10-29, 8 of 8 major AI crawlers can access the homepage. The test covers GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot, Bytespider (ByteDance), PerplexityBot, OAI-SearchBot and ChatGPT-User.
Most AI crawlers can read this site, but some are restricted or get limited content. A few robots.txt or server-side tweaks would open it up fully.
Allow AI crawlers in robots.txt, remove noindex from key pages, publish substantive well-structured content, and add JSON-LD structured data. Re-run the free scan after each change to track the score.