How to check if ChatGPT can read your website
ChatGPT can read your website when five things are true: robots.txt allows OAI-SearchBot and ChatGPT-User, your CDN or firewall does not block them, they receive your page and not an error, the content is in the HTML without needing JavaScript, and the page says plainly who you are. The same five checks apply to Claude and Perplexity under their own crawler names.
By Gavin Levine, Founder and Principal Consultant
1. Read the robots.txt your site serves
Open yourdomain.com/robots.txt in a browser. Look for OAI-SearchBot, ChatGPT-User and GPTBot, and for a blanket "User-agent: *" followed by "Disallow: /".
2. Check the CDN and firewall
A crawler can be refused before it ever reaches robots.txt, so check the security settings in front of your site too.
3. See what a crawler receives
Request your home page while identifying as the crawler, and look at the status code. 200 means the page was served. 403 means something refused it.
4. Check the content does not need JavaScript
AI crawlers mostly read the HTML they are sent and do not run JavaScript. View the page source, not the rendered page, and search it for a sentence from your main content. If the sentence is missing, a crawler sees an empty shell.
5. Check the page says who you are
A crawler that gets in still has to work out what the business is. The home page should state the name, what you do, where, and how to reach you, in text and in structured data. Consistent details across your site, your Business Profile and your listings make the answer easier to trust.
The quick way
Our free scan runs these checks in about 10 seconds and shows what it found. It reads your public home page, robots.txt, sitemap and llms.txt. Like any outside test, it cannot see inside your firewall.
Questions people ask
Why does ChatGPT say it cannot access my site?
Usually because the request was refused: robots.txt disallows its crawler, a CDN or firewall blocked it, or the server returned an error. Less often the page loaded but the content needed JavaScript.
If ChatGPT can read my site, will it recommend me?
Not necessarily. Access is the first requirement. Being recommended also depends on how clearly your pages answer the question and on what other sites say about you.
Is GPTBot the crawler ChatGPT uses to answer questions?
No. GPTBot collects training data. ChatGPT’s search results come from OAI-SearchBot, and ChatGPT-User fetches a page when a person asks about it.
See what AI crawlers find on your site
The scan checks whether ChatGPT, Claude, Perplexity and Google can get into your site, read it and tell who you are. You keep the results either way.