Can ChatGPT read your blog?
Only if your words are in the page's HTML. ChatGPT, Claude and Perplexity crawlers do not run JavaScript, so content that loads with scripts is invisible to them. Robots.txt rules and firewalls such as Cloudflare can block them too. Paste a blog post URL to see exactly what AI crawlers see, what's blocking them, and how to fix it.
Tests
What is an AI crawler?
An AI crawler is a bot that AI companies use to read web pages. Some collect training data for AI models (GPTBot, ClaudeBot). Others fetch pages to build AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) or to read a page a user asked about (ChatGPT-User, Claude-User). Unlike Googlebot, they read only the raw HTML and don't run JavaScript.
Training crawlers
GPTBot · ClaudeBot
Collects pages to train AI models.
Search crawlers
OAI-SearchBot · Claude-SearchBot · PerplexityBot
Builds the index AI answers cite.
User crawlers
ChatGPT-User · Claude-User
Reads a page a person asked about.
All three read raw HTML only. No JavaScript runs.
How to check if AI can read your blog
Three steps, about ten seconds. The checker fetches your post the way AI crawlers do, then tells you exactly what to change.
Paste a blog post URL.
Use a single article, not your homepage, since that's what AI answers cite.
Read your score and the failed checks.
The headline shows how many of your words AI crawlers can read.
Apply the fix for your platform.
Each failed check shows the change to make in Next.js, Nuxt, Astro, SvelteKit, Remix, React or WordPress. Run the check again to confirm.
Ready? Paste a post at the top of the page.
Free, no signup, and nothing about your page is stored.
Eight checks, one score
What this AI crawler checker tests
We fetch your post the way AI crawlers do and read only what they read. Each check is weighted by how much it decides whether AI can read the post at all.
- 01
Content without JavaScript
How many words of article text are in the raw HTML, and whether the article only appears after JavaScript runs.
Weight35 pts - 02
Firewall and CDN blocks
Whether Cloudflare or another firewall returns an error or challenge page to AI crawlers.
Weight20 pts - 03
robots.txt
Rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended.
Weight15 pts - 04
Noindex
Meta robots tags and X-Robots-Tag headers.
Weight10 pts - 05
Article schema
BlogPosting or Article JSON-LD with headline, author and date.
Weight8 pts - 06
Canonical tag
Whether it points to the post itself.
Weight5 pts - 07
Headings
One H1 and a logical structure.
Weight4 pts - 08
Answer first
Whether the opening paragraph gets to the point.
Weight3 pts
Why can't ChatGPT read some websites?
There are three common reasons, and only one of them shows up in robots.txt.
That's why this checker tests live access and whether the article depends on JavaScript, not just your robots.txt file.
1. The content loads with JavaScript.
Google renders JavaScript, but the crawlers behind ChatGPT, Claude and Perplexity don't. A study of crawler traffic on Vercel's network found that none of the major AI crawlers execute JavaScript, even though GPTBot alone made 569 million requests in a month. In a test of a fully client-rendered site, ChatGPT, Perplexity and Claude all failed to read pages that Google could. If your blog is a React, Vue or other single-page app, AI may see an empty page.
2. A firewall blocks the crawler.
Cloudflare sits in front of about 22% of websites and offers one-click AI bot blocking. From 15 September 2026, new Cloudflare domains block AI training and agent crawlers by default on pages that show ads. These blocks don't appear in robots.txt, so a robots.txt check alone misses them.
3. robots.txt says no.
About 6% of sites block GPTBot in robots.txt, often through a blanket rule written for other bots.
Training bots vs search bots: which should you allow?
AI companies run separate crawlers for separate jobs, so you can block one and allow another.
| Company | Builds search answers | Fetches pages when a user asks | Collects training data |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | — |
| Googlebot (AI Overviews) | — | Google-Extended (Gemini training control) |
Want to be cited? Allow the search and user crawlers.
Blocking training crawlers is a separate choice and doesn't remove you from AI search.
robots.txt for AI crawlers: what to allow and block
robots.txt tells crawlers which pages they may fetch. Each AI crawler has its own name (its user agent), so you can allow AI search and block AI training separately. Our checker reads your robots.txt and tells you which rule affects each bot.
Common mistakes the checker catches
A blanket block. User-agent: * with Disallow: / blocks every crawler, including AI search and Google.
Blocking a whole company by blocking one bot. Blocking GPTBot stops training, not ChatGPT search; OAI-SearchBot is separate.
A rule in the wrong group. Rules apply to the User-agent lines directly above them. A stray blank line or a missing User-agent line can attach a rule to the wrong bot.
Thinking robots.txt is enough. It's a request that well-behaved crawlers follow. A firewall can still block a crawler that robots.txt allows, which is why we also test live access.
The most common choice for blogs that want to be cited.
# AI search and user-requested fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# AI training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Search, user fetches and training all welcome.
User-agent: *
Allow: /Not sure what your file says?
Open yourdomain.com/robots.txt, or paste any page URL above and we'll read it for you.
Server-render the post
How to fix it in your framework
Next.js
Fetch post content in a Server Component or with static generation, not in useEffect or a client component. Check that the article body appears in View Source.
Nuxt
Keep SSR on (the default) and load posts with useAsyncData or useFetch. Avoid ssr: false on blog routes and <ClientOnly> around the article.
Astro
Pages are static by default, which is ideal. Don't render the article inside a client:only component.
SvelteKit
Load posts in a +page.server.js or +page.js load function and keep ssr enabled for blog routes.
Remix
Load posts in the route loader so the HTML is complete on first response.
React
A client-side app (Vite or Create React App) sends an empty HTML shell. Prerender blog routes at build time, or move the blog to a server-rendered framework.
WordPress, Ghost, Webflow
Usually server-rendered and readable. If the check fails, look at security plugins, Cloudflare bot settings and page-builder blocks that load content late.
Using ContioReach? You're already covered.
Every official starter (Next.js, Nuxt, Astro, SvelteKit, Remix and React) renders posts on the server and ships Article schema, so posts pass this check by default.
How the checks work, and their limits
How it works
- 1
We download your robots.txt and work out which rule applies to each AI crawler.
- 2
We request your page once for each crawler, sending that crawler's published user agent, and record the response code and any challenge page.
- 3
We read the HTML exactly as received, without running JavaScript, and count the visible words.
- 4
We look for signs that the article only appears after JavaScript runs: an empty app container, article text that sits in the page's script data or RSS feed but not in the HTML, and "enable JavaScript" messages.
- 5
We check noindex, canonical, headings and schema in the raw HTML.
What it can't tell you
It isn't the real crawler. We send the same user agent, but some firewalls also check the crawler's IP address, so a real GPTBot may be treated differently. We say "likely blocked" when the result is uncertain.
It checks one page at a time. Other posts on your site may behave differently.
Being readable doesn't guarantee citation. Whether AI assistants cite you also depends on your content's quality and authority.
Crawlers change. We re-test our assumptions regularly and update this page when crawler behaviour changes. Last verified: 9 October 2026.
Privacy: we don't store page content.
Results are kept for 24 hours for caching, and stay private to you.
AI crawler questions, answered
Still have questions? Contact us
An AI crawler is a bot that AI companies use to read web pages. Some collect training data (GPTBot, ClaudeBot); others fetch pages for AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Unlike Googlebot, they read only the raw HTML and don't run JavaScript.
Yes, if your content is in the page's HTML and nothing blocks OpenAI's crawlers. ChatGPT's crawlers don't run JavaScript, so content that appears only after scripts load is invisible to them. Firewalls and robots.txt rules can also block OAI-SearchBot and ChatGPT-User. Run the checker above on one of your posts to find out.
No. OpenAI's crawlers download JavaScript files but don't execute them, according to Vercel's analysis of crawler traffic. The same is true for Claude's and Perplexity's crawlers. Google is the exception: it renders JavaScript, so a page can rank in Google and still be blank to ChatGPT.
Paste a page URL into the checker above. It reads your robots.txt, fetches the page as each AI crawler, and checks whether your article text is in the HTML they receive. You can also search your server logs for user agents such as GPTBot, ClaudeBot and PerplexityBot.
It can be. Cloudflare and other firewalls can block AI bots at the network level, with no rule in robots.txt. Cloudflare offers one-click AI bot blocking and, from 15 September 2026, blocks training and agent crawlers by default on new domains' pages that show ads. The firewall check above detects these blocks.
Only if you don't want your content used to train OpenAI's models. GPTBot collects training data; OAI-SearchBot and ChatGPT-User decide whether ChatGPT can find and cite you. You can block GPTBot and still allow the search crawlers.
Add a group for OpenAI's search crawlers: User-agent: OAI-SearchBot, User-agent: ChatGPT-User, then Allow: /. These let ChatGPT find and quote your pages. GPTBot is OpenAI's training crawler and can be allowed or blocked separately. If you use Cloudflare, also check its AI bot settings, since a firewall block overrides robots.txt.
Only the ones that choose to follow it. OpenAI, Anthropic and Perplexity say their declared crawlers respect robots.txt, but it's a request, not a lock. To actually stop a bot, block it in your firewall or CDN.
Not by itself. Being readable is the requirement, not the guarantee. Whether AI cites you depends on how well your content answers the question, how trusted your site is and what competing sources say. But a page AI can't read can't be cited accurately at all.
Not much today. An Ahrefs study of over 137,000 domains found that 97% of llms.txt files got no requests from AI crawlers, and Google has said it doesn't use the file. Server-rendered content and clean HTML matter far more.
Yes. No signup is needed to run a check, and results aren't stored beyond a 24-hour cache.
Build a blog AI can always read
ContioReach is a headless CMS for blogs. Write in our editor, publish through a simple API, and use any framework. Every official starter renders posts on the server and includes Article schema, so ChatGPT, Claude and Perplexity see your full post.