Crawlers

A breakdown of the Search, Agent, and Training defaults Cloudflare put in force on September 15, 2026. Search crawlers stay allowed, while Agent blocks and mixed-purpose crawler inheritance are the real paths that cut citations in AI answers.

Naver Blog blocks GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot outright in robots.txt. Tistory and velog carry no AI-crawler-specific rules, and Brunch blocks training bots only. Here is what each platform's raw robots.txt says and how that changes an article's odds of being cited in AI search.

GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript. Here is why CSR/SPA content disappears from AI search citations, and the fix sequence from a curl audit to server-side rendering, based on official documentation.

A diagnostic breakdown of why ChatGPT, Perplexity, and Google AI Overviews cite competitors while your brand goes unmentioned: covering crawler access, machine readability, entity recognition, third-party mentions, and content structure, with a step-by-step fix for each.

How to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt: based on OpenAI's and Google's official documentation.