Research
/
Brand visibility

1 in 14 Software Company Websites Turned GPTBot Away at the Firewall

Published

2,661 software company websites checked on November 16, 2026, for robots.txt rules, firewall blocks, llms.txt files, and pricing pages.

Evidence
AI prompt data
Coverage
2,661 software websites
Summarize with AI

Firewalls turned away a visitor using GPTBot's name on 7.1% of software company websites, while robots.txt blocked ChatGPT's, Claude's, or Perplexity's search crawlers on 0.6%. That matters now because blocking no longer lives in robots.txt alone, and Cloudflare moved its default toward blocking AI crawlers. The blocks that do happen fall on AI training crawlers. They also come from firewalls that turn away a visitor using GPTBot's name where robots.txt sets no rule against it. A robots.txt check alone misses part of what an AI crawler meets.

The short version

  • AI search was rarely blocked: 16 of 2,584 software company websites (0.6%) blocked at least one AI search crawler in robots.txt, against 8 that blocked Googlebot.
  • Training bots were blocked about ten times as often: 157 of 2,584 websites (6.1%) blocked an AI training crawler, and 141 of those still let AI search in.
  • Firewalls added a second layer: 182 of 2,563 websites (7.1%) refused a visitor using GPTBot's name while serving the same homepage to a browser.
  • Almost half publish llms.txt: 1,251 of 2,655 websites (47.1%) answered with an llms.txt file.
  • Most showed no pricing page: 1,075 of 2,652 websites (40.5%) had a pricing page we could find, and where it showed a price, the price was almost always in the HTML AI crawlers read.

What to do with this

  1. Test the website as an AI crawler, beyond reading robots.txt. Request the homepage with the GPTBot, OAI-SearchBot, and ChatGPT-User user agents and check that the answer is the page, not a 403 or a challenge. Then check the firewall logs, since a firewall that verifies OpenAI's addresses can treat a test and the real crawler differently.
  2. If the website runs on Cloudflare, check the AI bot settings. One in nine Cloudflare websites here refused a GPTBot visitor, more than twice the rate elsewhere.
  3. Decide training and search separately. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, so blocking one does not block the other.

Read more: Agentic SEO

Robots.txt blocked AI search on 0.6% of websites

Each AI search crawler was blocked on 13 or 14 websites, and 16 blocked at least one. Training crawlers drew the blocks, led by Common Crawl's CCBot at 5.6% and OpenAI's GPTBot at 4.8%. One software company's robots.txt, for example, disallowed GPTBot and named OAI-SearchBot to let it in.

Training bots blocked far more than AI search

Share of 2,584 software company websites whose robots.txt blocks each crawler at the root path.

CCBotCommon Crawl, training
5.6%
GPTBotOpenAI, training
4.8%
Google-ExtendedGoogle, AI training
4.5%
ClaudeBotAnthropic, training
4.2%
PerplexityBotPerplexity search
0.5%
Claude-SearchBotClaude search
0.5%
OAI-SearchBotChatGPT search
0.5%
GooglebotGoogle Search
0.3%
0%2%4%6%
CrawlerRoleWebsites blocking itShare95% interval
CCBottraining144 of 2,5845.6%4.8% to 6.5%
GPTBottraining123 of 2,5844.8%4.0% to 5.7%
Google-Extendedtraining115 of 2,5844.5%3.7% to 5.3%
Meta-ExternalAgenttraining116 of 2,5844.5%3.8% to 5.4%
ClaudeBottraining109 of 2,5844.2%3.5% to 5.1%
Applebot-Extendedtraining108 of 2,5844.2%3.5% to 5.0%
ChatGPT-Useruser-triggered16 of 2,5840.6%0.4% to 1.0%
OAI-SearchBotsearch13 of 2,5840.5%0.3% to 0.9%
Claude-SearchBotsearch14 of 2,5840.5%0.3% to 0.9%
PerplexityBotsearch14 of 2,5840.5%0.3% to 0.9%
Claude-Useruser-triggered12 of 2,5840.5%0.3% to 0.8%
Perplexity-Useruser-triggered14 of 2,5840.5%0.3% to 0.9%
Googlebotsearch8 of 2,5840.3%0.2% to 0.6%
Bingbotsearch8 of 2,5840.3%0.2% to 0.6%
% of websites with a readable robots.txt (2,584) · root path disallowed · US-listed software companies and Y Combinator B2B companies, November 16, 2026 · Omnius crawl of software company websites

Only 591 websites (22.9%) named any AI crawler in robots.txt at all. US-listed software companies blocked AI search more often than Y Combinator startups, on 8 of 399 websites (2.0%) against 8 of 2,185 (0.4%).

Firewalls refused a GPTBot visitor on 7.1% of websites

We requested each homepage twice, once as a browser and once with GPTBot's user-agent string. On 182 websites the browser got the page and GPTBot got a refusal, most often a 403 from Cloudflare. On one US-listed software company's website, robots.txt set no rule against GPTBot, while Cloudflare answered our GPTBot request with a 403.

Firewalls turned away GPTBot's name on 7.1%

Websites blocking GPTBot in robots.txt, against websites whose firewall refused a GPTBot user agent but served a browser.

robots.txt blocks GPTBot123 of 2,584
4.8%
Firewall refused GPTBot user agent182 of 2,563
7.1%
… on Cloudflare websites115 of 1,010
11.4%
… on all other websites67 of 1,553
4.3%
0%3%6%9%12%
MeasureWebsitesShare95% interval
robots.txt blocks GPTBot123 of 2,5844.8%4.0% to 5.7%
Firewall refused GPTBot user agent, browser served182 of 2,5637.1%6.2% to 8.2%
Of those with a readable robots.txt, robots.txt also blocks GPTBot77 of 16845.8%38.5% to 53.4%
Explicit refusal only (no timeouts)141 of 2,5635.5%4.7% to 6.5%
… on Cloudflare websites115 of 1,01011.4%9.6% to 13.5%
… on all other websites67 of 1,5534.3%3.4% to 5.4%
% of websites · firewall test = one homepage request with the GPTBot user-agent string from our network, not OpenAI's · November 16, 2026 · Omnius crawl of software company websites

Our read: We think robots.txt and the firewall often disagree. Of the 168 refusing websites with a readable robots.txt, 77 also blocked GPTBot there, and 91 did not. Our test used GPTBot's name from our own network, so a firewall that checks OpenAI's addresses may treat the real crawler differently.

47.1% of websites publish an llms.txt file

The share was higher among Y Combinator startups (49.9%) than among US-listed software companies (33.3%).

Almost half publish an llms.txt file

Share of websites answering /llms.txt with a text file.

Y Combinator B2B companies1,104 of 2,214
49.9%
All software companies1,251 of 2,655
47.1%
US-listed software companies147 of 441
33.3%
0%20%40%60%
GroupWebsites with llms.txtShare95% interval
Y Combinator B2B companies1,104 of 2,21449.9%47.8% to 51.9%
US-listed software companies147 of 44133.3%29.1% to 37.9%
All1,251 of 2,65547.1%45.2% to 49.0%
All, strict (starts with a Markdown heading)1,218 of 2,65545.9%44.0% to 47.8%
% of live websites · /llms.txt answered 200 with text, not an HTML page · November 16, 2026 · Omnius crawl of software company websites

40.5% of websites had a pricing page

US-listed software companies published one far less often, 19.0% against 44.8% for Y Combinator startups.

Listed companies rarely publish pricing pages

Share of websites where a pricing or plans page was found.

Y Combinator B2B companies991 of 2,211
44.8%
All software companies1,075 of 2,652
40.5%
US-listed software companies84 of 441
19.0%
0%10%20%30%40%50%
GroupWebsites with a pricing pageShare95% interval
Y Combinator B2B companies991 of 2,21144.8%42.8% to 46.9%
US-listed software companies84 of 44119.0%15.7% to 23.0%
All1,075 of 2,65240.5%38.7% to 42.4%
% of live websites · pricing page linked from the homepage or at /pricing or /plans · November 16, 2026 · Omnius crawl of software company websites

Where a pricing page showed a price, it was almost always in the raw HTML, on 760 of the 807 rendered pricing pages that showed one (94%). On the other 47, or 4.5% of all 1,051 rendered pricing pages, the price appeared only after JavaScript ran, which hides it from crawlers that read HTML alone.

Where a price was shown, it was in the HTML

1,051 rendered pricing pages, by whether a real price appears in the raw HTML, only after JavaScript runs, or not at all.

Price in raw HTML760 of 1,051
72.3%
No price found244 of 1,051
23.2%
Price only after JavaScript47 of 1,051, hand-checked
4.5%
0%20%40%60%80%
OutcomePagesShare
Price in raw HTML760 of 1,05172.3%
Price only after JavaScript (hand-checked)47 of 1,0514.5%
No price found244 of 1,05123.2%
Pricing page found at all1,075 of 2,652 websites40.5%
% of rendered pricing pages · raw HTML is what crawlers that skip JavaScript read · November 16, 2026 · Omnius crawl of software company websites

Our read: We think the bigger pricing gap for AI is the missing price, not the hidden one. JavaScript hid prices on few pages, while about two in three websites showed no price on a pricing page we could find.

Methodology

Data. 2,661 software company websites from two public lists, checked on November 16, 2026: every company in Nasdaq's three software industries (441) and Y Combinator's active B2B companies (2,220).

Method. For each website we read robots.txt, requested the homepage as a browser and as GPTBot, checked for /llms.txt, and found and rendered the pricing page. "Blocked" means robots.txt disallows the whole website. A hand check confirmed the price detector on 39 of 40 pages.

Limits. One day, from one network. The firewall test used GPTBot's name, not OpenAI's addresses, so the real crawler may be treated differently.

Sources

Platform documentation

  1. Cloudflare, "Content Independence Day: no AI crawl without compensation," July 1, 2025. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/
  2. OpenAI, "Overview of OpenAI Crawlers," accessed November 16, 2026. https://developers.openai.com/api/docs/bots

Omnius articles this note builds on

Monthly Growth OpenLetter.

Learn how to scale user acquisition without scaling costs from our findings. We spent years exploring, so you don't have to.

Your submission has been received!              
Oops! Something went wrong while submitting the form.

Related research.

Become the answer of ChatGPT Google Claude Gemini Perplexity

Omnius is an AI-native SEO & GEO agency maximizing the probability of B2B SaaS & Fintech companies being included in AI answers, selected by agents, and chosen by humans.

Trusted by
Our technology is used by
Referenced by