The Hidden Truth of AI Traffic: A Deep Dive into Widespread Bot Spoofing

[DATELINE] In an era increasingly dominated by artificial intelligence, the digital landscape is undergoing a profound transformation. Websites are grappling with new forms of interaction, not just from human users but from a burgeoning population of AI assistants and crawlers. However, a recent, meticulously documented experiment by web platform creator Duane Forrester reveals a disturbing undercurrent: a vast majority of self-reported AI and search engine bot traffic is fraudulent. His findings, derived from a brand-new website, expose a silent epidemic of impersonation that has significant implications for web publishers, SEO professionals, and even the integrity of AI model training.

Forrester’s investigation into the logs of his newly launched platform, CitationIQ.com, uncovered that a staggering 81.8% of claimed AI assistant visits and an equally alarming 87% of alleged Googlebot requests were fakes. These impostors, often masquerading as legitimate entities, pose not only a data integrity challenge but also a potential security risk, with some attempting to access sensitive system files. This revelation underscores a critical vulnerability in how web traffic is currently measured and highlights the urgent need for robust verification protocols.

Main Facts: A Digital Identity Crisis Unveiled

The core findings of Forrester’s two-week study on CitationIQ.com paint a stark picture of digital deception:

• AI Assistant Traffic: Out of 33 requests claiming to be from AI assistants like ChatGPT-User or Claude-User, only six were verified as legitimate. The remaining 27 were imposters, resulting in an 81.8% spoof rate. Disturbingly, many of these fake AI bots were not seeking content but rather attempting to access sensitive configuration files like .env.production, secrets.yaml, and config.json.
• Googlebot Impersonation: The problem extends to traditional search engine crawlers. Of 799 requests identifying as Googlebot, a mere 107 were genuinely from Google’s verified IP addresses. This translates to an 87% spoof rate, confirming a long-standing issue of Googlebot being the most impersonated bot on the web.
• Training vs. Retrieval Bots: The study differentiated between "demand" fetches (live AI assistant interactions) and "scheduled" crawls (for indexing and training AI models). Even among legitimate crawlers, the landscape is diverse, with Anthropic’s ClaudeBot showing significant activity on the new site, surpassing verified Googlebot and OpenAI’s GPTBot.
• The Unverifiable Threat: An initial 16 requests claiming to be from Common Crawl’s CCBot were marked "unverifiable." Subsequent manual investigation across four independent angles (published IP lists, reverse DNS, Common Crawl’s public index, and WHOIS lookups) confirmed all 20 CCBot-labeled requests were imposters.
• Gemini’s Opacity: Google’s AI model, Gemini, proved impossible to measure directly. Unlike competitors that expose distinct bots for training and retrieval, Google bundles these activities under the general Googlebot crawl, using an invisible Google-Extended token as a permission flag rather than a distinct fetcher. This mirrors the "not provided" keyword data challenge faced by SEOs over a decade ago.
• Perplexity’s Ambiguity: Perplexity’s crawler presented a murkier picture, with 24 of 36 requests failing IP verification. However, Perplexity has been known to operate from addresses outside its published ranges, making a definitive "spoofed" label challenging without further information.

These findings are not just statistical anomalies but indicators of a systemic issue that compromises data integrity and introduces security vulnerabilities across the internet.

Chronology: The Experiment Unfolds

The genesis of this revealing experiment lies in the launch of CitationIQ.com, a platform designed to track the crucial gap between a website being fetched by AI and its content actually being used or cited in AI-generated answers. With zero marketing spend, Forrester anticipated modest traffic. His primary goal was to obtain a clear, accurate understanding of his early visitors, particularly robots and crawlers, as Google Analytics 4 typically handles human user data.

"I went looking for a quiet, accurate read of who (robots and crawlers, since Google Analytics 4 handles the rest) was visiting, expecting small numbers, and I got them," Forrester recounts. "What I did not expect was that most of even these modest numbers were lies."

The journey to uncover this deception began with a fundamental understanding of how bots identify themselves. When a bot accesses a webpage, it sends a "user-agent" string in the request header, declaring its identity—e.g., "ChatGPT-User," "Claude-User," or "Googlebot." Critically, this name is self-reported and can be easily faked. "It is a stranger at your door in a delivery uniform, and the uniform is easy to fake," Forrester explains.

To cut through this deception, Forrester devised a simple yet powerful verification method. Major bot operators—such as OpenAI, Anthropic, and Google—publish lists of the actual IP addresses their bots use. A request is deemed legitimate only if its self-reported name matches an entry in the log and its source IP address falls within the published range for that specific bot. This "IP is the proof" methodology formed the bedrock of his analysis.

Forrester built his verification system using a concise, 15-line Python script utilizing only the standard library. This script, at its core, loads a vendor’s published IP ranges (typically in JSON format) and then checks if a given IP address falls within any of those ranges. His system went beyond a simple pass/fail, incorporating three outcomes:

1. Verified: The IP address is within the published range.
2. Spoofed: The IP address is not within the published range, despite the bot claiming the name.
3. Unverifiable: The system could not determine legitimacy, often due to a list failing to load or a record being missing. This category proved crucial for further manual investigation.

The full working version of his script extended this core functionality to read actual log lines, map each bot name to its corresponding published list, handle the "unverifiable" state, and incorporate reverse DNS checks for operators like Common Crawl, which rely on them. This meticulous approach allowed Forrester to systematically expose the true nature of his website’s robotic visitors.

Supporting Data: A Deep Dive into Deception and Legitimate Activity

The data collected over two weeks on CitationIQ.com provides a granular view of both malicious impersonation and the genuine activity of AI and search crawlers.

The Demand Gap: AI Assistant Impersonation

The "demand signal" refers to live fetches made by AI assistants during real user sessions, identifiable by user-agent names ending in "-User." Of the 33 such requests logged, only six were verified as legitimate, translating to an 81.8% spoof rate.

The intentions of these fake AI assistants were particularly concerning. While genuine assistant fetches typically target actual content pages, the spoofed bots, despite claiming AI assistant identities, were actively "hunting for .env.production, secrets.yaml, and config.json." These are critical configuration files that could expose sensitive environment variables, API keys, or database credentials. This pattern strongly suggests these were credential scanners or malicious actors leveraging trusted AI names to bypass security filters. The IP verification method successfully flagged every one of these malicious attempts.

Forrester cautions against extrapolating these numbers too broadly, given the small sample size from a new site. However, he emphasizes its value as a personal baseline and a stark warning to other website owners.

The Bigger Number, Which Is Not News: Googlebot Spoofing

The scale of impersonation was even larger for Googlebot. Out of 799 requests claiming to be from Googlebot, only 107 originated from verified Google IP addresses. The remaining 692, representing approximately 87% of Googlebot-labeled traffic, were not Google.

This finding, while shocking in its scale, is not new. Googlebot has consistently been the most impersonated crawler on the internet for nearly two decades. Google itself explicitly advises webmasters to verify Googlebot via IP address rather than trusting the user-agent string alone. The data from CitationIQ.com simply reconfirms this persistent pattern, demonstrating its immediate and widespread nature even on a brand-new, unpromoted website. Some fake Googlebot requests even used user-agent strings associated with Google products that had been retired years ago, indicating lazy or outdated scanning practices.

Two Different Games: Retrieval and Training Crawlers

Beyond the "demand" fetches, Forrester’s analysis delved into "scheduled" crawlers responsible for indexing and training AI models. These are distinct bots—ChatGPT-User is not GPTBot, and Claude-User is not ClaudeBot. Stripping away the fakes, the verified crawl data revealed interesting patterns in how different AI entities interact with new content.

• Retrieval Crawlers: These bots build indexes that allow AI assistants to pull current information into answers. They are crucial for a website’s immediate visibility and relevance in AI-driven search.
• Training Crawlers: These bots harvest content that may be integrated into the foundational weights of future AI models. A visit from a training crawler is not about immediate referral traffic but about making a "deposit into a corpus used to build models that will answer questions for years, often without ever fetching you again." The payoff is delayed, compounding, and typically invisible to standard analytics dashboards.

On CitationIQ.com, the most active verified crawler was Anthropic’s ClaudeBot, with 166 confirmed crawls. This surpassed verified Googlebot (107 crawls), OpenAI’s GPTBot (46 crawls), and OpenAI’s dedicated search crawler (40 crawls). While this is a snapshot from a new site, it offers a glimpse into which AI entities are actively indexing new, unpromoted web content—a strategic signal for future web visibility.

The One I Had To Chase: CCBot

The "unverifiable" category proved invaluable in the case of Common Crawl’s CCBot. Common Crawl produces one of the largest open datasets used to train a significant portion of AI models. Forrester’s initial report showed zero verified CCBot requests, four spoofed, and 16 unverifiable. The 16 "unverifiable" entries prompted a manual investigation.

1. Published List Check: No CCBot-labeled request fell within Common Crawl’s published IP ranges.
2. Reverse DNS: Four requests resolved to non-Common Crawl hostnames, and the remaining 16 had no reverse DNS records, explaining their "unverifiable" status.
3. Corpus Check: Forrester checked Common Crawl’s public index for his domain across the three most recent monthly crawls. No record of his domain was found.
4. Ownership (WHOIS): WHOIS lookups on the raw IPs revealed that all traced to commodity hosting providers across several countries, typical infrastructure used by scanners.

Four independent verification angles converged on one conclusion: all 20 CCBot-labeled requests were imposters. This detailed chase highlights the importance of the "unverifiable" category as an invitation for deeper investigation, rather than a dead end.

The One I Could Not Measure: Gemini

A significant blind spot in AI bot measurement is Google’s Gemini. Unlike OpenAI, Anthropic, and Perplexity, which expose distinct, verifiable signals for their training, retrieval, and live user-driven fetches, Google bundles these activities. There is only one Googlebot crawl. Whether the content gathered feeds Gemini training is controlled by a robots.txt token called Google-Extended, which is not a crawler itself, but a permission flag on an existing crawl.

Consequently, there is no "Gemini fetcher" in logs by design, making it impossible to measure Gemini demand by name. Forrester’s script found no requests claiming to be Gemini, indicating even impersonators haven’t bothered with that name. However, it did catch four requests announcing themselves as Google-Extended while fetching pages. Since Google-Extended cannot fetch, these were immediately identified as fake based on name alone, prior to any IP check.

This approach by Google echoes the "not provided" era for keyword data, where granularity was replaced by a flag. While competitors offer verifiable, separate events for different AI interactions, Google bundles them, leaving webmasters with limited visibility beyond confirming a general Googlebot presence.

Two Honest Asterisks: Perplexity

Perplexity’s crawler presented a more complex scenario. While 24 of 36 requests failed the IP check, Perplexity has been known to fetch from addresses outside its published ranges. This means some failures could be impersonators, but others could be legitimate Perplexity activity operating off-list, making the "spoofed" label ambiguous in this specific case. Forrester acknowledges this nuance and reiterates the overall small sample size.

Official Responses: An Industry Grappling with Transparency

While Forrester’s article does not include direct "official responses" from major tech companies to this specific study, the context of his findings highlights a broader industry challenge concerning bot transparency and verification.

Google, for instance, has long acknowledged the issue of Googlebot impersonation. Its official documentation explicitly advises webmasters to perform reverse DNS lookups and IP verification to confirm legitimate Googlebot activity. This long-standing directive underscores the pervasive nature of spoofing and the company’s awareness of the need for rigorous checks. The fact that the problem persists, with an 87% spoof rate even on a new site, suggests that while the advice exists, its implementation and the broader deterrence of malicious actors remain insufficient.

The differing approaches to AI bot identification among major players like OpenAI, Anthropic, and Google also reveal a philosophical split. OpenAI and Anthropic, by providing distinct user-agents and verifiable IP ranges for their various AI functions (e.g., ChatGPT-User, GPTBot, Claude-User, ClaudeBot), offer a degree of transparency that allows web publishers to understand and manage their interactions. This enables more granular control via robots.txt and more accurate analytics.

Google’s strategy with Google-Extended for Gemini, however, points to a different philosophy—one that prioritizes a unified crawl while offering a post-hoc permission flag for AI training. While technically functional, this approach inherently limits the visibility of web publishers into how their content is being used by Google’s AI. This lack of distinct, verifiable signals for Gemini demand or training mimics the "not provided" issue that obscured search query data years ago, leaving webmasters once again in the dark about a critical aspect of their digital footprint.

The industry’s response to bot spoofing has largely been reactive, relying on webmasters to implement verification. There’s no widespread, standardized authentication mechanism for bots, making user-agent strings inherently unreliable. This creates an environment ripe for exploitation by malicious actors, scrapers, and even competitors seeking to gather intelligence under false pretenses. The lack of a unified "bot authentication standard" means that for the foreseeable future, the onus of verification will remain on individual website owners.

Implications: The Unseen Costs of Digital Deception

The implications of widespread bot spoofing extend far beyond skewed traffic numbers, impacting security, data integrity, resource allocation, and strategic decision-making for web publishers and SEOs.

Data Integrity and Analytics Skew

The most immediate impact is on data integrity. If 80-90% of reported AI or Googlebot traffic is fake, then any conclusions drawn from standard analytics logs regarding bot activity are fundamentally flawed. This can lead to misinformed decisions about server capacity, content performance, and overall SEO strategy. Webmasters might over-allocate resources based on inflated bot activity or misinterpret the true interest of AI models in their content.

Security Vulnerabilities

The discovery that spoofed AI assistants were actively probing for sensitive configuration files (.env.production, secrets.yaml, config.json) is a serious security concern. Malicious actors are clearly leveraging the trusted names of AI bots to slip past basic filters and conduct reconnaissance for potential data breaches. If not caught by IP verification, these attempts could lead to significant security compromises, exposing sensitive information, and potentially leading to full system access. This highlights the need for robust, multi-layered security protocols that do not rely solely on user-agent strings for identification.

Resource Allocation and Bandwidth Waste

Every request, legitimate or fake, consumes server resources and bandwidth. A high volume of spoofed bot traffic represents wasted infrastructure costs. For websites with significant traffic, the cumulative effect of hundreds or thousands of fake bot requests can be substantial, leading to unnecessary expenditures and potentially impacting legitimate user experience if server loads are mismanaged.

Strategic Visibility in the AI Era

The distinction between retrieval and training crawlers is strategically vital. While retrieval bots offer immediate visibility, training bots contribute to a site’s long-term influence on AI models. Understanding which models are genuinely training on a site’s content provides invaluable insights for future content strategy and IP protection. Google’s opaque approach to Gemini, however, creates a strategic blind spot, making it challenging for publishers to assess their long-term impact on the most dominant AI platform. This mirrors the frustration of the "not provided" era, where critical keyword data vanished, leaving SEOs to guess at user intent. In the AI era, the equivalent is "not provided" AI influence.

The Future of Content Attribution and Value

Forrester’s CitationIQ.com aims to address the "gap between being fetched and being used." This gap is central to the future of content creation and monetization in the AI age. If AI models are trained on content without clear attribution or compensation, understanding who is fetching content and how that content is eventually used becomes paramount. The widespread spoofing undermines this effort by muddying the waters of initial interaction. Without accurate data on legitimate fetches, it becomes even harder to advocate for fair attribution or compensation for content creators.

Call to Action for Webmasters and SEOs

The most critical implication is the urgent call to action for every website owner and SEO professional: verify your logs. Forrester’s method, a simple Python script checking IP ranges against published lists, is accessible and effective. He urges others to "Pull a date range, match the names, verify the IPs against the published lists, and find your real fraction. Then look at your Googlebot line and brace yourself."

For "unverifiable" entries, the advice is to embrace the detective work: "Pull the IPs, check the owner, query the corpus, and chase it until the picture resolves." This proactive approach is no longer optional; it is essential for maintaining data integrity, bolstering security, and making informed strategic decisions in the rapidly evolving landscape of AI and web interaction.

Conclusion: A New Era Demands New Vigilance

Duane Forrester’s experiment with CitationIQ.com serves as a powerful wake-up call for the entire digital ecosystem. The pervasive nature of bot spoofing, particularly from AI assistants and well-known crawlers like Googlebot, highlights a fundamental vulnerability in how web traffic is reported and understood. This digital identity crisis necessitates a paradigm shift in how webmasters approach their analytics and security protocols.

As AI continues to integrate more deeply into our online lives, the ability to accurately distinguish between legitimate AI interaction and malicious impersonation will become increasingly critical. The unseen costs of this deception—skewed data, wasted resources, security risks, and strategic blind spots—are too significant to ignore. The solution lies in proactive verification, a commitment to data integrity, and a willingness to chase the truth beyond self-reported claims. In an AI-first world, vigilance is not just good practice; it is a prerequisite for survival and strategic success. The era of blind trust in user-agent strings is unequivocally over.