LLM Bot Tracker – AI Crawler Detection & Analytics

Popis

Is ChatGPT, Claude, Perplexity, or Gemini crawling your website? This plugin automatically tracks every AI bot visit to give you the data you need! 🤖

LLM Bot Tracker is the most comprehensive AI crawler analytics plugin for WordPress, providing automatic 24/7 monitoring of artificial intelligence bots, search crawlers, and web scrapers accessing your content.

🎯 How It Works

Install Activate Done!

That’s it. The plugin immediately starts tracking all AI bot activity in the background. No configuration needed. View your data anytime in the admin dashboard at Tools > LLM Crawler Logs.

🚀 Why Track AI Bots & Crawlers?

The web is changing. AI-powered search engines like Perplexity, You.com, and SearchGPT are revolutionizing how people find information. Traditional SEO isn’t enough anymore – you need LLMO (Large Language Model Optimization).

This plugin automatically captures:
* Every AI bot visit – Timestamp, bot name, pages accessed, IP address
* AI crawler patterns – Which content AI systems value most
* Training data usage – When GPT, Claude, or Gemini scrape your content
* Crawl frequency – How often each AI system visits
* Geographic origins – Where AI bot traffic originates
* Response codes – Success rates and errors for AI crawlers

Without this data, you’re flying blind in the AI era of search.

✨ Core Features

🤖 Automatic Detection of 59 AI Bots Including:

  • OpenAI Family: GPTBot, ChatGPT-User, OAI-SearchBot (SearchGPT)
  • Anthropic Family: ClaudeBot, Claude-Web, Claude-SearchBot
  • Perplexity: PerplexityBot, Perplexity-User
  • Google AI: Google-Extended, GoogleOther, Google-CloudVertexBot, GoogleAgent-Mariner, Gemini-Deep-Research
  • Amazon AI: Amazonbot, NovaAct (Nova AI Agent)
  • Meta AI: FacebookBot, Meta-ExternalAgent, Meta-ExternalFetcher
  • Microsoft: Bingbot (AI-enhanced), MSN Bot
  • AI Assistants: Devin (Software Engineering AI), LinerBot (Academic Research), QualifiedBot
  • Common Crawl: CCBot (feeds multiple AI systems)
  • ByteDance: Bytespider (TikTok AI)
  • AI Search & Research: You.com (YouBot), Timpi (Timpibot), Allen Institute (AI2Bot), Cohere (Cohere-AI)
  • Data Extraction: Diffbot (structured data extraction for ML)
  • Others: MistralAI, Neeva bot, SemrushBot with AI

📊 Powerful Analytics Dashboard (Tools > LLM Crawler Logs)
* Complete log of all AI bot activity
* Advanced filtering by bot type, date range, URL path, IP address
* Configuration Tab – Smart cache detection with setup guides and bot patterns
* NEW: CSV export – download your data for offline analysis
* 30-day trend analysis – spot patterns and changes
* Top bots leaderboard – see your most frequent AI visitors
* Path analysis – discover your most AI-visited content
* Geographic IP tracking with location data
* Bot verification via IP lookup
* Response code tracking (200, 404, etc.)
* Bulk actions for log management
* Export filtered or complete datasets

⚡ Performance & Privacy
* Runs silently in background – zero configuration
* Lightweight – adds <0.01s to page load
* Smart data storage – automatic log rotation
* GDPR compliant – no human tracking
* No external API calls
* 100% local data storage

🎨 Optional Display Shortcodes
Want to display AI bot stats on your site? Use these optional shortcodes:
* [wpcs_llm_stats] – Statistics table
* [wpcs_llm_bar] – Animated bar chart
* [wpcs_llm_last100] – Recent activity feed
* [wpcs_llm_ip_list] – Compact IP list
* [wpcs_crawler_stats] – Combined view

📈 Who Needs This Plugin?

  • Content Creators – Know which AI systems value your content
  • SEO Professionals – Master LLMO and AI SEO with real data
  • News Publishers – Track AI interest in your stories
  • E-commerce Sites – Detect AI shopping assistant crawlers
  • Tech Companies – Monitor competitive AI intelligence gathering
  • Documentation Sites – See which docs AI systems reference
  • Educational Platforms – Understand AI training on your content
  • Any WordPress Site – Future-proof your content strategy

💡 Real User Success Stories

„Discovered ChatGPT was hitting our API docs 2,000+ times daily. We optimized those pages and saw a 40% increase in AI-driven traffic!“ – Tech Startup

„The background tracking revealed Perplexity visits our site every 6 hours. This data completely changed our content strategy.“ – Digital Publisher

„Found ByteDance scraping our entire product catalog at 3 AM daily. Now we can make informed decisions about blocking.“ – E-commerce Owner

🔍 Actionable Insights You’ll Get

Automatic tracking reveals:
* Which AI companies are most interested in your content
* Your most valuable pages according to AI systems
* Optimal posting times for AI crawler discovery
* Suspicious scraping patterns to investigate
* Content gaps AI systems are looking for
* Competitive intelligence on AI training data

Make informed decisions about:
* Which bots to allow or block in robots.txt
* Content optimization for AI-powered search
* Publishing schedules for maximum AI visibility
* Technical SEO for AI crawlers
* LLMO strategy development

🛠️ Advanced Administration

The plugin works automatically, but power users can:
* Filter logs by any parameter
* Analyze patterns across time periods
* Identify bot behavior anomalies
* Track crawler response codes
* Monitor crawl budget usage
* Set up custom tracking rules
* Export data for external analysis

📚 Resources & Documentation

🏆 Why Choose LLM Bot Tracker?

Automatic & Effortless – Install and forget, it just works
Most Comprehensive – Tracks more AI bots than any other plugin
Lightweight – Won’t slow your site like analytics plugins
Privacy-First – No cookies, no human tracking, GDPR ready
Actively Maintained – New bot detection added monthly
100% Free Forever – No premium upsells or feature gates
Professional Support – Built by Hueston, trusted WordPress experts since 2019

Join thousands of sites gaining AI visibility insights. Install LLM Bot Tracker today and understand your AI traffic immediately!

Additional Information

System Requirements

  • WordPress 6.5 or higher
  • PHP 7.4 or higher
  • MySQL 5.6 or higher
  • 10MB free disk space

Privacy Policy

This plugin only tracks AI bots and crawlers, never human visitors. All data is stored locally in your WordPress database. No external API calls are made. No personal data is collected, transmitted, or sold.

Credits

  • Built with ❤️ by the Hueston LLMO Team
  • Special thanks to the WordPress community for feedback and testing.

Legal Notice

This plugin is not affiliated with OpenAI, Anthropic, Perplexity, Google, Meta, or any AI company mentioned. All bot names and trademarks are property of their respective owners. This plugin simply detects and reports on publicly identifiable user agents.

Snímky obrazovky

  • Admin Dashboard Overview – Complete AI bot activity logs with filtering
  • 30-Day Trend Analysis – Visual charts showing AI crawler patterns over time

Instalace

Automatic Installation (Recommended)

  1. Go to Plugins > Add New in your WordPress admin
  2. Search for „LLM Bot Tracker“ or „AI crawler detector“
  3. Click Install Now then Activate
  4. That’s it! Tracking starts automatically
  5. View data at Tools > LLM Crawler Logs

Manual Installation

  1. Download the plugin ZIP file
  2. Go to Plugins > Add New > Upload Plugin
  3. Choose the ZIP file and click Install Now
  4. Activate the plugin
  5. Tracking begins immediately

After Activation

  • The plugin is already working – no setup required
  • Check Tools > LLM Crawler Logs to see collected data
  • Optionally add display shortcodes to pages if desired
  • Review our documentation for advanced features

Nejčastější dotazy

Does it require any configuration?

No! The plugin starts tracking immediately upon activation. It’s completely automatic. Just install, activate, and check your data whenever you want at Tools > LLM Crawler Logs.

Will this slow down my website?

No! LLM Bot Tracker runs efficiently in the background with zero impact on visitor experience. It typically adds less than 0.01 seconds to page processing – completely unnoticeable.

Does this track human visitors?

No. The plugin exclusively tracks identified AI/LLM crawlers and bots. It does not track, store, or process any human visitor data, making it 100% GDPR compliant.

Which AI bots and crawlers can it detect?

The plugin tracks exactly 59 AI/LLM bots:

OpenAI: GPTBot, ChatGPT-User, ChatGPT-Browser, OAI-SearchBot
Anthropic: ClaudeBot, Claude-Web, Claude-SearchBot, Claude-User, Anthropic-AI
Google AI: Google-Extended, Google-CloudVertexBot, GoogleAgent-Mariner, Gemini-Deep-Research, Gemini-AI, Bard-AI, APIs-Google
Perplexity: PerplexityBot, Perplexity-User
Meta: Meta-ExternalAgent, Meta-ExternalFetcher, FacebookExternalHit, FacebookBot, LinkedInBot
Major AI Companies: xAI-Bot, DeepSeekBot, HuggingFace-Bot, Character-AI, Groq-Bot, MistralAI-User, Mistral-Le-Chat
Search Engines: DuckAssistBot, Andibot, YouBot, Timpibot
Enterprise AI: Cohere-AI, Cohere-Command, Together-Bot, Replicate-Bot
International: PanguBot, PetalBot, Bytespider
Data & Infrastructure: Diffbot, AI2Bot, BrightBot, Webzio-Extended, ImagesiftBot, BigSur-AI, FirecrawlAgent, RunPod-Bot
Others: Amazonbot, Applebot-Extended, NovaAct, Devin, LinerBot, QualifiedBot, CCBot, Omgilibot, Omgili, ProRataInc

We continuously update detection patterns as new AI agents are released.

How do I view the tracking data?

Go to Tools > LLM Crawler Logs in your WordPress admin. You’ll see all AI bot activity, with powerful filtering and analysis tools. No external service or account needed.

Can I block unwanted AI bots?

The plugin provides the data you need to make informed blocking decisions. View bot activity in the admin dashboard, identify unwanted bots, then add them to your robots.txt file. Example:
`
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /
`

How accurate is the bot detection?

Very accurate. We use official user agent strings published by AI companies and verify against known IP ranges when possible. The plugin updates its detection patterns automatically.

Where is my data stored?

All data is stored locally in your WordPress database. No external services, no API calls, no third-party dependencies. You have complete control and privacy.

Can I export the tracking data?

Yes! Click the „📥 Export CSV“ button in the admin dashboard to download your AI bot logs as a CSV file. The export respects your current filters, so you can export specific date ranges, bot types, or all data. The CSV includes date/time, bot name, URL path, IP address, response code, and user agent information.

Does it work with caching plugins?

Yes! Bot detection happens at the WordPress core level before any caching. Works perfectly with WP Rocket, W3 Total Cache, WP Super Cache, and all major caching plugins.

How much data does it store?

The plugin intelligently manages data storage with automatic log rotation. Typically uses less than 10MB even with millions of bot hits. Old data is automatically pruned to maintain performance.

How often do you add new bot detection?

We monitor the AI industry constantly and typically add new bot detection within 2 weeks of public announcements. Updates happen automatically – you don’t need to do anything.

Do the display shortcodes slow down my pages?

No. The optional display shortcodes use smart caching and only load CSS when used. They’re completely optional – the core tracking works without them.

Is this better than server log analysis?

Yes! This plugin provides WordPress-specific context that server logs miss: page titles, post types, taxonomies, and more. Plus it’s accessible directly in your WordPress admin without technical knowledge.

What’s the difference between this and Google Analytics?

Google Analytics focuses on human visitors and conversions. LLM Bot Tracker specifically tracks AI bots and crawlers that Google Analytics ignores. They complement each other perfectly.

Do I need technical knowledge to use this?

No! The plugin is designed to be completely automatic. Install it and forget it. The data is presented in an easy-to-understand format in your WordPress admin.

Recenze

Přečtěte si 1 recenzi

Autoři

LLM Bot Tracker – AI Crawler Detection & Analytics je otevřený software. Následující lidé přispěli k vývoji tohoto pluginu.

Spolupracovníci

Přehled změn

1.6.0

Massive AI Bot Coverage Expansion – Now Tracking 59 AI Bots!

New AI Bots Added (27 new):
* Major AI Companies: xAI-Bot (Grok), DeepSeekBot, HuggingFace-Bot, Character-AI, Groq-Bot
* AI Infrastructure: Together-Bot, Replicate-Bot, RunPod-Bot, FirecrawlAgent
* Enterprise AI: Anthropic-AI, Cohere-Command, ChatGPT-Browser
* Search Engines: DuckAssistBot, Andibot
* Big Tech: FacebookExternalHit, FacebookBot, LinkedInBot, APIs-Google, Gemini-AI, Bard-AI
* International: PanguBot (Huawei), PetalBot (Huawei)
* Data Collection: BrightBot, Webzio-Extended, ImagesiftBot, BigSur-AI
* Legacy Support: ChatGPT-Browser, Bard-AI, Mistral-Le-Chat

Coverage Improvements:
* Now tracking 95%+ of all AI bot traffic
* Added support for enterprise AI training crawlers
* Improved detection for Meta/Facebook AI bots
* Added Google’s complete AI bot family
* Support for emerging AI startups and platforms

1.5.2

Major update with expanded bot coverage and new Configuration tab!

New Configuration Tab with UX Improvements:
* Visual Feedback – Beautiful success toast notification when copying patterns
* Warning System – Alerts when high bot activity detected or cache may not be configured
* Improved Layout – Clean table format for all caching plugin instructions
* Caching plugin setup guides for WP Rocket, LiteSpeed, W3 Total Cache, WP Super Cache, and Cloudflare
*

Expanded AI Bot Coverage – Now tracking 32 AI/LLM bots (up from 27):
* You.com (YouBot) – AI search engine crawler
* Timpi (Timpibot) – Decentralized search indexer
* Allen Institute (AI2Bot) – Academic AI research crawler
* Cohere (Cohere-AI) – Major LLM provider crawler
* Diffbot – Structured data extraction for ML pipelines

1.5.1 (2024-12-10)

  • Bug Fix: CSV export now correctly exports data when no filters are applied
  • Fix: Resolved issue with empty CSV exports containing only headers
  • Enhancement: Improved database query handling for export functionality

1.5.0 (2024-12-10)

  • New Feature: CSV export functionality for AI bot logs
  • Enhancement: Export filtered data or complete datasets
  • Enhancement: Shows record count before export
  • Added: Excel-compatible UTF-8 BOM for proper character encoding
  • Added: Export includes date/time, bot name, URL, IP, response code, and user agent
  • Security: Nonce verification for export actions
  • UI: Export button added to admin dashboard with record count

1.4.4 (2024-12-09)

  • Fix: Completely removed non-AI bot detection from core
  • Enhancement: Plugin now ONLY detects and logs AI/LLM traffic
  • Performance: Faster processing with fewer bots to check
  • Cleanup: Removed all traditional crawlers, SEO tools, and social media bots

1.4.3 (2024-12-09)

  • Refined: Focused exclusively on AI/LLM bots – removed non-AI crawlers
  • Enhancement: Improved bot classification for cleaner analytics
  • Documentation: Added complete list of all 27 tracked AI agents in FAQ
  • Optimization: Streamlined bot detection for better performance

1.4.2 (2024-12-09)

  • New: Added support for 6 new AI agents and assistants
  • New: GoogleAgent-Mariner – Google’s autonomous AI agent for multi-step tasks
  • New: Gemini-Deep-Research – Google Gemini’s research assistant agent
  • New: NovaAct – Amazon’s Nova AI agent for web interactions
  • New: Devin – Software engineering AI assistant that browses websites
  • New: LinerBot – Academic research AI assistant crawler
  • New: QualifiedBot – B2B conversational marketing AI agent
  • Enhancement: Updated bot count from 25+ to 30+ AI agents tracked
  • Database: No schema changes required (uses existing infrastructure)

1.4.1 (2024-12-09)

  • Performance: Dramatically improved AI Blind Spots tab loading speed (3-5x faster)
  • Performance: Added intelligent caching for AI discovery scores (7-day cache)
  • Performance: Implemented pagination for large datasets (50 items per page)
  • Performance: Removed expensive database queries for internal link checking
  • Enhancement: Added „Refresh Analysis“ button for manual cache updates
  • Enhancement: Improved AI scoring algorithm with better content evaluation
  • Fix: Tab switching no longer causes full page reload delays
  • Fix: Better memory management for sites with thousands of pages
  • Fix: Include images folder in plugin package for logo display

1.4.0 (2024-12-09)

  • New Feature: AI Blind Spots analysis – identify pages not visited by AI bots
  • New: AI Discovery Score – rates each page’s discoverability by AI systems
  • New: Tabbed interface in admin dashboard for better organization
  • Added: Coverage statistics showing percentage of content visited by AI
  • Added: Word count analysis for ignored pages
  • Improved: Admin UI with unified navigation
  • Database: Added page analysis table for caching AI visibility data

1.3.1 (2024-12-15)

  • Security: Enhanced input sanitization and data escaping per WordPress.org standards
  • Security: Added nonce verification to all admin actions
  • Code Quality: Full WordPress coding standards compliance
  • Performance: Optimized database queries for sites with 100k+ bot hits
  • Fix: Improved handling of malformed user agents

1.3.0 (2024-12-01)

  • Major Feature: Complete plugin redesign focused on AI/LLM bot tracking
  • New: Professional admin dashboard at Tools > LLM Crawler Logs
  • New: Advanced filtering and analysis tools
  • New: 30-day trend analysis with visual charts
  • New: Geographic IP tracking with location data
  • New: Smart background tracking system
  • Added: Detection for 15+ new AI-related bots including SearchGPT
  • Added: Claude-SearchBot and OAI-SearchBot detection
  • Added: Optional display shortcodes (5 types)
  • Improved: Database schema optimization for large datasets
  • Improved: Automatic log rotation for performance
  • Improved: Bot detection accuracy with IP verification

1.2.0 (2024-10-15)

  • Added: ClaudeBot (Anthropic) detection support
  • Added: PerplexityBot tracking
  • Added: You.com bot detection
  • Improved: Background tracking efficiency
  • Fixed: UTC timezone handling in logs

1.1.0 (2024-09-01)

  • Added: Google-Extended bot detection for Bard/Gemini
  • Added: Meta AI crawler support (Meta-ExternalAgent)
  • Added: Bytespider (TikTok) detection
  • Fixed: UTF-8 handling in URLs with special characters
  • Improved: Memory usage for high-traffic sites

1.0.0 (2024-07-15)

  • Initial release
  • Automatic crawler tracking
  • Basic admin dashboard
  • GPTBot and CCBot detection