In a recent analysis of 66.7 billion bot requests across over 5 million websites, web hosting company Hostinger uncovered important shifts in how AI crawlers are interacting with the open web. The findings show that while some AI bots are being increasingly blocked by site owners, others, particularly those tied to search and assistant tools, are gaining broader access.
These developments have meaningful implications for brands, content creators, and digital marketers everywhere, including locally for institutions like https:
//dacademy.in/, a leading Digital marketing institution in Calicut, offering comprehensive Digital marketing courses in Calicut and operating as a respected Digital marketing academy in Calicut.
AI Training Bots Are Facing the Cold Shoulder
Hostinger’s study found that AI crawlers designed to collect data for training large language models are increasingly being blocked by websites.
Key Highlights:
OpenAI’s GPTBot, a crawler used for model training, saw its coverage drop dramatically — from 84% of sites to just 12% over the study period.
Meta’s ExternalAgent also saw strong declines.
Many site owners are rejecting these crawlers via site rules like robots.txt and related bot exclusion standards.
Why this matters:
Training bots often collect data to improve or build large AI systems. As more sites decline access, publicly available training data shrinks — pushing AI developers to seek licensed content or other data sources.
AI Search and Assistant Crawlers Are on the Rise
In contrast to training bots, AI tools designed to power search and conversational experiences are gaining traction.
Hostinger’s analysis shows that OpenAI’s search bot, used by services like ChatGPT when it “crawls” pages for up-to-date answers or browsing, now has around 55% coverage. This means more sites aren’t blocking it, and some are even welcoming it, similar to how classic search engines (like Google) are handled.
This trend is significant for organizations like dacademy.in, because:
It increases the chances of content being discovered via AI search tools
It expands visibility for relevant keywords, such as Digital marketing institute in Calicut, Digital marketing academy in Calicut, and Digital marketing course in Calicut
It creates a new frontier beyond traditional SEO, where content quality and accessibility for assistant crawlers matter
Traditional Search Bots Remain Steady
While AI bots show volatility in access, traditional search engine crawlers (like Googlebot and Bingbot) remained relatively stable over the same period. This suggests that many site owners still treat classic SEO as foundational, neither blocking nor significantly altering how they interact with these trusted bots.
What This Means for Digital Marketing Strategy
For digital marketers and institutions, especially those promoting localized services like dacademy.in. This shift underscores several strategic takeaways:
1. Optimize for Both Search and AI Discovery
Content that ranks for keywords like:
Digital marketing institution in Calicut
Digital marketing academy in Calicut
Digital marketing course in Calicut
will benefit not only from traditional SEO but also from visibility in AI-powered search and assistant results.
2. Review Crawl Permissions Carefully
Decide whether your site should:
Allow AI search crawlers (to increase visibility)
Restrict training crawlers (to protect proprietary content)
This can be controlled via your site’s robots.txt and the emerging AI.txt standards.
3. Content Quality Matters More Than Ever
AI systems prioritize high-quality, structured content. Detailed pages about course offerings, instructor credentials, success stories, and clear keyword signals help both search and AI crawlers understand and surface your site.
The Hostinger study reveals a clear split: AI training bots are being blocked at growing rates, while AI search and assistant crawlers are expanding their reach. For local digital marketing education providers — such as those operating at https://dacademy.in/ — this presents an opportunity to enhance both traditional SEO and AI discovery.
Comments
Post a Comment