More than 550,000 requests, arriving several times a second, from bots identifying themselves as Meta crawlers, landed on one San Diego children’s author’s website over the past week and a half — enough to push the shared hosting plan behind her four websites from an 80% CPU warning on October 24 to 100% usage within days. At the low end of “several times a second” — saythree requests — a single bot generates about 260,000 requests per day, so the reported total is roughly two days of machine-paced load, bearing down on a plan sized for a small business.
The site belongs to Chelsea Paulus, a children’s author and illustrator who has run the business for a decade and went full-time as an artist last year, as Katherine Li reported in Business Insider. Paulus previously worked in tech and web design, which is why she noticed quickly. “This has never happened before,” she said.
On October 24 her host, Bulgaria-headquartered SiteGround, emailed to warn she had consumed 80% of her allocated CPU. The following Monday she opened WordFence, a WordPress security plugin, and watched request lines appear every second; the IP address resolved to a Meta location in Dallas, and the visitor identified itself as “Meta External Agent” — a fact SiteGround’s operators later confirmed. Meta’s developers page defines that agent as a crawler that “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
Two crawlers, one shared CPU pool
The flow looked like this: Meta crawler (Dallas IP) → several requests per second → Paulus’s WordPress site → one shared SiteGround CPU pool serving four websites → 80% CPU on Oct 24 → mitigation → second crawler arrives → 100% CPU by mid-week.
Paulus edited her robots.txt file — the standard file that tells crawlers which parts of a site are off-limits — and put up a firewall, but both can take 24 hours to take effect. SiteGround gave her code to paste into a hidden configuration file, which she described as “not something that anybody who’s a normal person running a website would want to touch.” That stopped the first bot temporarily. By mid-week a second crawler, “Meta Web Indexer,” appeared; Meta’s developer guide defines it as a crawler that “navigates the web to improve Meta AI search result quality for users.” It pushed her resource usage to 100%.
Meta’s answer
A Meta spokesperson told Business Insider that website managers control bot activity and pointed to its crawler documentation. “Like many companies, Meta uses web crawlers for a number of different purposes, and website managers can decide how our crawlers interact with their sites through tools like robots.txt,” the spokesperson said. “We provide publicly available guidance on ways to manage robots.txt for our crawlers, and we respect the preferences site owners set.” The reporting complicates the last clause: Paulus did set a robots.txt preference, plus a firewall, and the load continued until her host intervened at the server level — and a differently-named crawler then arrived anyway. SiteGround, for its part, told Paulus in an email exchange she screenshotted that its “server-side protection” is already limiting the attempts; the host did not respond to Business Insider’s request for comment.
The economics run one way. Bots are not customers and generate no income, so Paulus sees no case for paying SiteGround for a larger traffic allowance. As of Monday her site still struggled, and she removed the website link from her Instagram profile — a channel that drives real buyers — because the additional human clicks could take the site offline entirely. There are no laws preventing an AI bot from crawling a publicly accessible website. She is exploring Cloudflare’s human-verification tools, which are free to use at the moment.
“Meta needs to have guardrails in place to prevent this from happening,” Paulus said. “Imagine if the unregulated AI keeps doing this, how long can platforms like Cloudflare continue to offer free services?” Her other prediction is a cost curve: “If this keeps happening, I would assume that small businesses are going to fork out more, either for higher usage or for programs to better protect themselves.” The number to watch is the free tier: bot-blocking that costs nothing today is the only protection a site earning nothing from crawlers can absorb.