Meta's Bot Ate a Small Business Website Alive

On October 24, 2026, a San Diego children's book illustrator named Chelsea Paulus got an email from her hosting company, SiteGround, telling her she'd burned through 80% of her monthly CPU budget. By the following Monday, she was watching requests pile up on her security dashboard, several per second, all day, all tracing back to a Meta IP address in Dallas. By mid-week, a second Meta bot showed up and pushed her to 100%. As of November 3, Business Insider reported her website is still struggling, and she had to remove her website link from her Instagram profile to keep from tipping the whole thing over.

Paulus had been running her art business for a decade. She went full-time last year. Her website is her storefront, her portfolio, her booking page. And in 48 hours, Meta's crawler visited it more than 550,000 times, consuming nearly her entire traffic allowance, for free, to train a model she'll never use and never profit from. She put it plainly: "Bots are not customers and do not generate additional income."

What actually happened

According to the Business Insider report by Katherine Li, the crawler identified itself as "Meta External Agent." Meta's own developer documentation describes Meta External Agent as a bot that "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." A second bot, "Meta Web Indexer," followed. Meta's developer guide describes that one as a crawler that scours the web to improve Meta AI search result quality for users.

Paulus edited her robots.txt file and put up a firewall, but robots.txt changes can take 24 hours to propagate, and she needed the bleeding to stop immediately. SiteGround gave her code to drop into a server configuration file that, as she put it, "is not something that anybody who's a normal person running a website would want to touch." That bought her some time. Then the second bot arrived anyway.

Meta's response was essentially: this is your problem to manage. A Meta spokesperson told Business Insider, "website managers can decide how our crawlers interact with their sites through tools like robots.txt. We provide publicly available guidance on ways to manage robots.txt for our crawlers, and we respect the preferences site owners set." Which is technically accurate and also completely misses the point. Paulus had to know to do it, know how to do it, and do it fast enough to matter. Most small business owners would have just watched the site go down and called their nephew.

There are currently no laws that prevent an AI bot from crawling a publicly accessible website. Cloudflare offers bot verification tools that are free right now, and Paulus was exploring those as of the report's publication date. Her question about how long Cloudflare can keep that free if abuse scales up is a fair one, and nobody has answered it.

The storefront tax

Here is the principle I want you to walk away with, and I'm going to call it the Storefront Tax. Every business that put its information on the public internet in the last 30 years did so to attract customers. That was the deal. Now AI companies are sending crawlers to vacuum up that content to train models, and the bandwidth bill goes to you. You don't get credit. You don't get a cut. You get an email from SiteGround saying you're at 80% and counting.

It's a little like if a competitor sent a van to your parking lot every night to photograph your menu board, your prices, and your specials, and then used that to build a kiosk that answered customer questions without them ever walking in your door. Except the van never leaves, it circles the block 550,000 times, and it blows out your parking lot in the process.

The Storefront Tax is already being collected. The only question is whether you've noticed it yet and whether you've opted out.

This isn't totally new either. The first time Google's crawler showed up and hammered a poorly configured small business site, the conversation was almost identical. The difference is that Google built a polite crawler culture over 20 years, backs off when asked, and the traffic it sends back tends to justify the cost. AI training crawlers have no such incentive structure. They take and move on. Your website is already competing with AI-generated answers in search results, and now the same AI infrastructure is also eating your server budget. That's a two-front problem.

What it means for your business

If you are running a small business website, especially on a shared hosting plan, this is not a hypothetical. Shared hosting plans (think SiteGround's entry-level tiers, Bluehost, HostGator) cap CPU usage and bandwidth. A bot that hits your site several times per second will consume those resources just as fast as human visitors, faster actually, because bots don't pause to read anything. They just hammer the server.

If you're in the arts, professional services, home services, healthcare, or any field where your website holds a lot of content, images, or portfolio pages, you are a more attractive crawl target. More pages means more requests. More images means more bandwidth per request.

Paulus had a tech background. She knew what WordFence was, knew how to read an IP lookup, knew what robots.txt even means. Most of the business owners I work with do not have that background, and that's no knock on them. That's just the reality. Which means if this happens to you on a Tuesday afternoon, you are probably calling your web developer or your nephew, and you're losing business hours while you figure it out.

Cloudflare's free tier includes bot protection and human verification tools. It's not perfect, but it's real, and right now it costs nothing. That may not last. If you want a sense of what a well-built site infrastructure looks like for a local business, our Smart Sites platform addresses exactly this kind of foundation-level protection as part of the build. But you don't need us to do the basics this week.

Under the hood

For the technically curious: robots.txt is a plain text file that sits at the root of your website (yoursite.com/robots.txt) and tells crawlers which pages they are and aren't allowed to visit. You can block specific user agents, which is what Meta's bots identify themselves as. Adding a rule to disallow Meta External Agent and Meta Web Indexer is about four lines of text. The catch Paulus ran into is that robots.txt is a convention, not a technical barrier. A well-behaved bot respects it. A bot that ignores it will anyway. The harder enforcement layer is a Web Application Firewall (WAF), which Cloudflare provides, that can block traffic by user agent string at the network level before it ever touches your server. SiteGround's server-side protection was already doing some of this according to their email exchange with Paulus, but it wasn't enough to stop the second bot. Stacking robots.txt plus a WAF rule is the current best practice. Neither is foolproof, but together they are meaningfully better than nothing.

What to do this week

• Check your robots.txt file today. Go to yourwebsite.com/robots.txt in a browser. If you get a blank page or a 404, you don't have one. You need one. Your web developer can add it in 10 minutes, or your hosting provider can walk you through it. Add disallow rules for Meta External Agent and Meta Web Indexer at minimum. Search "robots.txt meta crawler block" for current syntax, since the exact user agent strings matter.
• Put your site behind Cloudflare. The free tier is legitimate protection. It acts as a proxy between the public internet and your server, and it includes bot filtering. If you are on shared hosting and you don't have Cloudflare in front of your site, you are unprotected. This is a one-time setup of maybe an hour, and it addresses the Storefront Tax problem at the network level rather than the file level.
• Check your hosting usage dashboard. SiteGround, Bluehost, and most shared hosts have a resource usage panel in your control panel. If you're regularly above 70% CPU, you are already vulnerable. That's not a catastrophe yet, but it's a warning sign worth acting on before a crawler finds you.
• Talk to your web person this week, not next month. If you have a developer or an agency managing your site, forward them this post and ask specifically: are we blocking AI crawlers, and do we have Cloudflare or a WAF in place? You shouldn't need to know the answers yourself. You just need to know to ask.

Skip it if

You're running a simple one-page site with almost no content, or you're already on a managed WordPress host like WP Engine or Kinsta that handles WAF rules and bot protection at the infrastructure level. Also skip the panic if your site is already behind Cloudflare with bot fight mode turned on. You're probably fine. Check anyway, but you can breathe.

One last thing

Paulus said something I keep coming back to: "Meta needs to have guardrails in place to prevent this from happening. Imagine if the unregulated AI keeps doing this, how long can platforms like Cloudflare continue to offer free services?"

That's the real question under the question. Right now, the Storefront Tax has a free offset. Cloudflare absorbs it. SiteGround patches it. Your nephew fixes it on a Sunday. But none of those buffers scale forever. The AI companies training on public web content are collecting something of value, and the cost is distributed invisibly across millions of small business owners who never agreed to the arrangement. Paulus noticed because she had the technical background to read the logs. Most people won't notice until the site goes dark.

Your website is your storefront. Act like it has a lock on the door.

Sources

• Katherine Li, "An artist says Meta's bots overwhelmed her website and left her scrambling to protect her business," Business Insider, November 3, 2026