Google May Ignore Robots.txt Rules Due to This File Error

A Disallow Rule That Wasn't Disallowing Anything

A Shopify store was getting hit with search-box spam: bad actors typing junk queries with embedded links into the site search, which Shopify dutifully turned into indexable URLs. The site's SEO added a disallow for /search in robots.txt, the standard move. Google indexed the spam pages anyway, according to a Reddit thread first reported by Search Engine Journal, and John Mueller stepped in to find out why.

Mueller pulled the client's actual robots.txt file and found the problem himself: a formatting error involving how the file's user-agent sections were structured was causing Googlebot to disregard the disallow rule entirely. Search Engine Journal's account of Mueller's reply cuts off before spelling out the exact syntax, but the diagnosis stands: the rule existed on paper and did nothing in practice, per Mueller and Search Engine Journal.

Why Marketers Should Care This Week

A disallow rule you've never questioned might be doing nothing, and you'd have no way of knowing without checking. Search-box spam is common enough on ecommerce platforms that plenty of teams have added a /search disallow and assumed the job was done. If your robots.txt has multiple user-agent blocks, that's exactly the setup Mueller flagged as fragile.

This is a smaller-scale version of a bigger truth about crawlers and file structure that shows up in how sites get exposed to unwanted scraping: the rules you write and the rules a bot honors are two different things, and the gap only surfaces when someone checks Search Console or pulls the raw file.

Pull your robots.txt this week and check indexed URL counts in Search Console against what you think is blocked. It's the same discipline covered in a proper martech stack audit for budget season: don't trust a setting because it's been sitting there since launch.