robots.txt Monitoring: Catch a Leftover Disallow Rule

What Happens When "Disallow: /" Reaches Production

On staging, the standard practice is to keep search engines out with a robots.txt containing User-agent: * and Disallow: /. The trouble starts when that same file ships to production.

The site renders normally. Uptime checks keep returning 200. Neither the client nor the agency notices anything. Meanwhile Googlebot obeys the file, stops crawling, and over days or weeks the index empties out. The problem surfaces when the client says "we don't show up in search anymore", and by then recovering rankings takes longer than the original launch did.

What makes this failure so awkward is that from an uptime perspective nothing is broken. robots.txt is a static text file served with 200 OK, and the site itself is healthy. Detecting it requires a separate monitor that asks whether the content of robots.txt is what you intended.

Three Things the robots.txt Monitor Checks

Miterl's robots.txt monitor (type: robots_txt) periodically fetches /robots.txt at the origin of the monitored URL and verifies the following.

Two details are worth knowing.

• The URL is normalized to the origin -- if you register https://example.com/about/, the monitor fetches https://example.com/robots.txt. robots.txt is only honoured at the origin root, so path and query are discarded
• Groups are parsed per RFC 9309 -- consecutive User-agent: lines form one group, the first rule line closes it, and Sitemap: is a non-group directive that is picked up wherever it appears

Note that this monitor evaluates rules for User-agent: *. A rule that blocks only a specific crawler (say, Googlebot alone) is out of scope. To check crawler-specific rules as part of a launch, use the pre-launch audit described below.

Why enable the Sitemap check

Compared to catching Disallow: /, a missing Sitemap: line sounds minor, but it happens constantly in agency work: a CMS migration, a hand-edited robots.txt, a template swap. Losing the line removes one of the ways search engines discover your sitemap, which delays crawling of newly published pages. Turning on robots_check_sitemap is cheap insurance.

Checking by Hand First

Before setting up the monitor, confirm the current state of the live site.

# Fetch robots.txt and print the status code
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt

# Is there a site-wide block right after User-agent: * ?
curl -s https://example.com/robots.txt | grep -A3 -i '^User-agent: \*'

# Is a Sitemap directive present?
curl -s https://example.com/robots.txt | grep -i '^Sitemap:'

A robots.txt carried over from staging typically looks like this. If it goes live, the second command above prints Disallow: /.

User-agent: *
Disallow: /

The shape you want in production looks more like this.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Creating the Monitor via the API

The robots.txt monitor is a regular Monitor resource. There is no extra plan restriction; just set type to robots_txt.

# robots.txt monitor (type: robots_txt) -- check for a site-wide block and a Sitemap line every 10 minutes
curl -X POST https://miterl.com/api/v1/monitors \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "example.com robots.txt",
    "type": "robots_txt",
    "url": "https://example.com/",
    "interval_seconds": 600,
    "robots_check_disallow_all": true,
    "robots_check_sitemap": true
  }'

To roll the monitor out across every site an agency manages, loop over a list. robots.txt changes rarely, so a 10-minute interval is plenty.

# sites.txt: one URL per line
while read -r url; do
  host=$(echo "$url" | sed -E 's#https?://([^/]+).*#\1#')
  curl -s -X POST https://miterl.com/api/v1/monitors \
    -H "Authorization: Bearer $MITERL_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"name\":\"$host robots.txt\",\"type\":\"robots_txt\",\"url\":\"$url\",\"interval_seconds\":600,\"robots_check_disallow_all\":true,\"robots_check_sitemap\":true}"
done < sites.txt

Pair the Pre-Launch Audit With Continuous Monitoring

robots.txt accidents come in two flavours: the ones that happen at launch and the ones that happen during later work. Assigning a different mechanism to each closes the gaps.

Because the pre-launch audit is an API call, adding it as the final step of a deploy pipeline removes the "launch with robots.txt blocking" path entirely. "Pre-Launch Test Automation" shows how to wire it in, and the "Pre-Launch Monitoring Checklist" covers everything else to verify before going live.

On the continuous side, the DOM integrity monitor is the robots.txt monitor's natural partner: it watches for an injected or leftover noindex meta tag, the other main route out of the search index. "WordPress Tamper Detection" explains that monitor in detail. Watch both robots.txt and noindex, and nearly every way a site silently drops out of search is covered.

Responding to a Detection

The response when the robots.txt monitor goes Down is straightforward, but having the order written down speeds up recovery.

1. Restore the correct robots.txt -- redeploy the production file if it lives in Git, or rewrite it by hand to the production shape shown above
2. Ask Search Console to recrawl -- confirm in the Search Console robots.txt report that the updated file was fetched, then request indexing for the key pages via URL inspection
3. Record how long the block lasted -- derive the window from the monitor's detection and recovery times and include it in the client report. The "Incident Report Template" shows how to write it up

If the block lasted only hours to a day, the impact on the index is negligible. The real value of a 10-minute robots.txt monitor is keeping every incident inside that "short enough not to matter" window.

Summary

• Carrying a staging Disallow: / into production is invisible to uptime monitoring because the site keeps returning 200
• The robots.txt monitor checks three things: the file exists, User-agent: * is not blocked site-wide, and a Sitemap: line is present
• The monitored URL is normalized to the origin, and groups are parsed per RFC 9309
• Use the pre-launch audit for launch day and the robots.txt monitor for everything after, so nothing slips through
• After a detection: restore, request a recrawl, and record the duration

Monitor type details are in the documentation. The robots.txt monitor is available on the free plan, so start with the sites you launched most recently.