How to Monitor a Website Behind Cloudflare or a WAF

Why a blocked monitor looks like a healthy website

When a web application firewall decides a visitor might be a bot, it rarely refuses the request outright. It serves a page instead: a browser check, a CAPTCHA, or an access-denied notice. That page is real HTML. It has a title, it loads without errors, and it usually arrives quickly because there is almost nothing on it.

A monitoring check that only asks whether the server responded will accept that page as a success. Uptime stays green, response time looks excellent, and no alert is raised. The site genuinely is fine — it is just not fine for the monitor, and no one finds out until someone looks at what the monitor actually saw.

User journey monitoring makes the problem more expensive. A journey is a stored claim about what a visitor can do on your site. If a run lands on a challenge page, every button it expects is missing, and a monitor that quietly drops the steps it could not find ends up checking nothing at all. The journey keeps passing, and the day your checkout genuinely breaks, nobody is told.

This is not a rare edge case. It happens after a WAF rule change, after a security vendor tightens defaults, during a traffic spike that triggers rate limiting, or simply because a monitoring provider's IP range picked up a poor reputation score from unrelated traffic.

How a monitor should detect that it has been challenged

The fix is for the monitor to recognise a challenge page as its own outcome, separate from both success and generic failure. Doing that reliably means refusing to trust any single signal, because every individual signal is misleading on its own.

Status codes are not enough. A firewall challenge often returns 403, 429, or 503, but a client-side browser check can return a normal 200 with an empty shell, and plenty of legitimate pages return 403 for reasons unrelated to bots. Page text is not enough either — a search for phrases like "verify you are human" will match any article that happens to discuss CAPTCHAs.

Looking for CAPTCHA elements in the page is the most tempting shortcut and the most damaging one. A great many healthy login, signup, and contact pages embed a CAPTCHA widget as an ordinary form control. Treating that widget as proof of a block means failing checks on pages that are working perfectly. NorthDuty only treats a CAPTCHA widget as evidence of a challenge when the response also carried a blocking status code or the page text is using challenge language.

NorthDuty gathers four independent views of every page before deciding: the response headers, which is where security vendors identify themselves most deliberately; distinctive markup belonging to a specific vendor's challenge; the visible page text; and the addresses of the scripts, frames, and forms the page loaded, which catches vendors that ship no recognisable markup at all. Strong, vendor-specific evidence is trusted on its own. Ambiguous evidence has to be corroborated before it counts.

One more distinction matters more than any of the detection work: whether the block can clear by itself. A managed browser check usually resolves after a few seconds without any human involvement, so NorthDuty waits briefly and re-tests, and abandons the wait early if the challenge escalates into something a browser cannot solve. A CAPTCHA that requires a person will never clear, so the check fails immediately, names the vendor doing the blocking, and tells you that allowlisting is the fix. That last part matters because the remedy always lives in your firewall configuration, not in the monitoring tool.

Conclusion

Bot protection and monitoring want opposite things, and when they collide the failure is silent. A blocked check that reports success is worse than a check that fails loudly, because it quietly replaces real information about your website with a confident, wrong answer.

NorthDuty treats a security challenge as a distinct result: it correlates response headers, page markup, visible text, and loaded resources before deciding it has been blocked, waits only on challenges that can actually clear, reports a journey that hits an interstitial as blocked rather than passed, and names the vendor responsible so you know exactly where to allowlist.