Bugproof
Log in Sign up
← All Guides

Troubleshooting website reachability

Sometimes a target website cannot be reached by automated tests. This is often due to strict firewalls that block automated traffic, but can usually be resolved by configuring the right authentication headers. You can improve reliability by adding custom headers at the project level.

Why a website might not be reachable

  • Bot protection / WAF: Some sites block unknown or headless user-agents, challenge traffic, or require additional signals.
  • Required authentication: Endpoints may need an Authorization header, signed cookies, or API keys to serve content.
  • Strict header checks: Servers can expect headers like User-Agent, Accept-Language, or Referer and respond differently when they’re missing.
  • Geo / network rules: Allow-lists, IP reputation, or country restrictions can deny access.

Tip: Before changing headers, double-check your base URL, confirm the site is publicly reachable, and ensure login steps (if required) are set up in your test flow.

Add custom headers to your project

  1. Go to your dashboard and open your project.
  2. Go to Edit project.
  3. Expand Advanced settings → Custom headers.
  4. Click Add header and provide a header key and value for each header you need.
  5. Click Save changes. New test runs will include these headers in browser requests.

Custom headers are sent with browser requests during tests. They won’t fix issues unrelated to request metadata (e.g., geo-blocking or account allow-listing).

Helpful header examples

Common headers needed for firewall and authentication challenges.

Authorization: Bearer YOUR_TOKEN
cf-access-client-id: YOUR_CF_CLIENT_ID 
cf-access-client-secret: YOUR_CF_CLIENT_SECRET
X-Custom-Auth: YOUR_AUTH_TOKEN
X-Forwarded-For: ALLOWED_IP
  • Authorization: Standard bearer token authentication for APIs and protected endpoints.
  • cf-access-client-*: Required headers when accessing sites behind Cloudflare Access.
  • X-Custom-Auth: Custom authentication tokens required by some firewalls.
  • X-Forwarded-For: Can help bypass IP-based restrictions if allowed IPs are configured.