When BGP Says Everything Is Fine, but the Internet Is Down

Two Internet service providers should provide an alternate path when one has a problem. But what happens when a provider continues to announce a valid route while its network can no longer carry the traffic?

That was the question raised by an overnight connectivity incident. The immediate symptoms suggested an Internet outage. The routing control plane, however, looked healthy.

The incident

After-hours reports indicated that users could not reach the Internet. The environment had two independent upstream providers, so a failure affecting one carrier should not ordinarily have resulted in a sustained loss of connectivity.

Both external BGP neighbors remained in the Established state. Each provider continued advertising a default route, and the local routing policy continued preferring the primary upstream. No adjacency loss or default-route withdrawal occurred to trigger the expected fallback.

The preferred provider was experiencing a fault farther into its network. Although the BGP session and route advertisement were intact, traffic was not successfully making it through that upstream path. The other provider remained a viable alternative, but the routing decision had no reason to change based on the signals it was receiving.

What BGP knew—and what it didn't

BGP communicates reachability information and applies policy to select paths. A session in the Established state confirms that two BGP speakers can maintain their control-plane relationship. Receiving a default route means the neighbor is advertising a path for destinations not otherwise known locally.

Neither fact independently proves that the provider can forward packets all the way to arbitrary Internet destinations. An upstream router can continue sending BGP updates even when a downstream link, routing process, or forwarding component is impaired elsewhere in the provider network.

In this case, the preferred default route remained eligible and continued to win route selection. The design was redundant in terms of connectivity, but its failover trigger did not cover the observed failure mode.

Why conventional failure detection wasn't enough

Physical link state and BGP hold timers are useful when a circuit or peering session goes down. BFD can detect certain path failures much faster than normal BGP timers, but when the forwarding problem occurs beyond the directly monitored peer, the session may still appear healthy. Route flap dampening addresses instability caused by repeated route withdrawals and announcements; it does not verify Internet reachability.

Those tools are valuable, but none should be confused with an independent check of whether the preferred upstream is delivering usable service.

The architectural response

At the time of writing, the redesign is planned and not yet deployed. The proposed direction is to move external BGP peering away from the security firewall and onto capable core routing infrastructure, while retaining the firewall's security inspection and NAT responsibilities.

The purpose of that change is not to suggest that next-generation firewalls cannot run BGP. Rather, separating edge-routing decisions from security enforcement creates room for a more tailored health-check and route-control workflow in this environment.

The intended design combines normal BGP peering with independent reachability testing through each upstream. When the preferred provider is still advertising a default route but fails the defined service-health criteria, routing policy or a controlled automation mechanism would make the impaired path less preferred or temporarily ineligible. Recovery checks and hysteresis would help avoid unnecessary switching caused by momentary packet loss.

Implementation considerations

Several details matter more than the device on which BGP happens to run:

• Per-provider probes: Ensure each test is pinned to the intended upstream; a probe that silently follows the surviving default route can produce a false healthy result.
• Multiple independent targets: Avoid declaring an entire ISP failed because a single external host stops answering. Choose suitable targets and thresholds.
• Route selection: Define exactly how probe state changes BGP path eligibility or preference, and how normal policy is restored after recovery.
• Traffic symmetry and firewall state: Account for NAT, inbound return paths, and existing stateful sessions. An outbound route change does not guarantee that established connections survive.
• Fail-safe behavior: Decide what should happen if probes fail through both providers, or the monitoring mechanism itself becomes unavailable.
• Observability and rollback: Log probe state, routing decisions, and changes, and preserve a tested way to revert the migration.

Any example configuration associated with this approach should be treated as a conceptual reference rather than a drop-in production configuration. The exact integration depends on the platform, software release, and desired failover policy.

How the design will be validated

Once implemented, testing should reproduce the original failure condition: keep the preferred provider's BGP session established and its default route advertised while deliberately preventing successful forwarding beyond the provider edge in a controlled test environment.

Validation will focus on whether reachability checks detect the impairment, whether the routing decision shifts to the alternate provider, and whether service recovers as expected. Additional tests should include complete peer loss, probe-target failure, both upstreams failing, and restoration of the preferred path. Detection time, route convergence, and effects on existing connections should be measured rather than assumed.

The takeaway

Redundancy is not simply a count of circuits, routers, or BGP sessions. It is the ability to detect the failures that matter to users and respond predictably.

This incident revealed a gap between a healthy routing control plane and actual Internet service. The planned redesign aims to close that gap with upstream-aware health validation and deliberate routing control. The final implementation and measured results will be documented after deployment.

← All case studies