Picture a design review where someone has drawn two sites with a router at each, two links between them and a replicated database, and everyone is quietly pleased because nothing exists only once. Then a digger cuts a duct and both links go at once. A second site duplicates components. What it doesn’t duplicate is whatever both sites depend on, it doesn’t make failure detection any more honest, and with exactly two sites there’s nobody to break a tie. That’s where the exposure sits.
This assumes you already know what VRRP, a witness and replication are, and that you have two sites sharing a layer 2 path and a service that has to survive losing either one. We’ve left out product choice, sizing, disaster recovery runbooks and backup. There are no commands because syntax is vendor-specific. The figures are arithmetic from the standards’ own formulas, so they describe the protocols rather than any product.
Start with what both sites share
Before you draw any failover, list each service’s dependencies in the order a request meets them: the path between the sites, power, name resolution, identity, monitoring and wherever configuration comes from. Mark anything that exists once, or exists twice and converges further along, because a dependency that exists once is a single point of failure with a second site sitting next to it.
Here’s an example (reasoning, not a project). Two inter-site links from different providers both leave the building through the same duct. The contracts say two links and monitoring shows two healthy interfaces, but one excavation takes out both, and no protocol knows about the duct. Get route diversity in writing, or treat the pair as one link.
Failover takes longer to notice than you’d think
Start with first-hop failover. RFC 9568 (April 2024) specifies VRRP version 3 with an Active Router and Backup Routers, and the RFC 5798 it obsoletes called the Active Router the Master. In plain terms, a Backup waits for roughly three missed advertisements before it decides the Active Router has gone, and the RFC pins that down with two formulas:
Skew_Time = ((256 - Priority) * Active_Adver_Interval) / 256
Active_Down_Interval = (3 * Active_Adver_Interval) + Skew_Time
Priority here is the Backup’s own (default 100) and the interval is the one the Active Router advertises (default 100 centiseconds, so one second). The table applies the formulas to a few inputs as exact fractional results in milliseconds; a platform that works in whole centiseconds will differ by up to a centisecond.
| Advertisement interval | Backup priority 100 | Backup priority 200 | Backup priority 254 |
|---|---|---|---|
| 1 s (100 cs) | 3,609.4 ms | 3,218.8 ms | 3,007.8 ms |
| 500 ms | 1,804.7 ms | 1,609.4 ms | 1,503.9 ms |
| 100 ms | 360.9 ms | 321.9 ms | 300.8 ms |
| 10 ms (1 cs) | 36.1 ms | 32.2 ms | 30.1 ms |
The RFC’s design figures, under four seconds at the default and under 1/25 of a second when tuned, match the first and last rows.
Among Backups, priority sets the order of promotion, but the gaps are tiny. The difference between two Backups is (difference * interval) / 256, so at 100 ms a priority gap of 10 is only 3.9 ms, and if the one-way delay between your sites is longer than that, the lower-priority Backup can promote itself before it hears the higher one. A graceful stop is quicker than a failure, too. An Active Router that stops participating advertises priority 0, so Backups only wait Skew_Time, whereas a site that loses power sends nothing and the full interval applies (sections 5.2.4 and 6.4.2).
Then comes the awkward part for two sites. Advertisements are link-local multicast with a TTL or Hop Limit of 255, which routers must not forward (section 5.1), so the sites need a shared layer 2 segment, which can be stretched or tunnelled. If that segment is cut while both sites are alive, each Backup times out and both routers go Active, which section 9 (Security Considerations) notes can cause as much disruption as no routers at all. When advertisements cross again, the higher priority stays Active (on a tie, the higher primary address). And because Preempt_Mode defaults to True, a preferred router that comes back takes the role from one that had already recovered the service, which is a second interruption.
VRRP also has no way for priority to follow the health of anything behind the router. The only change the specification describes is the 0 an Active Router sends when it stops participating, so as far as the protocol goes, an Active Router with a dead uplink stays Active. If failover needs to follow an uplink, find out how your platform tracks it, and test that it does.
BFD watches a path, not a service
RFC 5880 (June 2010) describes it as a Hello protocol that detects failure in the bidirectional path between adjacent forwarding engines. Each receiving system works out its own Detection Time (section 6.8.4). In Asynchronous mode that’s the remote Detect Mult multiplied by the remote system’s agreed transmit interval, which is the greater of the local Required Min RX Interval and the last received Desired Min TX Interval. RFC 8562 (April 2019) keeps that rule for point-to-point sessions.
As an illustration with arithmetic only, suppose router A uses Desired Min TX 100 ms, Required Min RX 100 ms and Detect Mult 3, and router B uses 300 ms, 300 ms and Detect Mult 5. A detects the loss of B after max(100, 300) * 5 = 1,500 ms, while B detects the loss of A after max(300, 100) * 3 = 900 ms. Senders also shorten each interval by a random 0 to 25 per cent (section 6.8.7).
BFD and VRRP are independent mechanisms with different failure meanings. VRRP asks whether the Active Router still advertises on a LAN, and BFD asks whether hellos are crossing one path between two forwarding engines. RFC 5880 leaves application-specific uses to separate documents and RFC 9568 doesn’t mention BFD, so a BFD session going down tells VRRP nothing unless a platform couples the two, which its documentation has to say and a test has to confirm.
A hello protocol only sees what hellos experience. Take hellos exactly 100 ms apart (36,000 an hour), a Detect Mult of 3 and independent loss with probability p per hello. The expected number of runs of three or more consecutive losses is about (1 - p) * p^3 * 36,000 per direction, and a two-way path roughly doubles the rate. That’s roughly 0.3 an hour at 2 per cent loss and 32 an hour at 10 per cent, and jitter raises both by about 14 per cent. A degraded path tends either to go unnoticed or to make the session flap, so rule out the path, and load on the devices themselves, before you lengthen the timers.
A passing health check doesn’t mean the service works
The gray failure paper (Huang and others, HotOS, May 2017) has a name for this: gray failures (the paper’s spelling), which it describes as failures “whose manifestations are fairly subtle and thus defy quick and definitive detection”. A system has one when at least one application sees it as unhealthy while the observer responsible for detecting problems sees it as healthy, which the authors call differential observability.
Every mechanism so far watches a proxy for the service. The table shows what each would see in a few failures; the VRRP and BFD columns follow from the formulas above.
| Failure | VRRP advertisements | BFD session | Majority quorum |
|---|---|---|---|
| One of two links down | Silent only if advertisements used that link | That session goes down after its Detection Time | No effect while the other link carries votes |
| Both links on one shared route | The Backup times out and becomes Active: two Active Routers | Both go down together, hinting at a shared cause | Partition: only a side holding a majority continues |
| Gray failure (service broken, path fine) | Advertisements keep arriving | Hellos keep arriving | Votes keep arriving |
The paper (section 4.2) suggests probing that approximates what applications observe, and warns that probing can burden a degraded system. We’d go further: make the health check cross the path the application uses and perform the operation it performs, then ask of each detector which failure would leave it green.
Two sites can’t settle an argument
The extended Raft paper (May 2014) sets out the majority rule for a replicated log. A cluster stays available while a majority of its servers are operational and can talk to each other, so five tolerate two failures. A majority also means at most one candidate can win an election in a term, and an entry counts as committed once it’s on a majority. A majority is floor(n / 2) + 1 of n voters, which assumes one vote per voter (some products weight votes differently).
| Voters | Majority | Failures tolerated | Layout across locations | After losing one location |
|---|---|---|---|---|
| 2 | 2 | 0 | 1 + 1 | Stops |
| 3 | 2 | 1 | 1 + 1 + 1 | Continues |
| 4 | 3 | 1 | 2 + 2 | Stops |
| 5 | 3 | 2 | 2 + 2 + 1 | Continues |
Under majority voting, two sites can’t survive the loss of either one. With one voter each, losing a site leaves one of two, and with two each it leaves two of four. Piling extra voters into one site only moves the problem, since losing that site takes them all. A third location with an independent vote does help, though a witness reachable only over the inter-site route isn’t independent. The partition cases show which side keeps its majority.
| Case | 1 + 1 | 1 + 1 + witness |
|---|---|---|
| Site A lost | B stops (1 of 2) | B and witness continue (2 of 3) |
| Witness lost | n/a | A and B continue (2 of 3), no margin |
| A and B cut apart, witness reaches B only | Both stop | B and witness continue, A stops |
In the Raft model every voter is a full replica that holds the data. Plenty of platforms offer something lighter, a witness or arbiter that votes or breaks a tie without holding a full copy, and what it can do (whether it keeps any state, whether it counts towards committing a write) varies. The tables above count votes, not replicas, so check your platform’s documentation for what its witness does and whether writes can continue on the surviving side.
Stopping on the minority side keeps the data consistent at the cost of availability there, which is usually the right trade for a writable data service. But a majority vote isn’t the only way to avoid split-brain, so automatic failover doesn’t always need a third location. As general engineering background, and not something drawn from the sources above, some designs promote a standby automatically only after they’ve reliably fenced the old primary, meaning powered it off or cut it off from shared storage, and others hand out ownership of the storage through a lease that has to expire or be broken. What’s available depends on the platform, so confirm it in the vendor’s documentation.
The catch is the shared-dependency question again. A fencing action that has to travel over the link that has failed, or whose success you can’t confirm because the site holding the device has gone dark, can’t be relied on, and an unfenced primary is exactly the split-brain you were trying to avoid. If you have no third location and no fencing you trust, we’d choose a designated primary and manual promotion, once the old primary is confirmed down.
What replication costs in latency
The Raft paper says the normal case needs a single round of messages to a majority. It follows that a write acknowledged after majority replication can’t finish faster than the round trip to the voters that complete the majority, with the leader counting as one. That holds when those voters keep a copy of the data. If the third location is a witness that only votes and holds no data, it can’t acknowledge a write, so the commit path then runs between the data-holding sites.
As an illustration with invented round-trip times, say A to B is 8 ms, A to the third location 20 ms and B to the third location 24 ms. If all three are full replicas, a leader at A needs one more acknowledgement and gets it from B in 8 ms, while a leader at the third location has to wait for A at 20 ms. With a vote-only witness, writes are acknowledged by A and B alone, so the floor is 8 ms again. Disk writes and queueing come on top.
As general engineering background, synchronous replication waits for the remote acknowledgement, which costs the round trip and narrows the loss window for acknowledged writes, while asynchronous replication acknowledges locally and loses whatever hadn’t yet reached the other site. Replication generally copies deletions and corruption faithfully, so on its own it isn’t backup. Our backup strategy post covers that side of things.
Test the failure you designed for
Write the expected figures down first, from the formulas above and your platform’s documented settings, so that a pass means “within what we expected” and not merely “it recovered”.
| Failure injected | Measure | Pass means |
|---|---|---|
| Cut one inter-site link | Time to detect on each mechanism | Inside the figure derived in advance; no impact |
| Cut every link at the shared point | Which side keeps the quorum; any second Active Router | In a quorum design, at most one side accepts writes (neither if no side holds a majority) and the other stops; no write acknowledged under the agreed replication mode is lost |
| Remove power from one site | Time to detect; time to recover; writes missing | Recovery and loss inside the agreed target and window |
| Degrade a link with loss or delay | Whether any detector fires; application latency | Something reports it before users do |
| Restore everything and fail back | Interruption on return; resynchronisation time | No more interruption than agreed, or preemption disabled deliberately |
Run the plan in a lab or an agreed maintenance window, and tell everyone affected first. If you can only afford one test, we’d make it the shared-route cut, because that’s the failure the design most likely gets wrong. Routine network maintenance checks are worth doing but won’t replace it, and someone other than the designers should run or review the tests. Looking at single points of failure and agreeing proportionate resilience is part of the work our network infrastructure page describes.
If you’ve got a two-site design on paper, or a failover nobody has ever tested, we’re glad to go through it with the engineers who run it, so get in touch if that would help.
