A hosting network is judged on its worst day, not its average one. Ours is built around a simple idea: every hub site has at least three independent ways out to the internet, and as much traffic as possible never touches transit at all because it is exchanged directly at public peering points. This post describes the mix, how we choose paths, and what happens when one carrier has a bad day.
Transit: three upstreams per hub
Frankfurt, New York and Singapore each connect to three transit providers over physically separate paths into the building, on separate routers. The providers are chosen to be diverse in their own backbones: a Tier 1 with a global network, a large European or regional carrier with dense domestic coverage, and a third with a different set of downstream customers. Amsterdam, Helsinki and Los Angeles have two upstreams each, plus a private backbone link to the nearest hub that carries traffic when both local upstreams are degraded.
Three is the number where a single carrier outage, a maintenance on a second and a route leak on a third can all happen in the same week without anyone noticing; it has happened twice.
Peering: keep it local
We are present at DE-CIX Frankfurt, AMS-IX, Equinix IX in New York and Singapore, and the smaller local exchanges in Helsinki and Los Angeles, with an open peering policy: anyone who asks gets a session. Public peering carries roughly 60% of our outbound bytes in Europe and 45% in Asia. Traffic to the large content networks, the eyeball ISPs and the cloud providers goes across the exchange fabric directly, one hop, without passing through a transit provider's backbone.
The effect for you is fewer hops and lower jitter to the networks your users are on. The effect for us is that a transit problem affects a minority of traffic, which is what makes the failover boring.
How we choose a path
BGP picks the shortest AS path by default, which is often not the fastest. We adjust it in three layers:
- Prefer peering over transit with local preference, since a direct session is almost always the better path.
- Prefer the cleanest transit when several offer the same prefix. A small fleet of probes measures loss and latency to a few thousand destination networks through each upstream, every minute. An upstream whose loss to a destination exceeds 0.5% for five minutes is de-preferred for that destination automatically; it comes back when the measurement clears.
- Engineer the few exceptions by hand: a handful of far-away destinations where the shortest path goes through a congested exchange get a static preference toward a longer but emptier route.
Inbound is harder to control, since the rest of the internet decides how to reach us. We announce the same prefixes to every upstream and use prepending and communities only to steer away from a path that a probe shows to be lossy, never to save money on a cheaper carrier.
What a bad day looks like
Last quarter, one of Frankfurt's upstreams suffered a backbone failure that added 40% loss to North American destinations for about an hour. The probes noticed within two minutes and de-preferred the carrier for the affected prefixes; traffic shifted to the other two upstreams, whose capacity is sized so that any one of them can carry the whole site. Customers saw a few minutes of elevated latency to the US and nothing else. The status page recorded a “degraded” entry for Frankfurt transit for the period, because that is what it was, even though the impact was minor.
A full site loss is the case the backbone exists for: if Amsterdam lost both upstreams, its traffic would leave via Frankfurt over the private link. That path has never carried production traffic outside of drills, which we run twice a year.
Capacity, and the rule for it
Every link is upgraded when its 95th-percentile utilisation passes 50% for two consecutive weeks. That means a link is never expected to run above half full in normal operation, which leaves the other half for the moment a neighbour link fails or an attack is being absorbed. Cheap hosting is usually cheap because someone skipped this rule; it is one of the few costs the pricing rule does not touch.
See for yourself
- The looking glass shows BGP paths, ping and traceroute from each site to any address.
- Our AS is registered in PeeringDB with the exchanges and the peering policy; sessions requests via the contact there are answered within a few days.
- The status page lists transit and peering as components per site, with history, so a degraded upstream is visible even when you did not feel it.