avoiding 255 / 31-bit prefixes

At OSSO, we've been using a spine-leaf architecture in the datacenter, using BGP and Layer 3 to the host. This means that we can have any IP address of ours just pop up anywhere in our network, simply by adding a prefix on a leaf switch. We sacrifice half of our IP space for this. But we gain simplicity by avoiding all Layer 2 tricks.

TL;DR: Avoid IP addresses ending in .255 for endpoints.

31-bit prefixes on Point-to-Point Links

Do we want IP 1.2.3.5 to show up somewhere? We'll just enable the 1.2.3.4/31 prefix on the relevant port on leafX. BGP handles that both IPs are now reachable on the leaf switch and 1.2.3.5 can be reached at the selected switch port.

This makes deploying and moving services a breeze. And we don't have to deal with Layer 2 hacks. But as you can see, we have to sacrifice one IP (the switch end) per prefix.

Normally, networks have to contain a network address (x.x.x.0 for a class C network) and a broadcast address (x.x.x.255). Thus, the smallest network would be 2 bits: e.g. 1.2.3.0/30:

The network address is obsolete and never used. And the broadcast address equates to “the other host”, so it is not needed for networks with only two parties. Therefore RFC 3021 specifies that for such small networks, we only need two addresses: a /31.

Wasting 75% of our address space would've been awful, but wasting 50% is a good trade-off for what we're winning.

The problem with Windows

Microsoft unfortunately did not read RFC 3021, so for those hosts we're stuck wasting two additional IP addresses. I've seen reports that one could use a /32 on the Windows side, and it will effectively work as if it was a /31, but this has not been verified by us.

The problem with some routers

Recently we ran into an issue where someone could connect to a service from work, but not from home. Investigation revealed that the problem arose because the endpoint was on an IP address that ends in .255.

Following the setup above, we can hand out all 128 prefixes from 1.2.3.0/31 up to 1.2.3.254/31. By convention we assigned the even IP — here 1.2.3.254 — to the switch: this meant that the endpoint/service got address 1.2.3.255.

This had been running fine for years, until now.

The user with issues was using the KPN network (86.90.0.0/16), with an Experia box V8 (Arcadyan, Astoria Networks VGV7519), S/N A30401xxxx, MAC 84:9C:A6:xx:xx:xx. Apparently that device treats addresses that end in .255 specially, and drops packets headed in that direction.

We tested this with two different public IP addresses that ended in .255. Neither IP would receive UDP or ICMP from this NAT-router. Traffic to neighbouring IPs had no issues. Maybe this is some overeager Smurf Attack protection?

(Here's a note to KPN: please fix your firmware!)

Avoid using 255 for endpoints

Moral of the story: we'll keep the convention of assigning the uneven address to the endpoint, except for the .255 address, which we'll assign to the switch instead. This should avoid problems with crappy routers in the future. We know the real network hardware beyond that will cope.