torkell | On the IP host model

Yesterday's discovery at work is a little-known aspect of IP known as the weak and strong host model. Roughly speaking, in the weak host model a system behaves as if all of its assigned IP addresses are available on all interfaces, while in the strong host model a system behaves as if each of its assigned IP addresses belongs to a specific interface.

Linux uses the weak host model, with apparently no way to control this.

This, combined with a somewhat complicated network setup, lead to a weird situation where two boxes can ping each other, have full access to other networks, but cannot create a TCP connection between each other.

Server A is multihomed with two physical network interfaces. Let's say eth0 is on 192.168.1.10/24 and eth1 is on 192.168.2.10/24.
Server B has a single network interface 192.168.2.20/24.
There is a router that permits all traffic between the two subnets, and permits a limited amount of traffic between each subnet and the main engineering LAN.

Right, now server B needs to make a connection to server A on 192.168.1.10. Server B doesn't have a direct route to server A, so it sends the TCP SYN packet to the router for 192.168.2.0/24.

Server A receives this packet on eth0. Because the weak host model permits packets to be sent from any interface, it uses eth1 (configured with IP address 192.168.2.10) to send the SYN/ACK with a source IP address of 192.168.1.10. This SYN/ACK packet bypasses the router.

Server B receives the SYN/ACK. It now has to send an ACK to server A. Again, since it doesn't have a direct route it has to send the ACK via the router.

The router eats the ACK. Presumably the router is running a stateful firewall, and so because it hasn't seen the SYN/ACK it "knows" that server B shouldn't have sent the ACK yet.

Because server A hasn't received the ACK it doesn't know that the connection has been established, and so after a few retransmissions on both ends it kills off the connection.

I eventually had to use the Linux ip command to set up a static route on server A specifically for server B. Oh, and just to make tracking down this problem more fun, the exact same configuration was working perfectly the previous day.