'Twas brillig, and the slithy toves did gyre and gimble in the wabe

Today's discovery is that if you're foolish enough to enable DHCP on an alternate network interface (eth3, in my case) on Linux, then the DHCP client will overwrite your hand-configured default route that actually works with the one it received from the DHCP server.

Which is not particularly helpful when eth3 is connected to a different LAN with a router that actually checks the IP addresses of packets it forwards. At which point it eats the SSH connection I was using (because that was to an IP address on eth0) and I had to wander down to the lab and dig out a keyboard and monitor.

I am becoming more and more convinced that the Linux network stack just Does Not Work as soon as you plug it into more than one network.

Linux has this glorious thing known as the out of memory killer. The documentation claims that when the system runs out of memory it carefully works out which process is responsible, and kills it. This is of course completely false. What it really does when the system runs out of memory is select the most mission-critical process on the system and kill that instead. It then kills a few more processes for good measure.

Linux also (by default) happily overcommits memory. This means that you've got no guarantee that any memory you've malloc'd is actually available for use until you try to write to it.

Interestingly there is a justification for this. On Linux, the only way to start a new process is to fork your existing process, creating a complete copy of it. You then replace the copy of your process with whatever you actually want to run (ok, there's vfork(), but the man page for that contains the wonderful gem "[don't use] vfork() since a following exec() might fail, and then what happens is undefined").

So back in ye olden days when you only had 8MB of memory, your 5MB emacs process would fork itself. Both emacs instances now come to a total of 10MB which is 2MB more than you have, but that's OK because the second process hasn't changed anything and so shares the physical memory of the existing process (via copy-on-write semantics). The second one then gets replaced by your 1MB shell or whatever, taking the total down to 6MB. But if the second process actually wants exclusive use of the entire 5MB, then you've got a problem. And that's "solved" by the out-of-memory killer.

It's a perfectly sensible way to work around the insanity of the fork()/exec() model, except computers today have crazy amounts of memory and so don't actually need this workaround. And this workaround would never have been needed if there was an actual "create new process" syscall. Remind me again why Linux is better?

This rant brought to you by three servers getting broken in various ways due to the out-of-memory killer nuking about a half-dozen processes per server. Hope you didn't actually need Tomcat. Or MySQL. Or cron.

Today's annoyance was discovering that Linux apparently cannot cope with a system where multiple logical volume groups have the same name. Instead of doing something useful like complaining about this (or even better, warning about this and continuing), it just hangs. Part-way through startup. Before the virtual consoles are up, or any daemons have started.

Fortuantly someone else had seen this before and suggested removing a bunch of volume mappings from the SAN. Turns out all the virtual machines on that system (configured with a separate volume and LUN mapping on the SAN) used the same default name for the volume group, and that causes lvm to sit there burning CPU without actually doing anything. For added annoyance, once the system booted I then had to add the mappings back so the virtual machines on the system could start.

Of course, at no point is there any debug pointing out that I have multiple volume groups with the same name, or for that matter anything beyond a "please wait" message and an apparently hung system. Why must Linux be so hard to use?

This set of commands, run in a fresh gdb instance and intended to set a conditional watchpoint, results in gdb segfaulting leaving an orphaned debuggee running:

(gdb) watch function::variable if function::variable == 0xff
(gdb) run

This set of commands, again in a fresh gdb instance, works:

(gdb) info address variable
(gdb) info address function::variable
(gdb) whatis variable
(gdb) whatis function::variable
(gdb) watch function::variable if function::variable == 0xff
(gdb) run

Discuss.

Today's discovery is that Linux gets rather offended when the SAN containing / vanishes.

Interestingly, one of the Linux variants involved was much less offended, and "worked" as long as you didn't try to actually touch the disc. Running stuff like "cat" and "ls" even continued to work, presumably because I'd already run them recently and so they were cached. The other linux variant, however, was far more offended and limited me to just bash. Specifically, those portions of bash that were currently loaded into memory. And while the first remounted everything read-only when the SAN came back, the second refused to believe the reappearance of the SAN and wouldn't even let me log in at a real console. For added fun, Ctrl+Alt+Del had no effect because, rather than sending a command to the kernel as in Windows, under Linux merely causes /bin/shutdown to be run.

To be fair I've no idea how Windows would hold up in a similar situation, although I did once uninstall the IDE controller drivers for the C: drive, and while that drive did vanish from the system nothing spectacular happened as a result. Even Internet Explorer continued to work, although stylesheets stopped being applied, and I could even do a controlled shutdown without anything being able to write to C:. Possibly enough of the driver remained to let the kernel speak to the disk, even if it wouldn't admit so.

Memo to self: Linux doesn't like it when I change the IDE controller. It *really* doesn't like it.

In fact, it hates it so much that it's gone and not so much mangled as randomised small but critical bits of the filesystem, and now various parts of it fall over in interesting ways whenever I run them.

Well, I've been meaning to upgrade to Slackware 10.2 for a while.