(no subject)

[torkell]

Today's discovery was a full /var/log/wtmp file.

Those of you who know what that is are probably staring at this going "WTF?". For those that don't know (i.e. non-die-hard-linux-geeks), this file tracks all logins and logouts. Every time someone (or something) logs in or out, an entry gets added to this. And, following the Unix philosophy, no program ever expects that this file might become full. Because, of course, such a thing could never possibly happen. Ever.

Ha. Ha. Ha.

It turned out that the ftpd variant we were using wrote to this file on login/out (oh yes - on Linux it's the responsibility of each individual program to log account usage, not the operating system), and this particular system had a 2GB file size limit. Why, I don't know - even FAT could handle files larger than that. Anyway, given that this is a load box it was quite easy to hit the 2GB limit, and when this happened rather than return an error code Linux's default behaviour is apparently to send a SIGXFSZ signal. And the default behaviour for *that* is to terminate the process.

Comments

[tau-iota-mu-c.livejournal.com]

sigxfsz, does that imply that it was trying to write a 2GB core file (from the manpage), implying that it had tried to allocate 2GB, but hit a 32 bit problem, failed, segfaulted, couldn't write the core file, because it had mmapped >2GB?

[torkell]

I don't think so - I think the execution path went something like open(), seek(EOF), write(), and the sigxfsz was on the write(). Core dumps are supposed to be slightly intelligent, in that the core dump will be truncated if it exceeds the core limit.

The part of the manpages that confused me is supposedly SIGXFSZ is only sent if you exceed the per-process file size limit, and ulimit happily told me that there was no limit set. There is supposed to be a ETOOBIG or similar error code from write(), but I never saw that in the strace.