gdb fail

[torkell]

It never ceases to amaze me just how backward the Linux development environment is.

Today I attempted to debug a test program that segfaults about 5 minutes after startup for no apparent reason. I managed to get a core dump of it, and loaded it into gdb in the hope of finding what was going on. Hahaha.

gdb could give me a valid stack trace showing the error, and could disassemble the program around the error to show me the actual instructions involved. However, gdb could not tell me the value of all the variables there (it claimed that some variables weren't even defined, nevermind that the program uses them all over the place!), nor could it actually match the disassembly up to the source.

Come on, folks, Visual C++ has been able to do this for decades! The Windows debugging tools are so far ahead it's embarassing for Linux.

I did actually discover a patch to gdb to achieve this, submitted April last year. Unfortuantly it's not in the latest released version of gdb (released March last year), and I really don't fancy building gdb from source myself.

Comments

[13th-einherjar.livejournal.com]

I wonder if this is at all related to any of the randomization techniques applied a while ago to make buffer overflows fail most of the time. In that it's supposed to be non-trivial to figure out how all of the variables are laid out in memory, because that way hackers can't write 8 bytes to 4 byte int a and know that the next 4 bytes are definitely going to int b. Probably no relation.

I find this strange, because one of my first reasons for switching to Linux was the ease of compiling and developing software. Back then, Linux systems still assumed you wanted to compile something, so the toolchain was already installed and pretty well standardized. And Windows offered 2 options a) pay thousands of dollars to have your IDE installed for you b) spend several days groping through undocumented system calls and crippleware compiler toolchains to write "Hello World" in the language and library of choice.

[torkell]

Nope, the lack of variable names turned out to be mainly due to optimisation. I say mainly - even after removing "-O2" from the compiler flags it still wouldn't reliably show the variables, possibly because some were declared in blocks within functions (as opposed to being at function scope).

[pewterfish.livejournal.com]

To be entirely fair, that the program uses them doesn't necessarily mean the variables were defined :D GCC and GDB have been known to disagree on this sort of thing from time to time.

In general, though, gdb is ... not the friendliest program in the world. The few times I've investigated it, its output has been so terse and unreadable that I've ended up going back to the tried and tested "print statements and see where they go wrong" method eventually. I hear there are a few third-party front-ends that make it more usable, though.

Since I'm about to get back into programming AVRs under Linux, I should probably get some more time in with gdb. Gonna be useful...

[torkell]

Turned out that the lack of variables was mainly due to optimisation. Presumably gcc's optimiser is a very different beast to MSVC, as when debugging optimised Windows DLLs WinDbg managed to keep track of everything. They still didn't reliably appear with optimisation disabled. Oh, and to get an annotated assembly listing you can use "objdump -S" but only if your debugging information is in Stabs format (which is neither documented nor the default) and your program is not optimised. This allowed me to finally track down the bug (someone declared a function-level temporary variable as static and then used that function as the entry point for 84 threads - the best part is this program worked when it was first used many years ago!)

On frontends... I've played with ddd (a X frontend for gdb), and it's a lot friendlier to use but does require X installed. It still completely failed at showing me the contents of things like std::vector, but that's more of a gripe with their standard library (and I did find a macro for gdb to decode those). I doubt you'll want to install X on your AVR, which leaves you with the wonderfully flaky gdb server system (which is a thin client setup, so you need your symbols on the target, and the remote debugger tended to segfault every time I told it to restart the debuggee).