How to debug using GDB Server

:: tutorial, tools, debug

History: Another high traffic page from my previous site being transitioned here.


If you’ve ever worked with embedded systems or built code for a system other than your build machine you’ve probably found that running code on native hardware can sometimes generate different results than on your Beige Box. Being able to run and debug your code on your target hardware can be very useful but sometimes quite difficult to do. Using GNU’s Debugger gdb can be extremely cumbersome on its own and I typically prefer to use the GUI client ddd. With most of my projects, the target hardware does not have X support and most of my interactions are through Serial or Ethernet. If you are in the same situation you might find gdbserver, which comes with gdb, of use.

For a while now I’ve been using building applications using a Cross Compiler Jail (see blog post) on my development system to make the use of cross compilers much simpler. I can compile inside a system that looks and feels like I’m on the target’s OS having gcc, arch, and a bunch of other attributes all pointing to a ARM CPU even though I’m compiling on an x86_64. No need for calling gcc-arm-unknown-linux-gnueabi-gcc when the jail macros it to gcc.

Having this build environment is great. I can run the target binary before I even drop it on the board. But it still not the same when it comes to debugging. For this I need gdbserver.

gdbserver is an instance of gdb you run on your target hardware that allows all its hooks to connect to a desktop version. Since I’m working with two different architectures, I need to run gdb inside my jail to get all the hooks working correctly, which is no problem when using Scratchbox. But if you want to use ddd you run into the same lack of X support situation. ddd has a solution for that but theres a bit of a twist.

When starting ddd or gdb from the development system, the location of a copy of the libraries found on the target needs to be set before stepping through the code. To get these libraries to load do the following:

On the target, start gdbserver with your application. The IP Address is that of the development computer, not the target.

$ gdbserver 192.168.1.2:1234 ./hello
Process ./hello created; pid=1591
Listening on port 1234

The next step is to start up ddd pointing it to your local debugger.

ddd --debugger "/scratchbox/login -d $HOME gdb ./hello"

/scratcbox/login should point to the location of your Scratchbox install’s login script and ./hello is the binary compiled for the target hardware on your build machine. ddd should now start up to the main section of your code.

Now we need to setup the Share Library Path. To do this go to the gdb console at the bottom of ddd. See the notes at the bottom for more details.

(gdb) set solib-absolute-prefix /home/developer/target_rfs

Now set a breakpoint for your main function.

(gdb) b main

Continue your application till it hits your breakpoint. We can see if the shared libraries have loaded:

(gdb) info shared
 From        To          Syms Read   Shared Object Library
 0x40241fc8  0x402db84c  Yes         /home/developer/target_rfs/lib/libstdc++.so.6
 0x402f72d4  0x40362e60  Yes         /home/developer/target_rfs/lib/libm.so.6
 0x403a5c84  0x403ae09c  Yes         /home/developer/target_rfs/lib/libgcc_s.so.1
 0x403cd870  0x404a6bb8  Yes         /home/developer/target_rfs/lib/libc.so.6
 0x404cec18  0x404cf8e8  Yes         /home/developer/target_rfs/lib/libdl.so.2
 0x404dcee0  0x404e6f34  Yes         /home/developer/target_rfs/lib/libpthread.so.0
 0x40534e70  0x4053933c  Yes         /home/developer/target_rfs/lib/librt.so.1
 0x40000a80  0x40011b88  Yes         /home/developer/target_rfs/lib/ld-linux.so.2
 (gdb)

Notes: When setting the Shared Library Absolute Prefix you will need to point gdb to the location, on your host system, where the target’s root file system is located. This should be the exact file system that is copied onto your target when flashing. When gdbserver starts the application to debug, it looks for the libraries on the target’s file system. gdbserver then communications to gdb running on your host the path it uses to load these libraries (i.e. /lib/libstdc++.so.6). By setting the Shared Library Absolute Prefix on your host system, the local running copy of gdb is able to attach the prefix to the location it receives from the target’s gdbserver. At that point gdb running on your host system is able to load the libraries allowing you to debug your code.