Tutorial: Reverse debugging with GDB 7

GDB 7 came out a few weeks ago, and one of the major new features is reverse debugging. This allows you to record the execution of a process, then play it backward and forward. This is incredibly useful for fixing those mysterious bugs that are so common in C/C++ programs.

In this post, I'll give a motivating example of why reverse debugging is so useful, then I'll give a summary of the commands you need to know about. You might also want to look at the official tutorial on the GDB wiki, which is a good reference.

Here's a typical program we might have in C. It's very simple, but it contains a subtle bug that causes it to crash when we run it.

#include <stdio.h>
#include <stdlib.h>

void initialize(int *array, int size) {
    for (int i = 0; i <= size; ++i)
        array[i] = 0;
}

int main(void) {
    int *p = malloc(sizeof(int));
    int values[10];

    *p = 37;
    initialize(values, 10);
    printf("*p = %d\\n", *p);
    free(p);

    return 0;
}

This program is compiled with the following command:

gcc -ggdb -std=c99 -O0 test.c -o test

When we run this program from the command line, we get a segmentation fault. What happened? We load it into gdb and enable recording:

(gdb) break main
Breakpoint 1 at 0x8048449: file test.c, line 11.
(gdb) run
Starting program: /home/jay/Code/test/test 

Breakpoint 1, main () at test.c:11
(gdb) record
(gdb)

The record command turns on recording. It must be issued after the program has started running, so the beginning of main is a good place to use it. If you are debugging code that runs before main (such as C++ global constructors), you may want to set a breakpoint in _start, the actual entry point of the program.

Once recording is enabled, we run the program until the segmentation fault occurs:

(gdb) continue
Continuing.
warning: Process record ignores the memory change of instruction at
address 0xdd45e1 because it can't get the value of the segment 
register.

Program received signal SIGSEGV, Segmentation fault.
0x0804847b in main () at test.c:15
(gdb)

After getting a weird warning (which appears to occur inside malloc), we find that the segmentation fault occurs at the call to printf. The only memory operation here is dereferencing p, so we print its value:

(gdb) print p
$1 = (int *) 0x0
(gdb) 

p should not be null! We set it at the beginning with malloc and never changed its value. We can find out where it was changed by setting a watchpoint and using the reverse-continue command. This works just like you would hope: the program runs backwards until the point at which p was changed. Note that we explicitly set a software watchpoint rather than a hardware watchpoint; GDB 7 seems to silently ignore hardware watchpoints when replaying the program.

(gdb) set can-use-hw-watchpoints 0
(gdb) watch p
Watchpoint 2: p
(gdb) reverse-continue
Continuing.
Watchpoint 2: p

Old value = (int *) 0x0
New value = (int *) 0x804b008
0x0804842c in initialize (array=0xbffff5c0, size=10) at test.c:6
(gdb) 

GDB stops on line 6 in the initialize function. Upon closer examination of the loop, we notice the off-by-one error in the condition. Since the array we passed to initialize is adjacent to p on the stack, we overwrite p when we write off the end of the array.

This kind of error is an example of a buffer overflow, a common bug in C. More severe versions of this bug can create security vulnerabilities. Overflows are usually quite difficult to track down because a variable like p could change many times before taking on a "bad" value like it did here. If we had set a watchpoint at the beginning of a more complex program, the debugger might stop hundreds of times before we found something interesting. More generally, we frequently debug by sneaking up on a fault from the front; if we accidentally pass the fault, we usually have to start over. Reverse debugging allows us to approach a fault much more quickly from behind. We just let the program run normally until we reach a point shortly after a fault has occurred (for instance, when the SIGSEGV signal is received). We then run the program backward until we find what went wrong. The debugger can do pretty much the same things running backward as forward, including setting new watchpoints and breakpoints.

Before we wrap up, here's a quick summary of useful commands for reverse debugging:

• record - enable recording mode. This command must be issued while the program is running, and you can only run the program back to the point where this command was issued. There is a performance penalty.
• reverse-step, reverse-next, reverse-finish, reverse-continue - just like the normal versions of the commands, but in the opposite direction.
set exec-direction reverse - switches the program execution direction to reverse. The step, next, finish, continue commands will work in the currect execution direction.
• set can-use-hw-watchpoints 0 - disables hardware watch points. This is necessary if you want to use watchpoints while running the program in reverse.

Once again, you need GDB 7 in order to use reverse debugging. It is supplied as part of Ubuntu 9.10 (Karmic) and other newer Linux distributions. Since reverse debugging a fairly new feature, it's not as complete as one would hope, so there's an official wish list which contains bugs and feature requests.

Finally, I'd like to reference Sebastien Duquette's tutorial on reverse debugging, which is where I found out that you need to disable hardware watchpoints. If you're interested in how reverse debugging actually works, Michael Snyder (one of the GDB developers responsible for it) has a short post on StackOverflow about it.