Debugging multiprocessing software with GDB

September 28, 2024

Backstory

Since ever I started the development of my high school finals project - a proxy library - I have had the desire to master the art of debugging multiprocessing software. Up until now I had no idea it was even possible due to the scarse general information available about it.

set follow-fork-mode [mode] seemed promising and quite honestly, worked as expected. Except, I realized that to debug my proxy, I was in need of a more dynamic approach such as:

being asked whether to follow the child/parent as I catch multiple forks
switching between the parent or multiple children instantly
releasing the parent while I only debug the child or vice versa

I read more about it online and it seemed as if the first of my requirements was an already supported feature of GDB - (set follow-fork-mode ask) - which would ask the user at runtime whether they want to follow the child or the parent the second they hit a fork/vfork syscall. Sadly, newer versions of GDB have had it cut off and deprecated for whatever reason.

The frustration got me really close to taking initiative and writing myself a little patch for the modern versions of GDB so they would also have this useful feature. But something told me there had to be a better way. And oh boy, better way there was.

Start

 1/* main.c */
 2
 3#include <stdio.h>
 4#include <unistd.h>
 5
 6int main(void) {
 7    fprintf(stdout, "--multiprocessing example--\n");
 8    for (int i = 0; i < 5; i++) {
 9        pid_t pid = fork();
10        if (pid == 0) {
11            fprintf(stdout, ">>hello from child: %d\n", i);
12            return 0;
13        }
14        if (pid < 0) {
15            fprintf(stderr, ">>failed forking\n");
16            return -1;
17        }
18    }
19
20    fprintf(stdout, ">>hello from parent\n");
21
22    return 0;
23}

We are going to step through this program - getting into the nitty griddy of debugging and seeing what options we have depending on our demands. First of all, use your compiler of choice. I am going to use GCC and turn on debugging info.

gcc -g3 main.c -o main

Before we continue, make sure you at least have the following setting enabled in .gdbinit:

set detach-on-fork off

A very convenient option that will block execution of both the parent and the new child - not letting one of them run unless you manually call continue.

Following the parent

This scenario is unsurprisingly simple. GDB does this automatically, therefore, I will not cover it.

 1[hemisquare@detached-hemi tmp]$ gdb main
 2(gdb) break main 
 3Breakpoint 1 at 0x1161: file main.c, line 7.
 4(gdb) run
 5Starting program: /home/hemisquare/tmp/main 
 6[Thread debugging using libthread_db enabled]
 7Using host libthread_db library "/usr/lib/libthread_db.so.1".
 8
 9Breakpoint 1, main () at main.c:7
107	    fprintf(stdout, "--multiprocessing example--\n");
11(gdb) s
12--multiprocessing example--
138	    for (int i = 0; i < 5; i++) {
14(gdb) 
159	        pid_t pid = fork();
16(gdb) 
17[New inferior 2 (process 51915)]
18[Thread debugging using libthread_db enabled]
19Using host libthread_db library "/usr/lib/libthread_db.so.1".
2010	        if (pid == 0) {
21(gdb) p pid
22$1 = 51915
23(gdb) # we are the parent

Following the child

Similar to the parent example, I will step until we hit the first fork syscall.

 1[hemisquare@detached-hemi tmp]$ gdb main 
 2(gdb) break main
 3Breakpoint 1 at 0x1161: file main.c, line 7.
 4(gdb) run
 5Starting program: /home/hemisquare/tmp/main 
 6[Thread debugging using libthread_db enabled]
 7Using host libthread_db library "/usr/lib/libthread_db.so.1".
 8
 9Breakpoint 1, main () at main.c:7
107	    fprintf(stdout, "--multiprocessing example--\n");
11(gdb) s
12--multiprocessing example--
138	    for (int i = 0; i < 5; i++) {
14(gdb) 
159	        pid_t pid = fork();
16(gdb) 
17[New inferior 2 (process 52175)]
18[Thread debugging using libthread_db enabled]
19Using host libthread_db library "/usr/lib/libthread_db.so.1".
2010	        if (pid == 0) {
21(gdb)

So, currently we are inside the parent “inferior”. And inferior is just a GDB term for processes, threads, or whatever it is that you are debugging. Through forks, we create another inferior which we can switch to. As you can see, we now have two inferiors:

1(gdb) info inferiors
2  Num  Description       Connection           Executable        
3* 1    process 52172     1 (native)           /home/hemisquare/tmp/main 
4  2    process 52175     1 (native)           /home/hemisquare/tmp/main 
5(gdb)

The ‘*’ indicates the inferior we are currently residing in. Inferior 2 is the newborn child we just forked. Let’s switch into the child!

 1(gdb) inferior 2 
 2[Switching to inferior 2 [process 52175] (/home/hemisquare/tmp/main)]
 3[Switching to thread 2.1 (Thread 0x7ffff7dab740 (LWP 52175))]
 4#0  0x00007ffff7e90b57 in _Fork () from /usr/lib/libc.so.6
 5(gdb) finish
 6Run till exit from #0  0x00007ffff7e90b57 in _Fork () from /usr/lib/libc.so.6
 70x00007ffff7e966e2 in fork () from /usr/lib/libc.so.6
 8(gdb) finish
 9Run till exit from #0  0x00007ffff7e966e2 in fork () from /usr/lib/libc.so.6
100x0000555555555192 in main () at main.c:9
119	        pid_t pid = fork();
12(gdb) nexti
1310	        if (pid == 0) {
14(gdb) p pid 
15$1 = 0
16(gdb) s
1711	            fprintf(stdout, ">>hello from child: %d\n", i);
18(gdb) 
19>>hello from child: 0
2012	            return 0;
21(gdb)

As you can see, here, we were able to step through the code of the child. Notice that the parent is still handing where we left it at. It is totally okay to now switch back into the parent inferior and continue the execution from there.

Conclusion

Knowing this is essential for debugging my proxy library. I am thankful I now know GDB just a little better and was hopefully able to provide you with a useful learning resource. I might extend this article in case I master more interesting multiprocessing or multithreading techniques.