Using fork() in a multithreaded process

Using fork() in a multithreaded process can be tricky, especially if one thread calls fork() while other threads are in critical sections of code.

POSIX says that following a fork(), the child's memory state is a copy of the parent's, but the child contains only a single thread; any other threads in the parent aren't duplicated in the child. If one of those parent threads was in the middle of manipulating a data structure when the fork() happened, that data structure will be in an undefined, possibly inconsistent, state in the child following the fork(). Regular mutexes can be used to protect different threads from interfering with each other, but protecting against fork() requires the forking code to be aware of the internals of your data structure synchronization.

For example, suppose the parent process has two threads, T1 and T2, and T2 has locked a mutex. If T1 calls fork(), the child process is a copy of the parent, including any mutexes, but with only one thread. If the thread in the child tries to lock the mutex, the operation fails because T2 has it locked—and there is no T2 in the child process, so no one can unlock the mutex. This can lead to a deadlock.

The simplest solution is to call fork() only from a single-threaded process, or to call it before you create additional threads. But, if you insist on using fork() in a multithreaded process, there are some things you can do:

• Use at-fork handlers
• Use forksafe mutexes
• Call exec*() soon after forking. POSIX requires that the child process use only functions that are async-signal-safe until it calls one of the exec*() functions.

You can use pthread_atfork() to register the following handlers:

• prepare — called before forking
• parent — called in the parent after forking
• child — called in the child after forking

You can call pthread_atfork() more than once if you want to register additional handlers; the parent and child handlers are called in the order that you registered them, and the prepare handlers are called in the reverse order.

In the prepare handler, you can lock all the mutexes, and then unlock them in the parent and child handlers. This ensures that no other threads can hold the mutexes across the fork(). However, if you have many different mutexes in different code that is only loosely coupled, getting the locking order correct to avoid deadlock can be tricky.

We've implemented a forksafe mutex as another way to deal with this problem. A forksafe_mutex_t is a replacement for a pthread_mutex but is mutually exclusive with fork() calls. The fork() function is aware of mutexes of this type and can't execute while any forksafe mutexes are held: fork() blocks while any forksafe mutexes are held, and while a fork() call is underway, any attempts to lock a forksafe mutex block until the fork() has completed.

You can use the following functions to work with forksafe mutexes:

• forksafe_mutex_destroy()
• forksafe_mutex_init()
• forksafe_mutex_lock()
• forksafe_mutex_trylock()
• forksafe_mutex_unlock()

They replace the corresponding pthread_mutex_*() functions; for more information, see the C Library Reference.

Note the following:

• There are two versions of the forksafe_mutex_*() functions:
  • The functions in libc are declared as weak symbols and use a regular mutex, simply calling the corresponding pthread_mutex_*() functions.
  • The functions in libforksafe_mutex use a forksafe mutex as described above.

  In order to use a forksafe mutex, you need to link your program against libforksafe_mutex.

• There's a performance cost to using forksafe mutexes.
• Forksafe mutexes have priority over fork() calls. This means that if a forksafe mutex is held by one thread, and a second thread is blocked trying to fork(), and then a third thread tries to lock a forksafe mutex, the third thread succeeds—it doesn't wait until the fork() call has been given a chance to run. This means that if you have a lot of different threads grabbing and releasing forksafe mutexes, an attempt to call fork() may be blocked indefinitely; it isn't allowed to proceed until all forksafe mutexes are released.
• While forksafe mutexes support priority inheritance between users of the mutex, there's no priority inheritance between the users of the mutex and a caller of fork(). A high-priority thread attempting to call fork() can be blocked by a low-priority thread that's holding a forksafe mutex.
• You mustn't hold a forksafe_mutex_t when you invoke fork(), or an obvious deadlock will occur. The same is true of a pthread_mutex_t protected by an at-fork handler.

At-fork handlers and forksafe mutexes will help, but it can still be tricky to correctly set up the mutexes in the child, so our recommendation is not to use fork() in a multithreaded process, or to use posix_spawn() instead.