Locking in the resource manager

The resource manager framework provides facilities for safe thread synchronization in your resource manager. Before you write a multi-threaded resource manager, make sure you are familiar with the default behavior of the shared data structures that thread synchronization uses and the requirements for safely overriding them.

Shared data structures for multi-threaded resource managers

The OCB contains the open mode of the file descriptor (RDONLY, WRONLY, RDWR), its current offset, and a count of how many times it has been duplicated (using dup()). Some of these items are static, but the offset changes frequently. Successive operations on the same file descriptor have a dependency on the offset set by the previous operation.

The attributes structure represents the file itself. It stores information that changes rapidly such as the file credentials, the access permissions, size, and timestamps.

Because POSIX requires that simultaneous reads and writes to a file don’t result in half-updates, the attribute structure needs to be locked for all operations. The resource manager framework meets this requirement by effectively having a single mutex for each attribute structure. By default, this mutex is manipulated by io_func_attr_lock() and io_func_attr_unlock(). The resource manager framework acquires it on the relevant attribute structure before it invokes any I/O handlers and releases it once the handler returns.

Because this mechanism finds the attribute structure to lock via the OCB associated with the file descriptor, it only works for file descriptor-based operations. Connect operations are pathname-based and do not have a file descriptor or OCB. Instead, the attribute structure of the mountpoint (the path given to resmgr_attach()) is the handle parameter of resmgr_attach(). In filesystem resource managers (i.e., when the _RESMGR_FLAG_DIR flag is passed to resmgr_attach()), the framework can provide the attribute structure for the mountpoint, but does not know how to locate the correct attribute structure for the file or directory being processed. Therefore, it is the responsibility of your own connect handlers to determine and acquire the attribute structure associated with the named file.

Note: Do not simply allocate a fresh attribute structure on every open(). For both locking and memory-mapped files to function correctly, every open() called on a file requires a call to iofunc_ocb_attach() against the same attribute structure.

Overriding the locking behavior

Before it invokes your I/O handler function, the resource manager framework invokes the functions pointed to by the resmgr_io_funcs members lock_ocb and unlock_ocb to lock and unlock the OCB structure (see iofunc_func_init() and resmgr_attach()). By default, iofunc_lock_ocb_default() and iofunc_unlock_ocb_default() are used, which simply lock or unlock the attribute structure associated with the OCB.

The attribute structure is locked and unlocked via the functions that the iofunc_mount_t member funcs points to (_iofunc_funcs members attr_lock, attr_unlock, and attr_trylock). When these function pointers are not populated (the default behavior), the OCB locking functions call iofunc_attr_lock() or iofunc_attr_unlock(), which means that the resource manager framework expects that calls to lock_ocb() and unlock_ocb() also lock or unlock the related attribute structure. This default behavior is "one-level" locking—attribute locks are used to lock OCBs—and should be sufficient for most resource managers.

However, if required, you can implement "two-level" locking, which allows your resource manager handler to release the lock on a shared attribute but preserve the OCB lock. This behavior is useful if your resource manager supports independent operations on the same named entry.

For both "one-level" and "two-level" locking:

You can either use the default iofunc_attr_lock() and iofunc_attr_unlock() for the attribute lock or implement your own and populate attr_lock and attr_unlock.
Make sure locking is recursive (see pthread_mutexattr_setrecursive()).

For "one-level" locks, iofunc_lock_ocb_default() and iofunc_unlock_ocb_default() use your attr_lock and attr_unlock if you populate them. You don't need to create your own functions for lock_ocb and unlock_ocb in resmgr_io_funcs.

Creating "two-level" locks requires you to create your own functions for lock_ocb and unlock_ocb. To allow your code to work well with the default resource manager handlers, make sure that these functions use the same attribute structure lock functions as the ones that iofunc_mount_t references.

For "two-level" locks, make sure your code also includes the following behavior:

Calls to lock_ocb() also lock the attribute.
The OCB is locked first and the attribute structure is locked second, which allows you to unlock iofunc_attr_t as needed while retaining the OCB lock.

For both default and custom locking (using one or two levels), it’s safe for a handler to temporarily unlock the attribute structure (or the OCB, or both the attribute and OCB, in the case of "two-level" locks), as long as it locks the affected items before it returns. If you use custom lock functions, make sure they cannot fail.

Having handlers that temporarily unlock structures means that other threads are allowed to run handlers. In particular, if you unlock the OCB, another thread could close it. After you lock the OCB again, the OCB is still valid (it is not free until after the handler returns) but you might need to be careful about anything in the OCB that your close_ocb handler might have invalidated.