By default, if a thread with a higher priority than the mutex owner attempts to lock a mutex, then the effective priority of the current owner is increased to that of the higher-priority blocked thread waiting for the mutex. The current owner's effective priority is again adjusted when it unlocks the mutex; its new priority is the maximum of its own priority and the priorities of those threads it still blocks, either directly or indirectly.
This scheme not only ensures that the higher-priority thread will be blocked waiting for the mutex for the shortest possible time, but also solves the classic priority-inversion problem.
The pthread_mutexattr_init() function sets the protocol to PTHREAD_PRIO_INHERIT to allow this behavior; you can call pthread_mutexattr_setprotocol() to override this setting. The pthread_mutex_trylock() function doesn't change the thread priorities because it doesn't block.
You can also modify the attributes of the mutex (using pthread_mutexattr_settype()) to allow a mutex to be recursively locked by the same thread. This can be useful to allow a thread to call a routine that might attempt to lock a mutex that the thread already happens to have locked.