Userspace components running on top of CMRX kernel can synchronize access to shared data using mutexes. CMRX kernel provides mutex implementation commonly known as futexes. Futex stands for fast userspace mutex, which means that most of mutex code actually runs in userspace and not in kernel. Decision to provide this information was made to provide faster synchronization in low-contention cases. As futexes don’t use kernel services for most of code paths, context switch from usermode to kernel mode and back is avoided, which makes futexes faster.
In turn, it creates one limitation - mutexes are userspace objects residing in process data regions and thus mutexes can’t be shared between processes. Yet in memory-isolated environment this is rarely ever wanted. CMRX mutexes are not recursive. Locking same mutex multiple times from same thread will deadlock that thread instantly.
Mutexes in kernel deal with priority invesion situation. This happens when a thread with lower scheduling priority locks a mutex and then thread with higher priority is resumed which wants to lock the same mutex. A high-priority thread effectively gets blocked by low-priority thread which may in turn get blocked by threads which are not related to this workload but their priority is higher than that of the thread holding the lock.
If a thread running under CMRX fails to lock a mutex because another thread is already holding it, the blocked thread provides the lock holder with its own scheduling priority. Lock holder gets his priority increased to whatever the priority of blocked thread is at the time of mutex being locked, so it can finish whatever job being protected by mutex as early as possible. When it unlocks the mutex, scheduling priority is returned back to its normal priority.
If multiple threads are waiting for a mutex that is locked, the thread with highest scheduling priority is allowed to lock the mutex once it is unlocked.
“Happy path” of mutex execution is implemented almost completely in userspace on all platforms that provide some form of atomic update capabilities. For mutex, “happy path” consists of uncontended locking. Mutexes behave the same regardless there are any waiting threads when a mutex is unlocked.
In this happy path scenario, kernel is not called during mutex locking. This happens entirely in userspace. Only if mutex locking fails due to anoother thread holding the lock, kernel is involved. Mutex unlocking always makes a system call to notify potential waiters.
As mutexes are userspace construct, they have to get allocated and initialized before the first use:
#include <cmrx/ipc/mutex.h>
/* .... */
futex_t mut;
futex_init(&mut);
The usage of mutexes is then similar to other environments which support them. Typical usage of mutex will be formation of critical section around access to shared piece of data:
futex_lock(&mut);
/* modify data */
futex_unlock(&mut);
Call to futex_lock() is blocking if mutex is held by another thread. If thread doesn’t want to block on this call, then it can use futex_trylock() instead. This function will return true if mutex was acquired by calling thread, false if mutex cannot be acquired but unlike mutex_lock() it will never block. Thread is free to handle this situation however it wants.
Attempt to recursively lock a mutex will cause a deadlock by thread waiting for itself to unlock the lock. This situation is not diagnosed in release builds as mutexes are often used in hot paths. Diagnostics is only performed in debug builds. Attempt to unlock a mutex from different thread than one that locked it yields an undefined behavior. This is also only diagnosed in debug builds as such diagnostics slows mutexes down and is caused by programming errors rather than runtime behavior.
CMRX also supports architectures which don’t provide support for atomic update in userspace. In case of such architectures, mutexes have same externally-visible behavior, just the internal behavior changes. All paths of mutex execution go through kernel, thus mutex processing is slower.