Inter-process communication (IPC) is way how memory isolated processes talk to each other in CMRX RTOS. We chose to implement remote procedure calling rather than message passing as main mechanism for IPC for practical purposes and that it provides more familiar semantics. Secondary IPC mechanism often used in CMRX are notifications that provide means of synchronization.
Here we’ll take look on IPC usage in CMRX, more specifically:
Remote procedure calling is a way how a thread owned by one process can call function (or procedure) residing in another process in a way this function will have access to its own memory. If this function was called directly then hardware would signal memory protection fault the very moment function would try to access memory of process it belongs to. This is due to violation of rule: Code has access to any memory residing in same process a thread belongs to. RPC bypasses this limitation in a secure way.
If a function is called via RPC mechanism, then kernel will grant this function access to memory belonging to owning process. This way a secure and well-defined interface to a block of memory can be created which provides effective data encapsulation similar to interface of object-oriented programming languages where internals of object are hidden from outside world while an interface is visible which allows observation and modification of this data in expected and controlled manner.
RPC calls are synchronous by nature. Calling function is blocked during the RPC call in a way similar how a function is blocked during nested function call. In fact, RPC call semantics is modeled after function call semantics. If any thread executes RPC call, its ownership is temporarily transferred to the process which owns the RPC service. This is the primary mechanism how RPC call gains access to caller’s process memory. RPC call itself is executing using caller’s stack which it has unlimited access to. So one of means of transferring data between caller and callee may be stack-allocated buffers.
Another mechanism is to define buffers which are shared during RPC calls. Developer is free to decide which buffers remain private (the default state) and thus are unaccessible to RPC-called functions and which buffers are shared with RPC code.
Processes that want to offer their services to other processes have to create so called RPC servers (used interchangeably with term RPC services). RPC servers are ordinary C structures which reference implementation of some existing API. Lets break this to individual pieces:
vtable.Lets be more specific:
An API structure is a list of actions some service supports. This is just a definition of API, not the implementation. Lets create an API for simple UART driver:
typedef struct {
bool (*init)(INSTANCE(this), uint32_t baud_rate);
int (*send)(INSTANCE(this), const uint8_t * buffer, uint8_t buf_len);
int (*receive)(INSTANCE(this), uint8_t * buffer, uint8_t buf_capacity);
} UARTInterface;
This API consists of three methods: init, send and receive which perform indicated actions. Each of methods accepts first argument in the form of INSTANCE(this) which is an idiom that gives the implementation access to RPC server instance. This form of instance specification will avoid generating compiler errors on type mismatches and remove need of either caller or callee to typecast pointers. API definition is implementation-agnostic, it doesn’t care which specific server is implementing the API.
Methods in APIs must have INSTANCE(this) as their first argument. Other than this argument, they may have 0 to 4 additional arguments which are either C integrals or pointers. Arguments may not be of floating point type nor structures passed by value. They may have return value of either void or any C integral type that fits into 32-bit quantity.
A process which decides to implement this API must provide specific implementations for methods in the API. These implementations are specific, because they access members of specific structure that implements this API. First, lets show how a RPC service that implements above API looks like:
typedef struct {
const UARTInterface * vtable;
USART_TypeDef * usart;
uint32_t base_clock;
} UARTService;
In above example, the structure declares three members. First one is reference to API which this service implements. Remaining are data members holding state information of service instance, in this particular case reference to USART peripheral this instance manages and frequency of peripheral clock. It is important to note that despite the structure of service is known to service user, the content of instance is inaccessible as it resides in memory of different process.
Now, once the declaration of service exists, we can start implementing API UARTInterface for this service:
#include <cmrx/rpc/implementation.h>
IMPLEMENTATION_OF(UARTService, UARTInterface);
static int usart_init(INSTANCE(this), uint32_t baud_rate) {
LL_USART_SetBaudRate(this->usart, this->base_clock, LL_USART_OVERSAMPLING_8, baud_rate);
return true;
}
static int usart_send(INSTANCE(this), const uint32_t * buffer, uint8_t buf_len) {
uint8_t cursor = 0;
for (; cursor < buf_len; ++cursor)
while (!LL_USART_IsActiveFlag_TXE(this->usart)) {}
LL_USART_TransmitData8(this->usart, buffer[cursor]);
}
return cursor;
}
static usart_receive(INSTANCE(this), uint32_t * buffer, uint8_t buf_capacity) {
uint8_t cursor = 0;
for (; cursor < buf_len; ++cursor)
while (!LL_USART_IsActiveFlag_RXNE(this->usart)) {}
LL_USART_ReceiveData8(this->usart, buffer[cursor]);
}
return cursor;
}
VTABLE UARTInterface uart_vtable = {
&usart_init,
&usart_send,
&usart_receive
};
First, we see that macro IMPLEMENTATION_OF has been used. This macro specifies that we are going to implement API UARTInterface for service of type UARTService. Its use has two major effects:
UARTService really declares that it provides UARTInterface API. If the declared API is different, structure is malformed or is missing API declaration completely, compiler will issue an error and build will fail.this argument is going to be typed as UARTService * automatically.Next, three functions are defined. These functions have exactly the same prototype as members in API declaration, including the INSTANCE(this) way of referencing the service instance. We can see that these functions are specified as static. This is intentional as it makes no sense to call them directly and thus their symbols are of no use externally.
Lastly, there is an instance of UARTInterface structure created, which is initialized with pointers to functions which were defined above. Note the VTABLE specifier in front of variable definition which makes this variable a legitimate API implementation. Only variables with this specifier are allowed to be used as API implementations for RPC calls. This prevents attackers from forging their own fake API implementations containing pointers to foreign functions and using these to gain access to foreign process’ memory.
We can see that bodies of functions use the this pointer as if it was typed to UARTService. This is supported and won’t cause any errors as the INSTANCE_OF macro sets correct type for this in the implementation phase. Function bodies have access to internal members of the service structure.
Last missing piece is the creation of RPC service instance. This boils down to simple definition of variable typed as UARTService:
UARTService usart1_svc = {
&uart_vtable,
USART1,
8192000
};
UARTService usart2_svc = {
&uart_vtable,
USART2,
8192000
};
Here we see two instances of UARTService being created, one providing access to USART1 peripheral, another providing access to USART2 peripheral. Both peripherals reference the same implementation of UARTInterface API.
Now that the RPC services were defined and implementation for APIs were created, it is possible to call these services. Caller will need access to declaration of UARTInterface and UARTService types and forward declarations of usart1_svc and usart2_svc variables. It can call this service like this:
#include <cmrx/ipc/rpc.h>
/* ... */
bool success = rpc_call(&usart1_svc, init, 115200);
rpc_call is a macro that wraps RPC call kernel syscall. The prototype of this macro is:
#define rpc_call(service_addr, method_name, args...) /* ... */
Argument service_addr is a pointer to RPC service instance, here we’ll use usart1_svc as service we call prepending & to get its address. Next, we’ll refer the method we want to call by its API name not the implementation name. Identity of function that implements the API is hidden from callers. Remaining arguments are arguments to selected method. The first argument to API methods (this) is not passed explicitly in rpc_call. It will be added automatically and its value will be value of service_addr argument.
Macro performs compile-time type checking of arguments passed to rpc_call() and expected arguments of the API usart1_svc implements (here: UARTInterface). If these are incompatible, build will fail. Macro will check that the method_name provided is one that exists in API specified service implements. If no such method exists, build will fail. Internally, RPC call is converted to CMRX kernel call which will verify that API implementation referenced by usart1_svc is a legitimate API implementation and won’t forward the call if this is not the case.
RPC call syscall is synchronous, it will cause the calling thread to be “forwarded” into process owning RPC service, which will temporarilly take ownership of the thread. Thus the rule that code can access any memory that belongs to process owning the thread is still valid during RPC call, just the identity of the owning process has changed temporarilly.
This implementation details has a consequence: Process which provides RPC services doesn’t have to create own threads. RPC services will be executed in the context of calling threads. On the other hand, calling threads must have stacks large enough to accommodate RPC calls.
In some cases, providing all potential callers with full internal structure of service may be impractical. For example if the full declaration uses types that are not common and it is unfeasible for callers having to include their headers as they are not using any of them. Knowledge of internal structure of service is of no practical use to callers anyway as it exists outside their reachable address space.
In such cases, it is possible to provide opaque alternative declaration for service and publish this service using this opaque type:
typedef struct {
const UARTInterface * vtable;
} UARTServiceT;
UARTServiceT * usart1 = interface_cast(USARTServiceT, &usart1_svc);
In this example type UARTServiceT is declared only containing reference to interface which is the same as in USARTService type before. Next, a variable usart1 is created which contains a type-casted pointer to usart1_svc variable from previous examples. Type-casting is performed using interface_cast() macro. This macro takes two arguments:
This macro wraps an ordinary typecast and compile-time check that original type of RPC service implementation and target pointer type after typecast are both referencing same API type and thus implement same API.
CMRX RTOS doesn’t allow unbounded sharing of memory. Two processes can’t agree upon common memory region both will be able to read and write. CMRX allows only bounded sharing of memory during RPC call. If you note the buffer argument in previous examples, this argument provides a pointer to buffer which is either read from or written into. These buffers are in caller-accessible address space, so normally, RPC service won’t have access to this memory.
In order to make these buffers shareable with any RPC server, their definition must look like:
#include <cmrx/ipc/shmem.h>
SHARED uint8_t uart_buffer[64];
The key is the SHARED specifier, which will ensure that any variables carrying it will be accessible by any RPC service called from this process. No additional work is needed, everything will be configured automatically.
In certain cases, asynchronous or much more simplistic way of communication is needed. CMRX kernel provides additional way of IPC which is suitable for synchronization between multiple threads. This mechanism is notify / wait primitive.
Any thread can issue a wait for notification system call. This call will block the thread until notification is received (or, if timeout period is specified, until wait times out). Such thread won’t be scheduled until resumed by either event. Any amount of threads can wait for single notification. If this notification arrives, the thread with highest scheduling priority will be woken up.
Notifications are queued, so if a notification is sent nobody is waiting for it is not lost. This allows direct use of notify / wait mechanism as replacement for counting semaphore. Notification is equivalent of “signaling the semaphore”, wait for object is the same as “waiting for semaphore”.
Following example demonstrates the use of notify / wait:
uint8_t buffer[64];
wait_for_object(&usart1_svc, 0);
rpc_call(&usart1_svc, receive, buffer, sizeof(buffer));
notify_object(&usart1_svc);
In the example above we see that wait_for_object and notify_object are both using usart1_svc as object being notified / waited for. usart1_svc is not a notification object, rather a RPC service which we defined earlier. This is due to the semantics of notify_object and wait_for_object. They accept an identifier of object which is a meaningless number. To simplify creation of unique identifiers, addresses of existing object can be used. So if you want to implement semaphore around access to RPC service, it is possible to use the address of RPC service instance as identifier to be signalled.
Note that above example is not entirely correct though - if RPC service needs exlusive access, then it should be maintained inside the implementation. Callers shouldn’t care about its needs. Second - example above models semaphore semantics which is not correct here. Mutexes should be used instead as semaphores (and notifications) allow external threads to signal the semaphore breaking the logic. In CMRX, mutexes are provided as another synchronization primitive and their implementation is also based on notifications.
In early stages of development, CMRX tried to support larger subset of POSIX API so pthread-like signals were added to the list of kernel-supported features. CMRX provides support for standard asynchronous signal delivery although the semantics of signal delivery is somewhat simplified. Signals are always delivered to threads, not processes. When signal handler is called, it is given mask of active signal bits which were set before handler was fired.
While this mechanism is still present in the kernel, it causes some substantial security issues and as such it is considered deprecated. We’ll remove them completely in some of future kernel releases. Don’t use signals to notify threads, use notifications instead. They are queued, so you won’t lose a notification if they come faster than you manage to serve them and notifications are delivered synchronously, which provides consistent thread state, unlike signals whose delivery is asynchronous and can interrupt receiving thread in any moment.
In the future, signals are expected to continue to exist, but their purpose will change to handle system-relevant and exceptional events which can’t reasonably be anticipated by a thread to be delivered.
Signal API is similar to that of POSIX. To send a signal to thread, send_signal() function can be used (aliased as kill()). This function recognizes two kinds of signals - user-provided and system-handled.
There are 32 user-provided signals, which set corresponding pending bit in thread’s signal mask. When signals are delivered and signal handler routine is dispatched, then this mask is provided as argument. Handler must be ready to process more than one signal. Once signal handler is dispatched, pending mask is cleared.
System-handled signals are used to implement system-specific tasks. These signals are never delivered to signal handler, rather manipulate thread state:
SIGSTOP - will force thread to stop, stopped thread wont’t be scheduled for execution and can be resumed by reception of user-defined signal or SIGCONT signal (see below)SIGCONT - will resume a stopped threadSIGKILL - will forcefully kill a threadSIGSEGV - same effect as SIGKILLFunction signal_handler() can be used to set signal handler routine for the current thread. This function will be injected on stack when signal is delivered. Signal reception can’t be disabled entirely, but if signal handler is either NULL or not set at all, the only effect of signal reception will be that a stopped thread will be resumed. Signal handler routine takes one argument, which is a bitmask of user-provided signals pending at the time of handler dispatch.