Hosted Execution
CMRX maintains identical behavior across all platforms. Whether running on ARM Cortex-M hardware or hosted on Linux, the kernel API, scheduler, threading, RPC, and notifications work the same way. Hardware-specific features like floating-point support may vary, but the core kernel remains consistent.
Operating systems face a trade-off: support many platforms with limited features, or provide rich features on select platforms. CMRX takes the latter approach, requiring hardware support for memory isolation as a baseline. This requirement determines platform eligibility.
Running Without Bare Metal
Portable kernels raise an interesting question: Does the target platform have to be a bare metal physical machine?
Normally, operating systems execute directly on physical hardware, having full and authoritative access to the CPU. In some cases, there might be a hypervisor involved if there are virtual machines. In many such cases, the operating system still behaves as if it was running directly on the target hardware. Facilities used by hypervisors on modern hardware are mostly hidden from the operating system and so it still believes, it controls everything.
Some operating systems take a different approach: running atop another OS.
How is this different from ordinary virtual machines, which sometimes run on top of another operating system? In case of virtualization, where hypervisor runs inside another operating system (e.g. in case of QEMU) the hypervisor creates an illusion of complete machine, including all peripherals that would normally be present on said machine. Virtualized operating system then talks to these virtual peripherals. Hypervisor provides translation from the virtualized peripheral to some real hardware. This translation, while usually extremely useful, has some drawbacks:
Translation mechanism for virtual peripherals causes performance penalties. Operating system believes that it is talking to the real hardware. This communication has to be intercepted by the hypervisor and then translated to whatever way valid for host environment. This translation causes some overhead as virtualized peripheral has to react to operating system input and cadence of data transfer is often different between the host system and the virtualized system.
In practice this overhead is avoided by creating so-called paravirtualized devices, which have an API defined for both host and guest side yet they don’t represent any physically existing device. OS still talks to such paravirtualized device as if it was real hardware. Yet its API is optimized for minimal need of translation and thus minimal overhead.
Second problem of virtualization is that if you intend to run an operating system targeted for some platform, you have to virtualize every essential part of such platform. Even if its presence is not useful for your use case. This is simply because the operating system running in the virtualization expects the peripheral to be there. If the peripheral is present on all machines, then it is highly possible that the operating system will expect it to be present even inside your virtual machine. If not, expect problems. This means that you have to provide implementation for more hardware than you really need.
Hosted kernel execution takes a different approach than virtualization. Whether this provides advantages depends on your use case.
Hosted operating system execution
When operating system runs in hosted mode, then there is no hypervisor that pretends some convenient hardware exists. Hosted operating system use their host operating systems as their target hardware. This is possible due to the portable nature of operating systems which hides properties of the hardware behind the curtain of hardware abstraction layer. Instead of talking to hardware directly, you talk to the hardware abstraction layer. Its implementation dictates how these commands are actually executed. In this case, the hardware may as well be just a software API of yet another system.
There are several advantages of such approach which are negation of disadvantages of virtualization:
Your HAL talks directly to the host operating system. There is no (or only very small amount of) translation. Where virtual machine presents a SPI FLASH device which is backed by file on your drive and there has to be SPI FLASH emulation composed of SPI peripheral on chip and SPI device listening to the communication, here the HAL call to write data can simply be executed as file write. No need to pretend that SPI or chip exist. This is much faster than emulating all the bus, chip and implementing all the communication.
You can also completely ignore peripherals / mechanisms you don’t actually need. Does real chip have security mechanism you don’t actually use or mode that has to be disabled? Virtual machine would have to support this peripheral as operating system would certainly try to disable it. In hosted mode, if HAL is not using that mode/peripheral, you can completely ignore it. This means no drivers, no implementation, nothing.
In reality, hosted execution has some disadvantages too.
As we are not talking to any existing hardware, the code that supports hosted execution has to be written from scratch. It is rarely possible to reuse existing drivers as they have completely different semantics.
Sometimes, hosted mode may behave slightly different than real hardware. These differences may be visible to the software running inside the operating system and may cause disruptions.
CMRX is not the only system that provides hosted execution environment. Very similar mechanism is provided by the FreeRTOS, albeit they provide it as a “simulator”. Notorious contender in the “run on top of another operating system” is Linux, which was historically able to run on top of both Windows and Linux itself. The latter is called user-mode Linux.
CMRX hosted environment
CMRX offers Linux/POSIX target as one of architectures you can choose when configuring the kernel. This environment provides no hardware simulation or emulation. It uses services provided by your host machine directly. As CMRX is a microkernel the set of host operating services used by the RTOS is very small:
- In order to implement threading support, Linux own threads are used. Each CMRX thread is backed by one Linux thread.
- To provide scheduling CMRX uses POSIX timer provided by the Linux kernel to generate timing signals.
- mprotect() mechanism is used to implement memory isolation (this feature is not supported as of time writing this article).
Memory Isolation in Hosted Environment
The hosted Linux implementation uses mprotect() to enforce process isolation. While MPU hardware uses region-based protection and mprotect() uses page-based protection, both mechanisms fault when a process attempts unauthorized memory access. From the application perspective, the behavior is identical: isolation violations terminate the offending process.
This means architectural isolation problems—such as a driver incorrectly accessing another process’s memory—will trigger segmentation faults during hosted development rather than requiring hardware debugging to diagnose. Test suites running in CI will catch these violations before hardware is available.
This means that the environment created by CMRX running on top of Linux behaves similarly to the environment created running on real hardware. Applications can still use threads, send and receive notifications and use RPC. All memory accesses are guarded by the memory isolation.
There are some unavoidable subtle differences though. As each CMRX thread is also a Linux thread, there are actually two schedulers deciding if thread is being executed or not. CMRX kernel has definitive word in prohibiting thread from execution, yet if thread is scheduled by CMRX, it is subject of scheduling policy of host Linux kernel. This arrangement affects timing expectations of running algorithms.
Normally, this is not an issue as the hardware Linux is running has performance much higher than a typical low-power device. Thus performance figures obtained by running the software on such hardware are not representative anyway.
Another difference which is not as subtle, but is completely avoidable is that Linux port may actually be a 64-bit environment. This depends on how you configure your build and/or if your host system supports 32-/64-bit multilib. If not, or if you don’t opt for 32-bit build, then the environment inside CMRX will be 64-bit environment. This may cause some problems for applications written with assumption that the environment is 32-bit only.
When to Use Hosted Execution
Frequently asked question here is: Why should I bother? Naturally, almost all users of CMRX are developers targeting real hardware, real microcontrollers. These devices are much cheaper and have orders of magnitude lower power consumption than a Linux machine. Moreover, they don’t even need Linux to run. So why would anyone want to run on top of Linux?
There are several use cases where this may actually be beneficial:
Faster development
Many device functions don’t require specific hardware. MQTT communication, for example, works identically with a hosted network adapter—the application code remains unchanged.
This gives the opportunity to move portion of the development out of the target hardware. Development using developers machine is often much faster than when real hardware is used as you can avoid repetitive flashing of the firmware to the target device and full hardware initialization of the development board after each reflash.
Software development without devboards
Hardware delays kill schedules. Boards arrive late, allocation conflicts arise, or early prototypes fail—derailing software development through no fault of the developers.
Hosted execution with a hardware abstraction layer bypasses these bottlenecks. Development can start as soon as system specifications are partially known, letting you test software ideas without waiting for hardware availability.
Automated testing and verification
Modern software testing is all automated. There are several stages of testing and you usually can’t avoid testing on the real hardware. Still, developers often perform stages on real hardware that could execute in a hosted environment.
Unit tests often don’t need any presence of the hardware as here, independent units of the firmware test in full isolation. In some cases, operating system services may be needed and it may be time-consuming to mock them. Here a hosted environment comes handy as you don’t have to mock the API. It is already there and it behaves the same as it will behave on the real hardware.
Further stages are even better candidates for using hosted environment as partially- or even fully-integrated firmware will most probably use operating system services. Here the ease of using hosted environment and mock just peripherals gives speed advantage.
While writing tests costs the same regardless of environment, executing tests in hosted mode provides the real advantage: speed and scalability.
Test execution in hosted environment makes even running tests much faster and thus cheaper. If you want to run a CI build that performs hardware-based testing, then your CI runner needs a development board attached to it permanently. To run tests, this boards needs some instrumentation connected which isn’t cheap. This one devboard only allows you to run one set of tests. You can’t run two sets of test in parallel. If you want, you have to buy and install another devboard and instrumentation.
The execution speed of tests is usually much slower on the real hardware as you have to flash the test to the board and completely reset it, then clean-up after the test.
In case of hosted test execution, there is no need to have real hardware attached. You may have as many test instances as you want, at any location you want. Test execution is also usually orders of magnitude faster as the CI hardware is orders of magnitude faster than microcontrollers. That means you don’t pay for the testing devboards and you can fit more tests in the same time.
Summary
Hosted execution eliminates hardware dependencies during development and testing. Start development before boards arrive, catch isolation violations in CI, and run parallel test instances at development machine speeds. The result: faster iteration cycles and lower infrastructure costs.