■ erXos · operating system

What if every Erlang process were an operating-system process?

erXos is an operating system built on that idea. Its drivers, file systems and services are Erlang processes, supervised and isolated by a small capability microkernel, and its shell feels like UNIX.

             __  __
   ___  _ __ \ \/ /  ___   ___
  / _ \| '__| \  /  / _ \ / __|
 |  __/| |    /  \ | (_) |\__ \
  \___||_|   /_/\_\ \___/ |___/

erXos
development build: trusted code only
Erlang/OTP 29 [erts-17.1.1] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]

rc: the command shell; erl starts the Erlang shell
% # The command shell runs rc's language, written in Erlang.
% ls /
NOTICES
bin
lib
manifest
root
% echo hello from erXos > /tmp/hi
% cat /tmp/hi
hello from erXos
% # How many processes run? Each is an Erlang process.
% ps | wc -l
      62
% erl
Eshell V17.1.1 (type help(). for help)

% Crash a process that divides by zero, and catch it
% with a monitor. The shell carries on.
1> {_, R} = spawn_monitor(fun() -> 1 / list_to_integer("0") end).
=ERROR REPORT==== 1-Jan-1970::00:00:02.651025 ===
Error in process <0.150.0> with exit value:
{badarith,[{erlang,'/',[1,0],[{error_info,#{module => erl_erts_errors}}]}]}

{<0.150.0>,#Ref<0.3910384898.1.60465>}
2> receive {'DOWN', R, _, _, {E, _}} -> {crashed, E} end.
{crashed,badarith}

% A ring of N processes passes a token round it Laps times.
3> Ring = fun(N, Laps) ->
..     Last = lists:foldl(fun(_, Next) ->
..         spawn(fun Loop() -> receive 0 -> Next ! 0;
..                   T -> Next ! T - 1, Loop() end end)
..       end, self(), lists:seq(1, N)),
..     Alive = erlang:system_info(process_count),
..     Last ! N * Laps,
..     (fun Wait() -> receive 0 -> ok;
..                    T -> Last ! T - 1, Wait() end end)(),
..     {Alive, N * Laps}
..   end.
#Fun<erl_eval.41.130099583>

% 5,000 processes alive at once, 100,000 messages.
4> Ring(5000, 20).
{5061,100000}
erXos Recorded session
01 / The idea

One machine, one Erlang system.

Drivers, file systems and system services are Erlang processes, supervised like any other. When one crashes, its supervisor restarts it.

The machine stays open. From an Erlang shell you inspect, trace and change it, all the way down to kernel objects. That is the goal, and the design is built around it: a kernel that knows nothing of Erlang, and an Erlang system that runs the machine.

In erXos, Erlang's process is the operating system's process: one process space, one message system and one view of identity, supervision and inspection span every runtime.

G1

Secure by construction

Authority is a capability someone handed you. Knowing a pid, a path or a node name grants nothing.

G2

Millions of processes

Erlang processes are cheap to create and cheap to switch. erXos keeps them cheap.

G3

Crashes are routine

A driver wedges its device. It fails alone, its supervisor restarts it, and the machine carries on.

G4

A live machine

Upgrade a driver in place. Follow any process to the line of source it is executing.

G5

One kernel, many runtimes

The kernel hands out CPU time. ERTS comes first. C programs share the machine on the same terms.

G6

Cross-platform

AArch64 on QEMU first, then the Raspberry Pi 4 and 5. Other architectures are ports, not redesigns.

02 / Principles

Four rules shape every change.

P1

Erlang wherever practical

Erlang runs the drivers, the file systems, the network stack and the shell. C is for the kernel, device shims and per-byte work. Practical is the key word.

P2

Beauty and simplicity

A few concepts, used everywhere. A little speed is a fair price for a lot of simplicity.

P3

Declarative where it simplifies

One manifest says what the machine runs. The build checks it. Where a description is simpler than code, it replaces the code.

P4

UNIX look and feel

Shell, files, pipes, ps. FreeBSD programs, recompiled from source. UNIX is how the system looks, not what it is built on.

03 / Architecture

A small kernel. Everything else is a process.

The kernel knows threads, memory, capabilities and interrupts. It knows nothing of Erlang. Everything else runs in user mode, the runtimes included.

A domain is an address space with its own runtime. It is also an Erlang node. Processes in different domains link, monitor and message each other as Erlang processes always have. A crashed domain looks like a dead node.

Surface
UNIXshell · files · pipes · terminals
Domains
System domainERTS · domain service · console · services
Network domainERTS · network drivers · TCP/IP stack
Storage domainERTS · storage drivers · file systems
DomainERTS · your Erlang application
DomainERTS · another application, isolated from the first
Domainnative programs on musl
Interface
liberxthe kernel interface, one ABI for every runtime
user mode ↑ · ↓ kernel mode
Kernel
Capability microkernelthreads · memory · capabilities · interrupts · CPU grants
Machine
AArch64QEMU virt · Raspberry Pi 4 and 5 to follow
A1

Two-level scheduling

The kernel grants CPU time. Each runtime schedules its own processes within that grant, without a kernel call.

A2

A budget for everything

Every kernel allocation is charged to a budget. Delete the budget, and everything charged to it goes too.

A3

Private namespaces

Straight from Plan 9. There is no global root, so a path is the visible face of a capability.

erXos stands on the shoulders of giants. Erlang/OTP gave it processes and supervision; L4 and Fiasco, the kernel; Plan 9, namespaces and files; QNX, UNIX on a microkernel; Genera, a whole machine open to inspection. What is new is how the pieces meet.

04 / Status

Early, but it boots.

Phase 1: erXos runs on a stand-in kernel, a near-stock Fiasco behind erXos's own interface, which erXos's own kernel replaces in Phase 2.

Running

ONQEMU virt

AArch64, on 1 to 8 CPUs.

ONRaspberry Pi 4

Serial console, SD card and Ethernet, on the board, with live quotes and SSH over it. tools/erx netboot stages its boot files for a TFTP boot from a Mac. The kiosk's dashboard runs full screen at 1920x1080.

ONIn the browser

The console above, on an AArch64 emulator.

ONDomains

Storage, network, display and sessions each run their own ERTS, in user mode; a fault's end line names where it struck.

ONDrivers in Erlang

Console, virtio block, net and gpu, SD, GENET and the Pi's framebuffer.

ONFiles

Per-domain namespaces, /tmp, littlefs on /data, EROFS images, a system partition whose blocks are checked against a dm-verity tree when a guest reads them, and each service's own tree at /data, /apps and /machine.

ONNetwork

A TCP/IP stack in Erlang: DHCP, ping, host, gen_tcp connections out and in, and SSH login for root in development builds, whose first boot imports the card's authorized_keys. OTP's ssl and inets are in every domain release with a trust store; HTTPS fetches are tested on the host against loopback servers.

ONShell

rc, written in Erlang, with job control, a tty for each terminal master, reboot and halt.

ONInspector

Processes, kernel objects and the domains a session's scope covers at /proc; every session also sees each CPU's load and the clock, which QEMU virt sets from its PL031 real-time clock; /proc/time shows who set it.

ONNetwork time

ntpd sets the clock at boot from an NTP server (the Pi 4 demo image uses pool.ntp.org; off elsewhere until enabled), and ntpctl steps the clock or shows the status.

ONChecked

OTP's own test suites, against stock OTP.

Next

NEXTerXos's kernel

Begun as host-tested C cores.

NEXTPrograms

Pipes are built and session processes start programs, which can kill and probe themselves, with FreeBSD's errno and signal numbers in musl; a program's file tree (/bin, /dev, /etc) is written. The shell runs programs alone and in pipelines with Erlang commands, Ctrl-C ends them and leaves $status as sigint; files take 64-bit offsets, and a program started with a window gets a /dev of its own (draw, kbd, mouse, tested on the host over stand-ins, not yet in a QEMU cell); a read that waits for a key or a mouse event is held by a keeper, so it does not hold up the program's other files, and /dev/tty answers ENXIO as a file. Terminal input for programs, sockets and POSIX-lite come next.

NEXTDisplay

Windows work on QEMU virt and take its keyboard and tablet, and the Pi 4 has a display domain. Clients draw with Plan 9's draw protocol, share a /clipboard, and LVGL draws through erx-lvgl in a session, which the display suite checks on QEMU. The kiosk's dashboard app is drawn and checked on the host, and its live quotes source is host-tested. Inter fonts render to subfonts with FreeType, and a rio-style window policy is a host-tested core; neither has a QEMU cell yet.

NEXTUSB, Bluetooth and WiFi

Host-tested cores only: hub and port states, descriptor and HID decoders, a Bluetooth HCI codec and controller process, and WiFi's join machine, SAE exchange and EAPOL-Key codec. No working drivers yet.

05 / Correctness

Checked against answers the code did not write.

A test only proves what it checks. So erXos compares its code with stock OTP, a bounded model checker and a weak-memory simulator, and each check says what it covered.

Most checks come with a deliberately broken twin or a seeded defect that they must catch, so a check that cannot fail gets noticed. These checks have limits. A model checker checks a model, not machine code. A litmus test checks a small program, not the compiled lane. A race detector finds data races but cannot show that a protocol is right.

C1

Stock OTP as the oracle

OTP 29.1.2's emulator, kernel, stdlib and compiler test suites run on erXos and on stock OTP, and the results are compared case by case. erXos's C library also runs musl's own tests.

C2

Weak memory, under herd7

Litmus tests of the lane's wake handshake and ring positions run under Arm's AArch64 model and RC11. With its fences the protocol must never reach the bad outcome. Without them it must.

C3

TLA+ models

TLC checks models of the lane, connections and links, each with seeded defects it must catch. Models for the host link, updates and the scheduler are planned and not written.

C4

CBMC

Bounded model checking of the budget arithmetic and the capability table's lookup, rights and slot closing, for every input within stated bounds.

C5

Fuzzing and sanitizers

The rule is a fuzz target for every parser of untrusted input, and the known gaps are tracked. Host tests of the C code run under AddressSanitizer and UndefinedBehaviorSanitizer, and concurrency tests under ThreadSanitizer.

C6

Rules the build enforces

Rules of the form “only X may do Y” fail the build on a new site. The stack depth of the C17 cores is computed from their call graph, and two clean builds of the production boot image must match byte for byte.

06 / Try it

The real thing, in your browser.

The console at the top of this page runs erXos itself. Press Boot erXos and your browser downloads the same boot image that QEMU runs, then starts it on v64, an AArch64 machine emulated in WebAssembly. The kernel, the system domain and the command shell all come up inside the page, in a few seconds.

Nothing runs on a server, and nothing you type leaves your machine. Reset starts the machine again from the same image.

Kernel
Capability microkernel
System
Erlang/OTP
Surface
UNIX
Target
AArch64 · QEMU virt
Stage
Phase 1 · early
T1

The command shell first

The console opens in the command shell, rc's language written in Erlang. Write a file to the RAM file system and read it back: echo hello > /tmp/hi, then cat /tmp/hi.

T2

Then Erlang

erl starts the Erlang shell. q(). ends it and brings back the % prompt.

T3

Look around

ls / lists the root of the namespace. ps lists the processes, which are Erlang processes. top shows them live under uptime, memory and run queue; Ctrl-C ends it. dmesg prints the system log from boot on.

T4

On the network

ifconfig shows the network device and the address it got by DHCP. ping -c 3 8.8.8.8 sends real ICMP through erXos's Erlang stack and driver. A small router emulated in this page answers every echo itself: no packet leaves your browser.

07 / Download

Run it on your own machine.

D1 Board

Raspberry Pi 4

Download card .img.xz · 37.9 MB

Write it to a microSD card with dd or Raspberry Pi Imager.

Serial console at 115200 baud; needs a bootloader from 2022-12-01 or later.

# diskN: the card in diskutil list; everything on it is erased
diskutil unmountDisk /dev/diskN
xz -dc sdcard-e593e5c11e4cf3a0.img.xz \
  | sudo dd of=/dev/rdiskN bs=4m
diskutil eject /dev/diskN
SHA-256
499c4da7f9118eeba3dcaa72898be7ab5713a40eaf595bdb77176808405b582a
Source
ff625903e433 · build record
D2 Emulator

QEMU virt

Download image .boot.gz · 43.6 MB

The image the console above runs. Boot it with QEMU; Ctrl-A X quits.

qemu-system-aarch64 -machine virt,gic-version=3 \
  -cpu cortex-a76 -m 256 -nographic -nic none \
  -global virtio-mmio.force-legacy=false \
  -netdev user,id=net0 \
  -device virtio-net-device,netdev=net0,bus=virtio-mmio-bus.23 \
  -kernel image-b32b5c4d256bf3ca.boot.gz
SHA-256
0b9557d71373022d39a982ab7cbf68a35b88e894b0e1f049d9ff860d2c5bdcc0
Source
ff625903e433

Read the release notes first: 0.1 is a development preview that runs trusted code only. The console above runs on v64: license, third-party code.