## https://sploitus.com/exploit?id=66B2B8B7-FAEE-525B-B226-83FB4EB7556C
# hyperlight-overlaybd-poc
**Can the output of a sandboxed job become an immutable, content-addressed
filesystem layer that a *different machine* mounts straight out of a registry —
with no containerd, no Kubernetes, and without ever downloading the layer?**
This is a proof of concept that answers yes, and shows where the seams are.
## The idea
Two pieces of technology that normally live in different worlds:
- **[hyperlight-sandbox](https://github.com/hyperlight-dev/hyperlight-sandbox)**
runs untrusted code inside a micro-VM. The isolation boundary is a
hypervisor, not a namespace. A job gets a read-only `/input`, a writable
`/output`, and nothing else — no network, no host filesystem.
- **[overlaybd](https://github.com/containerd/overlaybd)** presents a stack of
OCI layers as a block device. Layers are content-addressed blobs, the device
is a real `/dev/sdX`, and a *remote* layer is range-read from the registry on
demand rather than pulled and unpacked first.
Put them together and a job's output directory stops being a directory. It
becomes a **layer**: an immutable, content-addressed artifact that a registry
can store, another machine can mount, and nothing downstream can modify.
That is the entire claim. The PoC exists to find out what breaks on the way
there — and something always does.
## Phase 1: one host
```mermaid
%%{init: {"flowchart": {"wrappingWidth": 340}}}%%
flowchart LR
subgraph JOB1["job 1 — produce"]
direction TB
J1["job 1 sandbox (micro-VM)/input (ro) · /output (rw)"]
MA["/mnt/obd-aext4, mounted rw"]
SDX["/dev/sdXTCMU + tcm_loop"]
DA["device Alower: ext4_64 baselayerupper: sparse writable layer (rw)"]
J1 -->|"writes /output"| MA
MA --> SDX
SDX --> DA
end
subgraph JOB2["job 2 — consume"]
direction BT
DB["device Blower 1: ext4_64 baselayerlower 2: job1-layer.commit (ro)"]
SDY["/dev/sdYTCMU + tcm_loop"]
MB["/mnt/obd-bext4, mounted ro,noload"]
J2["job 2 sandbox (micro-VM)/input (ro) · /output (temp)"]
DB --> SDY
SDY --> MB
MB -->|"immutable /input"| J2
end
DA -->|"overlaybd-commit(device torn down first)"| DB
```
Job 1 writes its results to `/output`. That directory is ext4 on an overlaybd
device whose upper layer is sparse and writable. When the job finishes, the
orchestrator unmounts, tears the device down, and commits that upper layer into
a read-only overlaybd layer.
A second device then stacks the committed layer as a *lower* and hands it to
job 2 as `/input`. Job 2 can read every byte job 1 produced and cannot change
any of it — writes to `/input` fail with `PermissionError` inside the guest,
and the mount itself is read-only on the host.
## Phase 2: two hosts and a registry
Phase 1 proves the layer chain works, but both halves share a disk, which is
the easy case. Phase 2 removes the shared disk: the machine that produces the
layer and the machine that consumes it are different hosts whose only common
ground is a registry.
```mermaid
%%{init: {"flowchart": {"wrappingWidth": 340}}}%%
flowchart LR
subgraph LOW["LOW SIDE — AcrPush"]
direction TB
J1["job 1 sandbox"]
MA["/mnt/obd-aext4, mounted rw"]
DA["device Alower: ext4_64upper: sparse writable layer (rw)"]
COMMIT{{"overlaybd-commit"}}
LAYER["job1-layer.commit"]
J1 -->|"/output"| MA
MA --> DA
DA --> COMMIT
COMMIT --> LAYER
end
ACR[("Azure Container Registrycontent-addressed blobs")]
subgraph HIGH["HIGH SIDE — AcrPull"]
direction BT
DB["device Blower 1: ext4_64 (local)lower 2: job layer (streamed)"]
MB["/mnt/obd-bext4, mounted ro,noload"]
J2["job 2 sandbox"]
DB --> MB
MB -->|"/input (ro)"| J2
end
LAYER -->|"oras push (OCI artifact)auth: managed identity"| ACR
ACR -->|"HTTP range reads (registryFs)auth: managed identity"| DB
```
The low side pushes the committed layer to Azure Container Registry as an OCI
artifact. The high side resolves the tag to a blob digest and mounts that layer
**without downloading it** — overlaybd range-reads blocks over HTTP as the
filesystem asks for them.
No Kubernetes, no containerd, no `docker pull`, and no copy of the layer file
anywhere on the high side's disk. The registry is used purely as
content-addressed blob storage. Each VM authenticates with its own managed
identity, so no registry password exists anywhere.
## What the PoC does, step by step
Six stages, each runnable on its own, each ending in a PASS/FAIL banner. They
are built as a ladder: every stage adds exactly one new thing to the one below
it, so when something breaks you already know what introduced it.
### Stage 0 — preflight
Read-only checks with actionable failure messages: kernel modules, configfs,
`/dev/kvm`, the overlaybd daemon, the binaries, and whether the Python SDK
actually imports. The only stage that runs without root.
Requirements that just the sandbox stages need are reported separately, so a
host that cannot run Hyperlight still gets a clean bill of health for the
overlaybd half instead of a wall of red.
### Stage 1 — the sandbox, alone
No overlaybd at all. An input directory round-trips into the guest as
read-only `/input`, the guest writes to `/output`, and the files land on the
host.
**What it proves:** the sandbox half works over ordinary directories, so
anything that breaks later is the block device, not the micro-VM. It also pins
down one surprise: `/output` is wiped at the start of *every* `run()`, which is
why the PoC never points `output_dir` at a mount root.
### Stage 2 — overlaybd, alone
No sandbox. Device A is a baselayer plus a sparse writable layer, mounted rw. A
marker file is written as root, then the device is unmounted, torn down, and
committed. Device B stacks the committed layer and mounts it read-only.
**What it proves:** the write → commit → reuse-as-lower chain holds on its own.
If the marker shows up on device B and writes to device B are refused, the
layer mechanics are sound before a sandbox is ever involved.
### Stage 3 — the whole pipeline
Stages 1 and 2 wired together, in the order that actually works:
1. Create the sparse writable layer. Sparse matters — the default
log-structured layer is append-only and tuned for image conversion, not for
a job creating and deleting files.
2. Launch device A over configfs, mount it rw, and create the job's output
directory *inside* the mount rather than at its root.
3. Construct sandbox 1 **after** the mount exists, and run job 1. It reads
`/input/seed.json` and writes `result.json`, `notes.txt` and
`manifest.json`.
4. Verify the files on the host, then **drop the sandbox object**, sync,
unmount, and tear device A down completely.
5. Commit the writable layer — only once the device is gone.
6. Launch device B with the committed layer as a lower and no upper, mounted
`ro,noload`.
7. Run job 2 with device B as its `/input`. It re-derives job 1's checksum,
prints something derived from it, and **asserts that writing to `/input`
fails**.
**What it proves:** job 2's input is exactly job 1's output, and it is
genuinely immutable. That negative test in step 7 is the one that matters — it
is the difference between "the files copied across" and "the chain is
read-only".
### Stage 4 — push (low side)
Stage 3's job 1, with the local commit file replaced by a registry push. The
layer goes up as an OCI artifact via `oras`, authenticated with the VM's
managed identity.
### Stage 5 — stream (high side)
Job 2 on a different machine, with its `/input` backed by a layer that only
exists in the registry. The high side resolves the tag to a digest itself,
writes an identity-issued token into overlaybd's `cred.json`, and launches a
device whose second lower is remote.
**What it proves:** the layer is portable. Discovery happens on the consuming
host rather than being handed over out of band, and the filesystem is mounted
from a blob that was never downloaded as a file.
## How we know it works
Every stage was re-run end to end on two fresh `Standard_D4s_v5` VMs in
`westus3` on Ubuntu 24.04, built from nothing by `azure-vm.sh up`:
`containerd-overlaybd` 1.0.18 from packages.microsoft.com, hyperlight-sandbox
0.5.0, oras 1.3.0, and a Basic ACR. Stage 4 completes in about 4.0s and stage 5
in about 3.6s.
For phase 2 in particular, these were checked rather than assumed:
- job 2 on the high side reads job 1's `run_id` out of the streamed layer, so
it really consumed what the low side pushed;
- re-running `phase2` produces a **new** layer digest and the high side picks
up the new one, so it is not quietly serving a stale cache;
- no committed layer file exists anywhere on the high side's disk, while
`/opt/overlaybd/registry_cache` does hold the blocks it streamed;
- `/var/log/overlaybd.log` on the high side shows `__open_ro_remote: open file
from remotefs: https:///v2/.../blobs/sha256:...`;
- a push from the high side is refused with `unauthorized ... Action:push`
while its manifest fetch succeeds, so `AcrPull` really is all it has;
- writing to `/input` from inside the sandbox is still refused, so the
read-only chain survives being streamed.
## Running it
On a Linux host that meets the [prerequisites](#prerequisites):
```bash
sudo ./setup.sh # one-time: install the obd-rs package, start the daemon, create .venv
sudo ./run_poc.sh all # stage0 -> stage1 -> stage2 -> stage3, each with a PASS/FAIL banner
```
Individual stages:
| Command | What it runs |
| --- | --- |
| `sudo ./run_poc.sh stage0` | Preflight only: kernel modules, `/dev/kvm`, daemon, binaries, SDK importability. |
| `sudo ./run_poc.sh stage1` | The sandbox alone over plain temp directories: `/input` -> `/output` round trip. |
| `sudo ./run_poc.sh stage2` | overlaybd alone: device -> write -> teardown -> commit -> relaunch ro. |
| `sudo ./run_poc.sh stage3` | The full job 1 -> commit -> job 2 pipeline. |
| `sudo ./run_poc.sh stage4` | Low side: job 1 -> commit -> push the layer to a registry. |
| `sudo ./run_poc.sh stage5` | High side: stream that layer from the registry -> job 2. |
| `sudo ./run_poc.sh clean` | Unmount and remove leftover devices from an interrupted run. |
`stage0` is the only one that runs without root. `stage4` and `stage5` need
`POC_REGISTRY` and run on *different* machines, so they are not part of `all`.
### On Azure, including the cross-machine flow
`azure-vm.sh` creates a registry and two VMs, installs the PoC on both, and
runs the stages:
```bash
az login
./azure-vm.sh up # registry + lo and hi VMs, setup on both, stages 0-3 on lo
./azure-vm.sh phase2 # stage4 on lo (push) then stage5 on hi (stream)
./azure-vm.sh sync # re-upload this directory after editing it locally
./azure-vm.sh down # delete the resource group and the local key
```
Two VMs, because the point of phase 2 is that the producer and the consumer are
different hosts sharing only a registry:
| VM | Role on the registry | What it runs |
| --- | --- | --- |
| `hlobd-poc-lo` | `AcrPush` | stages 0-3, and stage4 (commit + push) |
| `hlobd-poc-hi` | `AcrPull` | stage5 (stream + job 2) |
Each VM gets a system-assigned managed identity and the matching role scoped to
the registry, so **no registry password exists anywhere**.
Defaults to `Standard_D4s_v5` on Ubuntu 24.04 in `westus3`; override with
`POC_AZURE_SIZE`, `POC_AZURE_LOCATION`, `POC_AZURE_RG` and friends. The size
has to be **x86_64 with nested virtualisation**, because stages 1 and 3-5 start
a Hyperlight micro-VM *inside* the VM and so need `/dev/kvm`. The `D*s_v5` and
`D*s_v3` families work; burstable `B` sizes do not.
Everything runs through `az vm run-command` rather than ssh, so it still works
where a policy closes inbound SSH:
```bash
az vm run-command invoke -g hlobd-poc-rg -n hlobd-poc-hi \
--command-id RunShellScript --scripts 'cd /home/azureuser/poc && ./run_poc.sh stage5'
```
Logs land on the VMs at `/var/log/poc-setup.log` and `/var/log/poc-all.log`.
### Prerequisites
`setup.sh` installs or checks all of these, and `preflight.sh` prints an
actionable fix for anything missing:
- **Linux with `/dev/kvm`, x86_64 for the sandbox stages.** Hyperlight is a
micro-VM; without KVM it fails with `No Hypervisor was found for Sandbox`.
Stages 0 and 2 need neither KVM nor x86_64.
- **`target_core_user` and `tcm_loop` kernel modules**, and configfs mounted at
`/sys/kernel/config`. These are what turn an overlaybd config into a
`/dev/sdX`.
- **[obd-rs](https://github.com/juliusl/obd-rs)**, installed from its GitHub
releases. It carries `obdctl`, which this PoC drives the device lifecycle
through, and depends on the `containerd-overlaybd` package from
packages.microsoft.com for the `overlaybd-*` binaries and the daemon unit.
`OBD_RS_VERSION` pins the release; `OBD_RS_PACKAGE` points at a local file
instead.
- **Python >= 3.10** for the `hyperlight-sandbox[wasm,python-guest]` SDK, which
`setup.sh` installs into `./.venv`. The SDK needs nothing else from the host:
its native extension links only libc/libm/libgcc, and the Wasm guest ships
pre-AOT-compiled in the wheel, so there is no daemon, kernel module or
compile step for the sandbox half.
Working files live under `/var/lib/hyperlight-overlaybd-poc` (`POC_WORK_DIR`),
and the mounts are `/mnt/obd-a` and `/mnt/obd-b`. Stage artifacts are
deliberately left behind so a committed layer can be inspected after a run.
## What we learned
### A layer does not have to be an image
The interesting discovery of phase 2 is that **the layer is pushed as an OCI
artifact, not a container image**, and overlaybd streams it anyway.
That works because overlaybd never reads a manifest for a remote layer. Given a
lower with a `digest` and `size`, plus a top-level `repoBlobUrl`, it builds
`/` and reads that blob directly
(`image_file.cpp:__open_ro_remote`). A blob is content-addressed and sits at
`/v2//blobs/` no matter which manifest type references it, so an
`oras push` artifact is exactly as streamable as an image. `oras` stores the
file verbatim, so the blob digest is simply the `sha256` of the commit file.
The high side's device config ends up looking like this — one local lower and
one remote:
```json
{
"repoBlobUrl": "https:///v2//blobs",
"lowers": [
{ "file": "/opt/overlaybd/baselayers/ext4_64" },
{ "digest": "sha256:04f7bec5...", "size": 167936 }
],
"resultFile": "..."
}
```
Auth is the ordinary Docker registry v2 bearer flow. overlaybd reads
`/opt/overlaybd/cred.json`, answers the registry's `www-authenticate` challenge
with HTTP basic, and exchanges that for a bearer token
(`registryfs_v2.cpp:authenticate`). ACR accepts an identity-issued refresh
token as the password when the username is the all-zero GUID
`00000000-0000-0000-0000-000000000000`, so stage 5 does `az login --identity`,
then `az acr login --expose-token`, and writes the result into `cred.json` at
mode `600`.
Things worth knowing before leaning on any of this:
- **`AcrPush` also grants pull.** There is no push-only built-in role, so the
lo/hi split models the intent rather than enforcing a one-way boundary. The
reverse direction *is* enforced.
- **The refresh token expires** in hours, which is why stage 5 mints a fresh
one on every run rather than caching it.
- **`repoBlobUrl` is a single top-level field**, so all remote lowers in one
device must come from the same repository.
- **"Streaming" means fetch-on-demand, not zero-copy.** overlaybd caches the
blocks it reads, so a small layer ends up fully cached; what it avoids is
downloading and unpacking the layer *before* the mount.
- **The baselayer stays local.** Only the committed job layer streams.
### Ordering is most of the difficulty
Getting any of these wrong produces confusing failures, so all of them are
encoded in the scripts and exercised by a passing run:
1. **Mount before constructing a sandbox, and drop the sandbox before
unmounting.** A sandbox captures its preopens as cap-std directory file
descriptors at construction time. `umount` returns `target is busy` while
the sandbox object is alive, and succeeds immediately after it is dropped.
2. **Never point `output_dir` at the mount root.** `/output` is wiped at the
start of *every* run, so the mount root would lose `lost+found` and anything
else living there.
3. **sync, then umount, then tear the device down, then commit.** Committing a
live or dirty device captures a torn filesystem, and `overlaybd-commit` will
happily run against a device the daemon still has open.
4. **Mount the consuming side `ro,noload`.** All of its layers are read-only,
so ext4 cannot replay a journal.
5. **configfs teardown is order-sensitive**: LUN symlink → `lun_0` → `tpgt_1` →
`naa.*` → backstore → HBA. Cleanup is idempotent, registered with `atexit`
and on SIGINT/SIGTERM/SIGHUP, and also sweeps stale entries by naming
convention so a run killed with `SIGKILL` can be cleaned up with
`run_poc.sh clean` rather than manual surgery.
6. **Check `resultFile` after enabling a device.** overlaybd reports launch
failures there rather than through the configfs write.
7. **Never hardcode `/dev/sdX`.** `tcm_loop` *recycles* the SCSI address
triple, so the resolver waits until the node's `dev_t` matches sysfs and the
node is actually readable, and teardown waits for the device to disappear.
Without those waits, back-to-back runs intermittently fail with
`mount: /dev/sdb is not a valid block device`.
8. **`overlaybd-create` refuses to overwrite.** Its outputs are opened
`O_EXCL|O_CREAT`, so each stage works in a fresh directory.
### The guest environment is smaller than you expect
The packaged Wasm Python guest is Python 3.14 built for WASI with a trimmed
stdlib and a WASI preopen filesystem. Probing it directly:
- Available: `json`, `os`, `time`, `math`, `random`. Not available (all
`ModuleNotFoundError`): `datetime`, `hashlib`, `base64`, `textwrap`,
`traceback`, `glob`.
- Reads and writes by path work. **Directory enumeration does not**:
`os.listdir('/input')` raises `FileNotFoundError` even for a valid preopen,
and `os.path.exists()` returns `False` for a file that `open()` then reads
successfully.
- Writing to `/input` fails with `PermissionError`, which is the negative test
job 2 relies on.
The missing `readdir` is the one that shaped the design. Rather than treat it
as a dead end, the PoC covers enumeration from both sides: job 1 writes a
`manifest.json` listing what it produced, so the **layer is self-describing**
and job 2 enumerates from that; and the host separately lists the mount and
asserts it contains exactly job 1's files.
The manifest turned out to be the better answer anyway — it is much closer to
how a real layer chain would carry its metadata. If a future SDK release
implements `fd_readdir` for preopens, job 2 can drop it and list `/input`
directly; nothing else in the pipeline changes.
## Limits of this PoC
**Everything runs as root.** configfs and `mount(8)` need it, and `/dev/kvm` is
`root:kvm` on most distros, so the orchestrator does not drop privileges around
the sandbox step. That is fine for a PoC and is **not** how this should be
deployed: the sandbox step only needs `/dev/kvm` plus two directories, so a
real implementation would keep the device and mount work in a privileged helper
and run the job itself unprivileged in the `kvm` group.
**The sandbox stages are x86_64-only.** This is a Hyperlight limitation rather
than a packaging gap: `hyperlight-sandbox-backend-wasm` publishes
`manylinux_2_34_x86_64` and `win_amd64` wheels only;
`hyperlight-sandbox-python-guest` is tagged `py3-none-any` but its
`python-sandbox.aot` is an x86-64 ELF object, so that wheel is only nominally
portable; and there is no source fallback, because `hyperlight-host` does not
compile for aarch64. Preflight reports this as a stage-specific failure rather
than a blocking one, so an aarch64 host still validates the overlaybd half.
**No trust boundary.** `AcrPush` grants pull as well, and both VMs sit on the
same VNet with public registry access. The two-sided split models the shape of
a real deployment; it does not enforce one.
Also out of scope: containerd or accelerated-container-image integration, zfile
compression, the live-snapshot API, multi-tenancy, and network permissions in
the sandbox. Single pipeline run, clarity over robustness — except for the
cleanup trap, which is mandatory because TCMU and configfs leftovers otherwise
need manual surgery.
## Troubleshooting
- `No Hypervisor was found for Sandbox` — `/dev/kvm` is missing or not
accessible. Run as root, or `sudo usermod -aG kvm $USER`.
- `No matching distribution found for hyperlight-sandbox-backend-wasm` — you
are not on x86_64.
- `overlaybd-tcmu.service: Main process exited, code=exited, status=203/EXEC` —
the PMC unit points at `/opt/overlaybd/bin/overlaybd-tcmu` but the package
installed the binary under `/usr/bin/overlaybd/`. The obd-rs package ships
the drop-in and symlinks that reconcile the two; `sudo ./setup.sh` installs
it.
- `resultFile` is not `success` — check `tail /var/log/overlaybd.log`; usually
a bad path in the device JSON or a layer file the daemon cannot open.
- `obdctl: command not found` — the obd-rs package is not installed. Re-run
`sudo ./setup.sh`.
- `obdctl preflight` reports `overlaybd-tcmu running` as `[FAIL]` — the daemon
is not up. `sudo systemctl start overlaybd-tcmu`, or without systemd,
`sudo /usr/share/obd-rs/shell/obd-setup.sh --daemon auto`.
- `umount: target is busy` — something still holds the mount; in this PoC that
is a sandbox object that has not been dropped.
- Leftover devices after a `SIGKILL` — `sudo ./run_poc.sh clean`.
- `ensurepip is not available` — `setup.sh` installs `python3-venv` for you on
apt hosts; on other distros install the venv package and re-run.
- `failed to get user home directory: $HOME is not defined` — something is
calling `oras login` under `az vm run-command`, which has no `$HOME`. The
PoC passes credentials per command over stdin instead.
- `unauthorized: authentication required ... Action:push` on the high side —
working as intended: that VM holds `AcrPull` only.
- stage5 fails at `resultFile` with an auth error — the ACR refresh token has
expired. Re-run stage5; it mints a fresh one.
- `empty repoBlobUrl for remote layer` — a lower was given a `digest` without a
top-level `repoBlobUrl` in the device config.
## What is in the repo
| File | Purpose |
| --- | --- |
| `setup.sh` | One-time host setup. Idempotent; ends by running `preflight.sh`. |
| `preflight.sh` | Stage 0. Read-only checks with actionable failure messages. |
| `run_poc.sh` | Entry point: `stage0`..`stage5` / `all` / `clean`. |
| `azure-vm.sh` | Create the registry and both VMs, run the stages, tear it all down. |
| `poc_common.py` | Paths, logging, `check()` assertions, the PASS/FAIL banner. |
| `poc_registry.py` | Registry glue: managed-identity token, `cred.json`, oras push/resolve. |
| `overlaybd_device.py` | Adapter over `obdctl`: the device lifecycle and the cleanup trap. |
| `stage1_sandbox.py` | Stage 1: sandbox only. |
| `stage2_overlaybd.py` | Stage 2: overlaybd only. |
| `stage3_pipeline.py` | Stage 3: the full pipeline. |
| `stage4_push.py` | Stage 4: low side. Job 1, commit, push the layer. |
| `stage5_stream.py` | Stage 5: high side. Stream that layer back and run job 2. |
| `guest_script_1.py` | Job 1's program. Runs **inside** the sandbox, not on the host. |
| `guest_script_2.py` | Job 2's program. Runs **inside** the sandbox, not on the host. |
## License
Licensed under the [Apache License, Version 2.0](LICENSE).