Zygote density tier

The zygote tier is an optional, opt-in execution backend for the worker. Instead of one hardened container per job, it routes each job to a long-lived, per-(language, version) warm parent pool container that fork()s one hardened child per job. The children share the parent's pre-imported library pages copy-on-write, so a node fits far more concurrent "heavy" interpreted sandboxes in the same RAM.

It sits alongside the default DockerSocketRunner behind the same Runner interface — a density-vs-isolation tier, not a replacement. It is OFF by default (ZYGOTE_ENABLED); dev, CI, and any worker that doesn't flip the flag behave exactly like a plain per-job-container worker.

Why it exists

RAM is the hard ceiling on how many sandboxes a worker node can hold. A heavy Python sandbox (numpy + pandas + matplotlib imported) is ~110 MB resident, and a perf-2x / 4 GB worker OOMs at ~30 concurrent. But that 110 MB decomposes into ~70 MB of identical library/interpreter pages + ~40 MB of unique user working set. The shared 70 MB is the lever.

A zygote parent imports the science stack once; every forked child shares those ~70 MB physically via copy-on-write and pays only for its own ~40 MB working set. The measured result:

~2.7× density on the same 2 CPU / 4 GB box. The follow-up question — does that win survive making it safe for untrusted code? — was answered: per-child hardening is free. Layering distinct UID, no-new-privs, mount/PID/network namespaces, and a per-child cgroup-v2 sub-cgroup on top of the fork left the curve flat (full hardening: 81 concurrent, ~41.6 MB marginal). Hardening doesn't touch the CoW-shared import base, so it consumes effectively zero RAM.

What runs where

The worker (when zygote is enabled) wraps two runners in a TieredRunner and routes each job by a single, manifest-driven predicate — there is no language-name branching. A job goes to the zygote tier only when both:

1. a ZygoteRunner is configured (ZYGOTE_ENABLED=true), and
2. the job's resolved manifest is zygote-eligible — it declares a non-empty preimport array.

When the zygote runner is nil (the default) every job goes to Docker, so an unset ZYGOTE_ENABLED worker is byte-for-byte the old behaviour for all four languages.

Opting a language in

The tier decision is entirely in the manifest. To make a language zygote-eligible, add a non-empty preimport array to its languages/<lang>-<version>/manifest.json — the modules the parent should import once at warm time:

{
  "language": "python",
  "version": "3.12",
  "preimport": ["numpy", "pandas", "scipy", "sklearn", "matplotlib.pyplot"]
}

Remove the preimport key (or leave it empty) and the language routes to the Docker tier. This only makes sense for interpreted, heavy-import languages — compiled languages (Rust) and no-import workloads (SQLite) get no CoW benefit, and the zygote tier does not support a compile step. The image must also ship a zygote agent for that language (Python ships zygote_agent.py).

Security posture

The zygote tier deliberately inverts the per-sandbox privilege posture for the pool container, and that is acceptable only under a specific deployment + threat model.

Why the pool container is privileged

To create namespaces and cgroup leaves for its children, the agent needs CAP_SYS_ADMIN + CAP_SETUID/CAP_SETGID and a writable cgroupfs — i.e. the pool container runs privileged with host cgroups (--cgroupns=host). This is the opposite of the per-job Docker sandbox, which runs --cap-drop ALL.

This is acceptable only because on Fly the worker runs its own dockerd inside a Fly Machine, which is a Firecracker microVM — the microVM is the real host boundary. code-runner's threat model is host-escape-only: sandbox→sandbox is discounted (sessions are ephemeral; no secret lives in a sibling). A privileged pool container can at worst escape into the worker's inner dockerd / Machine — never onto the Fly host. So in-container caps are not the boundary that matters. This is also why the tier is gated behind ZYGOTE_ENABLED, default OFF: dev and CI keep the strong per-sandbox posture, and the privileged pool only ever runs where Firecracker backs it.

The per-job Docker-tier hardening is unchanged — R / Rust / SQLite (and any zygote fallback) still run --cap-drop ALL, no-new-privileges, read-only root, network=none, seccomp. Only the warm-pool parent is privileged; the per-child it forks is re-hardened by the agent.

Per-child hardening

For each job the agent double-forks and hardens the child before user code runs: a private PID namespace (sees only itself in /proc), a distinct UID (UID_BASE + n), PR_SET_NO_NEW_PRIVS, a private network namespace (no network even though the pool container has one), a private mount namespace with its own /tmp tmpfs and remounted /proc, a per-child cgroup-v2 leaf (memory.max + pids.max from the job's own limits), and an fd scrub (every fd > 2 is closed before user code). Job files ride into the child via fork() and are materialized into the child's own private /tmp — siblings can never read them. A child cannot read a sibling's /proc/<pid>/mem, /tmp, or FDs.

Resilience: fallback to Docker

Enabling the zygote tier cannot break Python. If a zygote-eligible job's Create fails for any reason — the warm pool won't start, the agent dial fails, the image lacks the agent — the TieredRunner transparently falls back to the always-present hardened Docker tier rather than failing the job. The fallback is logged loudly (WARN: zygote Create failed; falling back to Docker tier) and counted via code_runner.zygote.fallback.count. A degraded tier silently serving everything via Docker is therefore observable, never invisible — watch that counter. Only Create-time failures fall back; a cancelled context is not treated as a zygote fault.

Operations

Config knobs

All zygote knobs are only meaningful (and only validated) when ZYGOTE_ENABLED=true; a Docker-only worker never trips on them.

Pool lifecycle

• Warm (lazy). A parent is created on the first job for a (language, version) and kept warm.
• Idle reap. After ZYGOTE_POOL_IDLE_MS with no in-flight jobs, the parent is torn down to reclaim RAM (code_runner.zygote.parent.reap.count).
• Dead-parent detect + respawn. If the agent/container dies it is detected on the next dial, dropped, and respawned on the next request (code_runner.zygote.parent.respawn.count). In-flight jobs that lose their connection fail cleanly and the slot is released — no leak.
• Slot accounting is unchanged. The worker's existing semaphore (WORKER_MAX_SANDBOXES) still bounds concurrency; the pool is orthogonal.

On Fly: no extra Machine flag

The pool container needs to run privileged with host cgroups. On the Fly reference deploy this needs no extra Machine flag: the worker already runs its own dockerd inside the Machine (it is the dind daemon, already privileged), so launching a privileged pool container needs no [vm] change and no extra cap. See Deploy to Fly.io.

Observability

All instruments emit through the same OTel meter scope as the rest of the worker (OTEL_EXPORTER_OTLP_ENDPOINT is the on switch). job_id is never a metric attribute; only the low-cardinality language / version are attached.

Verifying zygote is actually engaged

ZYGOTE_ENABLED=true does not by itself prove jobs run on the warm pool. Confirm with three signals: code_runner.zygote.pool.warm_parents > 0 for the expected (language, version); code_runner.zygote.fallback.count is flat (a rising rate = zygote degraded, Python silently served by Docker); and the agent's startup log line appears while you do not see the worker WARN: zygote Create failed. On Fly/Linux with --cgroupns=host you should also not see the one-time per-child cgroup enforcement unavailable WARN — seeing it on Fly means per-child memory.max/pids.max are not being enforced.

R status

R runs on the Docker tier for now — it does NOT run on the zygote tier in v1.1. R's manifest.json has no preimport, so it is not zygote-eligible and the TieredRunner routes it to the DockerSocketRunner — R works in prod exactly as today (per-job hardened container), just without the warm-pool CoW tier. R cannot call unshare/mount/prctl without native code, and evaluating embedded R in a double-forked, UID-dropped, freshly-namespaced child is fragile (its signal handlers, allocator, and longjmp error handling are suspect post-fork). The native C helper already compiles cleanly in the R image, de-risking a future revisit, but the end-to-end relay self-test is not yet done.

When to enable

• You run Fly workers (Firecracker microVM host boundary) — this is a hard requirement.
• Your workload is heavy interpreted (Python with the science stack), where RAM is the ceiling and most of that RAM is shared import pages.
• You want ~2.7× more concurrent sandboxes per node without a hardware change.

Keep it off for dev, CI, non-Fly deploys, or compiled/no-import workloads — there's no density win and the per-sandbox posture stays stronger.

See also