Autoscaling on Fly

This is the operational companion to Scaling & Statelessness. It targets a bursty workload — e.g. an exam platform where many students run code in the same few minutes, with quiet stretches in between. New worker machines boot fast by forking their /var/lib/docker volume from a golden snapshot.

Work the tiers in order. Each stands on its own; later tiers add automation, not correctness.

The capacity model

Effective concurrency = running machines × WORKER_MAX_SANDBOXES. A job holds its slot until it exits or hits a clock — so for interactive work, a slot is pinned for up to idleMs / wallTimeMs while a student thinks. Size for the peak simultaneous slots you need, not the total number of students.

Example: 120 students, ~each running once per minute, each run holding a slot ~10s → ~20 simultaneous slots at peak. With WORKER_MAX_SANDBOXES=5 that's 4 machines. Interactive prompts that wait on input hold longer — budget accordingly.

Tier 0 — Baseline capacity

The floor every other tier builds on. Two knobs:

• WORKER_MAX_SANDBOXES — concurrent sandboxes per node. Size the VM to match: each sandbox needs its memory (memoryMb) + CPU share, on top of dockerd. A shared-cpu-2x / 2 GB node comfortably runs a handful of light runs; bump both together for more.
• Machine count — fly scale count N -a code-runner-worker.

# raise per-node concurrency (redeploy picks it up; or set as a secret/env)
#   WORKER_MAX_SANDBOXES lives in deploy/fly/worker/fly.toml [env]
fly scale count 3 -a code-runner-worker          # 3 nodes
fly scale show     -a code-runner-worker          # verify

Tier 1 — Scheduled pre-warm (recommended for known schedules)

Reactive autoscaling always lags: the queue has to build, the autoscaler polls (~15s), then a machine boots. For predictable bursts you already know the start time, so don't wait — pre-warm.

A cron/job (in edalef's own scheduler, or a Fly Machine cron) raises the worker count a few minutes before each exam and lowers it after:

# before the exam window
fly scale count 6 -a code-runner-worker

# after it closes
fly scale count 1 -a code-runner-worker

This needs no Prometheus, no autoscaler app, no code changes — just a deploy token (fly tokens create deploy -a code-runner-worker) and flyctl (or the Machines API) in your scheduler. It's the highest-ROI move for an exam platform.

Tier 2 — Reactive autoscaling with fly-autoscaler

fly-autoscaler runs as its own Fly app and adjusts the worker count from a metric every ~15s. It reads from Prometheus or Temporal — there is no native Redis collector — so the queue depth has to be a scrapable Prometheus metric first.

The in-flight metric (built in)

fly-autoscaler reads Prometheus, so the load has to be a scrapable Prometheus metric. The worker exposes one directly: set WORKER_METRICS_PORT (the Fly worker config uses 9091) and it serves /metrics with:

• code_runner_slots_used — live sandboxes on this node (in-flight work).
• code_runner_slots_max — this node's WORKER_MAX_SANDBOXES.
• code_runner_queue_depth — global LLEN jobs:queue.

Fly scrapes it via the [metrics] block already in deploy/fly/worker/fly.toml:

[metrics]
  port = 9091
  path = "/metrics"

Scale on in-flight = running + queued: sum(code_runner_slots_used) + max(code_runner_queue_depth). This is what makes scale-down safe — the target only falls when sessions actually END (slots free), so a node mid-session is never stopped. (Scaling on queue depth alone would stop a node the moment its queue drained, killing any sessions still running on it.)

The autoscaler app

A ready-to-deploy config ships at deploy/fly/autoscaler/fly.toml (image flyio/fly-autoscaler:0.3). It targets code-runner-worker, queries sum(code_runner_slots_used) + max(code_runner_queue_depth) from the org's managed Prometheus, and runs the Expr expression min(6, max(1, ceil(inflight / 20))) (20 sessions/machine, floor 1, ceiling 6). Edit the org name, query, and bounds for your deployment, then:

fly apps create code-runner-autoscaler -o edalef
# token to read+scale the worker, and to read Prometheus
fly tokens create deploy   -a code-runner-worker | fly secrets set FAS_API_TOKEN=-        -a code-runner-autoscaler
fly tokens create readonly edalef               | fly secrets set FAS_PROMETHEUS_TOKEN=- -a code-runner-autoscaler
fly deploy -c deploy/fly/autoscaler/fly.toml --ha=false -a code-runner-autoscaler

Use FAS_STARTED_MACHINE_COUNT (start/stop a pre-created pool). Size the pool with deploy/fly/worker/provision-pool.sh grow N (it pre-creates snapshot-forked volumes so the new machines boot in seconds) rather than a bare fly scale count, which would give new machines empty volumes that pay the one-time GHCR pull. The worker needs a per-machine ext4 volume (overlay2 can't run on the overlay rootfs), so prefer start/stop of a pre-warmed pool over FAS_CREATED_MACHINE_COUNT create/destroy.

Start/stop of an existing pool machine reuses its populated volume (marker hit) and is near-instant; scale-up reacts in ~15s; keep the floor (max(1, …)) ≥ 1 so the first job of a burst never waits. Pair with a generous MAX_QUEUE_DEPTH (below) so a burst queues — giving the autoscaler something to react to — instead of returning 429.

Tuning for bursts

Recommended path for edalef

1. Tier 0 now: raise WORKER_MAX_SANDBOXES off the starter 2 and set a sane machine count for expected peak.
2. Tier 1 as the primary: a scheduled pre-warm driven by the exam calendar.
3. Tier 2 as the safety net once the queue-depth /metrics endpoint exists.