start-temps-cluster
Start (or restart) a local multi-node Temps cluster using Docker-in-Docker — one control plane + 3 worker nodes, each a privileged DinD container running its own dockerd + `temps agent`, wired with the real multi-host overlay (VXLAN, compute_cidr allocation) via `tools/dev-cluster/` in whichever checkout/worktree you run it from. Invoke when the user says "start the temps cluster", "spin up a multi-node dev cluster", "test this with multiple workers", "bring up worker nodes locally", "docker in dind cluster", or wants to verify cross-node behavior (node targeting, cluster DNS, overlay networking, node join/mTLS) that a single-node `start-temps` server cannot exercise. Distinct from `start-temps` (single-node, native binary, port-slot based) and `start-temps-ee` (single-node EE binary) — this is the only path that actually has more than one node.
gotempsh · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add gotempsh/temps --skill start-temps-clusterAbout this skill
name: start-temps-cluster
description: >
Start (or restart) a local multi-node Temps cluster using Docker-in-Docker —
one control plane + 3 worker nodes, each a privileged DinD container running
its own dockerd + temps agent, wired with the real multi-host overlay
(VXLAN, compute_cidr allocation) via tools/dev-cluster/ in whichever
checkout/worktree you run it from. Invoke when the user says "start the temps
cluster", "spin up a multi-node dev cluster", "test this with multiple
workers", "bring up worker nodes locally", "docker in dind cluster", or wants
to verify cross-node behavior (node targeting, cluster DNS, overlay
networking, node join/mTLS) that a single-node start-temps server cannot
exercise. Distinct from start-temps (single-node, native binary, port-slot
based) and start-temps-ee (single-node EE binary) — this is the only path
that actually has more than one node.
Start a local multi-node Temps cluster (Docker-in-Docker)
Wraps tools/dev-cluster/ — a docker-compose harness that runs a real Temps
control plane plus 3 worker nodes, each a privileged DinD container, with the
actual overlay network (VXLAN, compute_cidr allocation, cluster DNS) wired
up exactly as it would be on real hardware. Use this whenever a fix or feature
needs more than one node to prove anything — single-node start-temps
cannot exercise cross-node scheduling, the overlay, or cluster DNS at all.
Topology
┌──── temps-underlay (10.42.0.0/24, docker bridge, the "VPC") ────┐
│ │
│ postgres 10.42.0.5 (TimescaleDB pg18, internal) │
│ control-plane 10.42.0.10 DinD + `temps serve` │
│ host ports 80 → 80, 443 → 443 (NOT 8080 — see Gotcha below) │
│ │
│ worker-1 10.42.0.21 worker-2 10.42.0.22 worker-3 10.42.0.23│
│ each: privileged DinD + `temps agent`, own dockerd │
└──────────────────────────────────────────────────────────────────┘
Allocator carves /24s from 172.20.0.0/16, one per worker:
worker-1 → 172.20.0.0/24
worker-2 → 172.20.1.0/24
worker-3 → 172.20.2.0/24
Cross-node container traffic flows over a VXLAN tunnel pinned to the
underlay IPs above. Within-node traffic uses the unchanged
`temps-app-network` Docker bridge (same as single-node temps).
The temps source tree is bind-mounted read-write at /workspace inside every
DinD container, and cargo build --bin temps runs inside Linux — no
cross-compiling from macOS. The cluster builds from whichever checkout you
run ./dev-cluster in. cd into the right worktree first if you're testing
a fix that isn't on the checkout's current branch.
Prerequisites
- Docker Desktop (or any Linux Docker host) running —
docker versionmust succeed. - ~3 GB free RAM, ~5 GB free disk (compiled binary cache + 4 worker volumes + TimescaleDB volume).
GeoLite2-City.mmdbat the repo root of the checkout you're running from (<checkout>/GeoLite2-City.mmdb, gitignored) — the proxy plugin refuses to start without it. Download it from MaxMind's GeoLite2 program (free registration required) and place it at the repo root. If you keep multiple worktrees of this repo, copying the file between them is faster than downloading it again.
Only one cluster at a time, across the ENTIRE machine
Unlike start-temps (which allocates a port "slot" per checkout so many
worktrees run side by side), this harness uses fixed container names
(temps-dev-control-plane, temps-dev-worker-1/2/3, temps-dev-postgres)
and fixed host ports (80/443). There is no per-checkout isolation — starting a
second dev-cluster from a different worktree while one is already running
will collide.
Before running up, always check first:
docker ps -a --filter "name=temps-dev-" --format "{{.Names}}\t{{.Status}}"
If containers are already there and belong to someone else's in-progress
work, don't down/reset them without asking — treat it like any other
shared, hard-to-reverse state.
There's also a separate mTLS-hardening variant (docker-compose.harden.yml,
driven by e2e-harden.sh, container prefix temps-harden-*, 2 workers only)
used for testing require_mtls / node-identity hardening specifically. It's a
different compose project name/port set from the base cluster, but still only
one can run at a time on its own.
Quick start
cd <checkout>/tools/dev-cluster
./dev-cluster up # first run: ~5-10 min (compiles the workspace inside Linux)
# subsequent runs: ~10-30s (cargo build no-ops)
up blocks until: control plane's TCP listener is up, all 3 workers have
registered as nodes rows, and every worker has an allocated compute_cidr.
It prints admin credentials and the join-token path on success.
./dev-cluster status # node + overlay state across the cluster
./dev-cluster logs control-plane # or worker-1/worker-2/worker-3/postgres
./dev-cluster shell worker-1 # shell into a worker's DinD host
./dev-cluster restart control-plane # bounce after a binary rebuild
./dev-cluster down # stop, KEEP volumes (fast restart)
./dev-cluster reset # stop + DELETE all volumes (fresh slate, asks to confirm)
Gotcha: the script says port 8080, the cluster actually serves 80/443
docker-compose.yml publishes 80:80 and 443:443 (production-shaped —
TEMPS_ADDRESS=0.0.0.0:80), matching what e2e-harden.sh and every API
recipe below assume (http://localhost/...). But dev-cluster's own up
command waits on /dev/tcp/127.0.0.1/8080 and prints web UI: http://localhost:8080 — that's stale/drifted from the compose file and will
never resolve. Use port 80, not 8080, for every API/browser interaction.
If `./dev-cluster up