​

Controller

Last verified against code: 2026-03-01

​

Summary

​

Key Concepts

• Controller Loop: The persistent select loop in controllerLoop() that fires on a configurable ticker (default 30s) and on config file changes. Each tick runs the full reconciliation, wisp GC, and order dispatch pipeline. Implemented in cmd/gc/controller.go.
• Config Reload: The debounced mechanism by which filesystem changes to city.toml and pack directories trigger a full config re-parse. An atomic.Bool dirty flag is set by fsnotify; at the top of each tick, if dirty is true, tryReloadConfig() re-parses and validates the config. Rejects workspace name changes (requires controller restart).
• Graceful Shutdown: The two-pass agent termination sequence: (1) send Interrupt (Ctrl-C) to all sessions, (2) wait shutdown_timeout, (3) force-kill survivors via Stop(). Implemented in gracefulStopAll().
• Supervisor vs Standalone Compatibility: gc start now validates an existing city, registers it with the machine-wide supervisor, ensures the supervisor is running, and waits for the city to become active. gc supervisor run is the canonical foreground control loop. Hidden gc start --foreground remains as a compatibility path for the legacy per-city controller.
• Pool Evaluation: The process of running each pool agent’s check shell command, parsing the output as an integer, clamping to [min, max], and building the corresponding agent instances. Pool checks run in parallel via goroutines.
• Nil-Guard Tracker Pattern: Optional subsystems (crash tracker, idle tracker, wisp GC, order dispatcher) follow a nil-means-disabled convention. Callers check if tracker != nil before use. This avoids conditional plumbing and keeps the loop body clean.

​

Architecture

​

Data Flow

gc start --foreground
  │
  ├─ 1. Require an initialized city
  ├─ 2. Fetch remote packs
  ├─ 3. LoadWithIncludes(city.toml)  →  *config.City + Provenance
  ├─ 4. ensureBeadsProvider()        →  start dolt server if bd backend
  ├─ 5. ValidateRigs() + resolve paths
  ├─ 6. initAllRigBeads()           →  per-rig .beads/ databases + routes
  ├─ 7. MaterializeSystemFormulas()  →  embed system formulas as Layer 0
  ├─ 8. ResolveFormulas()            →  symlinks in .beads/formulas/
  ├─ 9. ValidateAgents() + hooks
  ├─10. newSessionProvider()         →  tmux / exec / k8s / subprocess
  ├─11. runController()
  │     ├─ acquireControllerLock()   →  flock LOCK_EX|LOCK_NB
  │     ├─ startControllerSocket()   →  Unix socket for IPC
  │     ├─ build trackers (crash, idle, wisp GC, order)
  │     └─ controllerLoop()
  │           ├─ watchConfigDirs()   →  fsnotify on config + pack dirs
  │           ├─ initial reconciliation
  │           └─ ticker loop:
  │                 ├─ if dirty: tryReloadConfig() + rebuild trackers
  │                 ├─ buildAgents(cfg)  →  evaluate pools in parallel
  │                 ├─ doReconcileAgents()
  │                 ├─ wispGC.runGC()
  │                 └─ orderDispatcher.dispatch()
  │
  └─ shutdown:
        ├─ gracefulStopAll()         →  interrupt → wait → kill
        ├─ record controller.stopped event
        └─ release lock + remove socket + pid

​

Main Loop Detail

1. Dirty check (dirty.Swap(false)): If config files changed since the last tick, tryReloadConfig() re-parses city.toml with includes and patches. On success, all four trackers are rebuilt from the new config. On failure (parse error, validation error, name change), the old config is kept and an error is logged.
2. Agent list build (buildAgents(cfg)): Re-evaluates the desired agent set. For pool agents, evaluatePool() runs check commands in parallel via goroutines (cmd/gc/pool.go:43). Suspended agents and agents in suspended rigs are excluded. Fixed agents are resolved individually. Each agent gets its environment, prompt, hooks, overlay, and session setup expanded.
3. Reconciliation (doReconcileAgents()): Declarative convergence — make running sessions match the desired list. See Health Patrol for the reconciliation state machine, crash loop quarantine, and idle tracking details.
4. Wisp GC (wg.runGC()): If enabled (wisp_gc_interval and wisp_ttl both set), queries closed molecules via bd list and deletes those older than the TTL cutoff.
5. Order dispatch (ad.dispatch()): Evaluates gate conditions for all non-manual orders. See Health Patrol for gate evaluation and dispatch details.

​

Key Types

• controllerLoop() (cmd/gc/controller.go:226): The main loop function. Accepts all dependencies as parameters for testability: config, build function, session provider, reconcile/drain ops, all four trackers, event recorder, and I/O writers.
• runController() (cmd/gc/controller.go:335): The top-level orchestrator. Acquires the flock, opens the Unix socket, builds trackers, enters the loop, and performs graceful shutdown on exit.
• tryReloadConfig() (cmd/gc/controller.go:137): Config reload with validation. Rejects workspace name changes. Returns the new config, provenance, and revision hash on success.
• gracefulStopAll() (cmd/gc/controller.go:169): Two-pass shutdown. Interrupt all sessions, wait shutdown_timeout, then force-kill survivors.
• DaemonConfig (internal/config/config.go:377): Configuration struct holding patrol_interval, max_restarts, restart_window, shutdown_timeout, wisp_gc_interval, wisp_ttl. All durations have *Duration() accessor methods with sensible defaults.

​

Invariants

• Single standalone controller per city: At most one standalone controller runs per city directory. Enforced by flock(LOCK_EX|LOCK_NB) on .gc/controller.lock. A second gc start --foreground fails immediately with “controller already running.”
• Config reload preserves city identity: tryReloadConfig() rejects any reload where workspace.name changes. The city name is locked at startup; changing it requires a controller restart.
• Tracker rebuild is atomic per tick: When config reloads, all four trackers (crash, idle, wisp GC, order) are rebuilt in the same tick before reconciliation runs. No tick ever uses a mix of old and new tracker state.
• Dirty flag is edge-triggered, not level-triggered: The atomic.Bool is set by the fsnotify goroutine and cleared by dirty.Swap(false) at the top of each tick. Multiple filesystem events within a single tick coalesce into a single reload.
• Pool check commands run in parallel: evaluatePool() calls for all pool agents in a single buildAgents() invocation run concurrently via goroutines. Results are processed sequentially after wg.Wait().
• Supervisor-managed and standalone runtimes share reconciliation code: CityRuntime.run() and doReconcileAgents() power both the machine-wide supervisor path and the hidden standalone gc start --foreground path.
• Graceful shutdown sends Interrupt before Stop: gracefulStopAll() always sends Interrupt() to all sessions before sleeping shutdown_timeout and calling Stop() on survivors. Zero timeout skips the grace period entirely.
• Socket cleanup is best-effort: The Unix socket at .gc/controller.sock is removed on startup (stale cleanup) and on shutdown. Crash-orphaned sockets are cleaned up by the next controller start.
• No role names in Go code: The controller operates on resolved config, runtime session names, and provider state. No line of Go references a specific role name.
• SDK self-sufficiency: All controller operations (config watch, reconciliation, pool scaling, order dispatch, wisp GC, graceful shutdown) function with only the controller process running. No user- configured agent role is required for any infrastructure operation.

​

Interactions

​

Code Map

​

Configuration

[daemon]
patrol_interval = "30s"     # reconciliation tick frequency (default: 30s)
max_restarts = 5            # crash loop threshold (default: 5, 0 = unlimited)
restart_window = "1h"       # sliding window for restart counting (default: 1h)
shutdown_timeout = "5s"     # grace period before force-kill (default: 5s)
wisp_gc_interval = "5m"     # wisp GC run frequency (disabled if unset)
wisp_ttl = "24h"            # how long closed wisps survive (disabled if unset)

[session]
provider = ""               # "", "fake", "fail", "subprocess", "exec:<script>", "k8s"

[beads]
provider = "bd"             # "bd" (default), "file", "exec:<script>"

[orders]
skip = ["noisy-order"] # orders to exclude from dispatch
max_timeout = "120s"        # hard cap on per-order timeout

[workspace]
suspended = false           # when true, no agents are started

​

Testing

​

Known Limitations

• No hot-reload for structural changes: Changing workspace.name requires a full controller restart. tryReloadConfig() rejects name changes and keeps the old config.
• Debounce window is global: The 200ms fsnotify debounce applies to all watched directories. A burst of changes across multiple pack dirs produces a single reload, which is correct but may delay detection of a single file change by up to 200ms.
• Pool check commands can stall the tick: Although pool checks run in parallel with each other, the controller tick blocks on wg.Wait() until all checks complete. A hung check command blocks the entire reconciliation cycle. There is no per-check timeout.
• Socket probes are for discovery, not liveness: Per-city controller status uses controller.sock ping responses, and supervisor status uses supervisor.sock. Liveness still comes from flock for singleton control loops and runtime.Provider.IsRunning() for agents.
• Unix socket has no authentication: Any local process with filesystem access to .gc/controller.sock can send “stop” to shut down the controller. File permissions (0o755 on .gc/) provide the only access control.
• Tracker state is in-memory only: Crash tracker history, idle tracker timestamps, and order dispatch state are all lost on controller restart. This is intentional (matches Erlang/OTP supervisor restart semantics) but may surprise operators expecting persistence across restarts.

​

See Also

• Health Patrol — reconciliation state machine, crash loop quarantine, idle tracking, and order dispatch details
• Architecture glossary — authoritative definitions of controller, pool, provider, rig, and other terms used in this doc
• Config struct definitions — DaemonConfig, City, Agent, PoolConfig struct fields and defaults
• Runtime Provider interface — the provider interface that the controller uses for all session operations
• Orders architecture — gate types, dispatch model, and order configuration
• Formulas architecture — formula resolution, layering, and symlink materialization
• Nine Concepts overview — how the controller relates to the five primitives and four derived mechanisms