​

Machine-Wide Supervisor

​

Summary

​

Motivation

​

Pain today

1. Operational overhead of N controllers. A developer running 3 cities (one per project) has 3 controller processes, 3 API ports, 3 sets of tmux sessions to monitor. There is no single pane of glass. gc status only shows the current city. Checking all cities requires cd-ing into each directory.
2. No cross-city visibility. Mission Control (the dashboard) connects to one API URL. Monitoring multiple cities requires multiple browser tabs pointed at different ports, or re-launching with different --api flags.
3. Resource waste. Each controller runs its own fsnotify watcher, reconciliation goroutine, and HTTP listener. On machines with 5+ cities this is 5+ processes doing the same structural work independently.
4. Future multi-tenancy. Hosting Gas City as a service (multiple customers on the same box) requires tenant isolation, resource limits, and a single management plane. The current architecture has no concept of “who owns this city.”

​

Erlang/OTP parallel

​

Design principles alignment

• NDI (Nondeterministic Idempotence): A machine-wide supervisor converges all cities to their desired state on every tick, regardless of which cities were added, removed, or crashed since the last tick. The registry file is the desired state; running city supervisors are the actual state.
• SDK self-sufficiency: The machine supervisor is pure infrastructure. It does not require any user-configured agent role to function.
• Bitter Lesson: A unified API surface gets MORE useful as models improve — agents can query cross-city state, external tools can monitor all cities, dashboards can show a fleet view.

​

Guide-Level Explanation

​

Registering cities

# From inside a city directory
cd ~/bright-lights
gc register          # adds this city to machine supervisor

cd ~/dark-alleys
gc register          # adds another city

gc cities            # list registered cities
# NAME            PATH                     STATUS
# bright-lights   /home/user/bright-lights running
# dark-alleys     /home/user/dark-alleys   running

​

Starting the machine supervisor

# Start the machine supervisor (manages all registered cities)
gc supervisor start

# Register the current city and ensure it is running
gc register

​

Stopping

gc supervisor stop    # stops ALL cities, then the supervisor
gc unregister         # removes current city from supervisor
gc stop               # unregisters and stops the current city

​

API access

# List all cities
curl http://localhost:8080/v0/cities

# City-scoped requests (new URL pattern)
curl http://localhost:8080/v0/city/bright-lights/agents
curl http://localhost:8080/v0/city/bright-lights/agent/worker/output
curl http://localhost:8080/v0/city/dark-alleys/beads?status=open

# Cross-city queries
curl http://localhost:8080/v0/agents              # all agents, all cities
curl http://localhost:8080/v0/events/stream        # global event stream

​

Backward compatibility

​

Dashboard

gc dashboard --api http://localhost:8080
# Queries GET /v0/cities, picks the first one,
# prefixes all API calls with /v0/city/{name}/...
# City switcher is follow-on work.

​

Config

[supervisor]
# API port for the machine-wide supervisor.
# City-level [api] sections are ignored when running under the supervisor.
port = 8080
bind = "127.0.0.1"

# Patrol interval for the supervisor's own reconciliation
# (checking city health, not agent health -- that's per-city).
patrol_interval = "10s"

# Future: tenant isolation
# [tenants.acme]
# cities = ["bright-lights", "dark-alleys"]
# resource_limit = { max_agents = 50, max_cities = 5 }

[[cities]]
path = "/home/user/bright-lights"

[[cities]]
path = "/home/user/dark-alleys"

​

Reference-Level Explanation

​

1) Two-Level Supervision Tree

machine supervisor process (PID 1234)
~/.gc/supervisor.lock (flock)
~/.gc/supervisor.sock (unix socket)
|
+-- HTTP listener :8080
|   +-- /v0/cities
|   +-- /v0/city/{name}/*  --> dispatches to cityState
|   +-- /v0/* (compat)     --> dispatches to sole city (if only one)
|
+-- supervisor reconciliation loop (10s tick)
|   +-- for each registered city:
|       +-- ensure cityState exists
|       +-- city reconciliation loop (per-city patrol_interval tick)
|           +-- agent start/stop/drift detection (unchanged)
|           +-- crash loop quarantine (unchanged)
|           +-- idle timeout (unchanged)
|           +-- order dispatch (unchanged)
|
+-- per-city isolation:
    +-- bright-lights/
    |   +-- cityState { cfg, sp, stores, events }
    |   +-- tmux -L bright-lights (own tmux server)
    |   +-- .gc/events.jsonl (own event log)
    |
    +-- dark-alleys/
        +-- cityState { cfg, sp, stores, events }
        +-- tmux -L dark-alleys (own tmux server)
        +-- .gc/events.jsonl (own event log)

​

2) New Types

// Package supervisor manages multiple cities from a single process.
package supervisor

// Registry tracks registered cities. Backed by ~/.gc/cities.toml.
type Registry struct {
    mu     sync.RWMutex
    cities map[string]CityEntry // keyed by city name
    path   string               // path to cities.toml
}

// CityEntry is one registered city.
type CityEntry struct {
    Name string // derived from workspace.name or dir basename
    Path string // absolute path to city root
}

// Supervisor is the machine-wide controller.
type Supervisor struct {
    registry *Registry
    cities   map[string]*CityRuntime // keyed by city name
    mu       sync.RWMutex
    config   SupervisorConfig
}

// CityRuntime holds the running state for one city.
// This is essentially today's controllerState + reconciliation loop,
// extracted from cmd/gc/controller.go into a reusable unit.
type CityRuntime struct {
    state    *controllerState  // existing type, unchanged
    cancel   context.CancelFunc
    loop     *controllerLoop   // existing reconciliation loop
    watcher  *fsnotify.Watcher // per-city config watcher
}

​

3) API Routing

// resolveCity extracts the city name from the URL and returns the
// corresponding State. For /v0/city/{name}/..., it uses the path
// segment. For /v0/... (no city prefix), it uses the sole registered
// city (if exactly one) or returns 400 "city required".
func (s *SupervisorServer) resolveCity(r *http.Request) (api.State, string, error)

Request: GET /v0/city/bright-lights/agents
                      |
                      v
              resolveCity("bright-lights")
                      |
                      v
              cityRuntime.state (implements api.State)
                      |
                      v
              handleAgentList(w, r)  <-- UNCHANGED handler

GET  /v0/cities                    # list registered cities
GET  /v0/city/{name}               # city detail (status, agent count, etc.)
POST /v0/city/{name}/register      # register a city (alternative to CLI)
POST /v0/city/{name}/unregister    # unregister a city
GET  /v0/events/stream             # global SSE stream (all cities, tagged)

​

4) Supervisor Reconciliation

On each supervisor tick:
  1. Read ~/.gc/cities.toml (desired state)
  2. Compare against running CityRuntime map (actual state)
  3. For new cities:  load config, create CityRuntime, start city loop
  4. For removed cities: cancel city loop, graceful agent shutdown
  5. For existing cities: no action (city loop handles its own health)

​

5) Per-City Isolation (Unchanged)

• Lock file: ~/.gc/supervisor.lock (replaces per-city .gc/controller.lock)
• Control socket: ~/.gc/supervisor.sock (replaces per-city .gc/controller.sock)
• API port: Single port from supervisor.toml (replaces per-city [api] port)

​

6) Global Event Bus

// Multiplexer merges events from multiple city providers into one
// stream, tagging each event with its source city.
type Multiplexer struct {
    providers map[string]events.Provider // city name -> provider
}

func (m *Multiplexer) Watch(ctx context.Context, afterSeq uint64) (Watcher, error)

​

7) Concurrency Model

supervisor goroutines:
  1. main: supervisor reconciliation loop
  2. per-city: N city reconciliation loops (one goroutine each)
  3. HTTP server: shared across all cities (goroutine per request)
  4. fsnotify: 1 for cities.toml + N for per-city config dirs

Locking:
  - Supervisor.mu: protects cities map (add/remove city)
  - Each CityRuntime.state.mu: protects per-city state (existing RWMutex)
  - No cross-city locks needed (cities are independent)

​

8) Backward Compatibility

​

9) Future: Multi-Tenant Isolation

machine supervisor
+-- tenant "acme"
|   +-- city "acme-prod"
|   +-- city "acme-staging"
+-- tenant "bigcorp"
    +-- city "bigcorp-prod"

• Resource limits: max agents, max cities, CPU/memory cgroups
• API authentication: tenant API keys, JWT with tenant claim
• Network isolation: per-tenant bind addresses or Unix sockets
• Data isolation: per-tenant ~/.gc/tenants/{name}/cities.toml

/v0/tenant/{tenant}/city/{city}/agents    # tenant-scoped
/v0/city/{city}/agents                    # single-tenant (default)

• City names are unique within a registry (extend to unique within tenant)
• No global mutable state shared between cities
• API routing is a prefix match that can gain another segment

​

Primitive Test

​

Drawbacks

1. Complexity cost. One process managing N cities is harder to reason about than N independent processes. A bug in the supervisor takes down all cities, not just one. Erlang solves this with per-supervisor crash isolation; we’d need similar care (a panicking city loop must not crash the supervisor).
2. Blast radius. Today a misbehaving city.toml only affects one controller. With a shared supervisor, a city that causes excessive CPU or memory pressure affects all cities on the machine. Resource limits (cgroups, goroutine budgets) add complexity.
3. Migration burden. Users running per-city controllers must migrate. The --standalone escape hatch helps, but two modes means two code paths to maintain.
4. Per-city [api] port becomes obsolete. Users who built tooling around city-specific API ports must migrate to the unified port with city prefixes. The single-city backward-compat shim buys time but doesn’t eliminate the migration.
5. Lock file location. ~/.gc/ is user-scoped. Running the supervisor as a system service (multiple users) requires a different location (/var/run/gc/). Two code paths for lock location.

​

Alternatives

​

A. Do Nothing (Status Quo)

​

B. Proxy-Only Aggregator (No Shared Process)

proxy :8080
  /v0/city/bright-lights/* --> http://localhost:8081/*
  /v0/city/dark-alleys/*   --> http://localhost:8082/*

​

C. Kubernetes-Style: Cities as CRDs

​

D. systemd/launchd Integration

​

Resolved Questions

1. Lock file location. Use $XDG_RUNTIME_DIR/gc/ with fallback to ~/.gc/. System service deployment is a future concern — a GC_HOME env var override covers that case when it arises.
2. City name uniqueness. Path is the primary key in the registry (unique by definition). City name is the display label used in API URLs. Registration rejects duplicate names — the user must set a unique workspace.name in city.toml. Explicit, no magic.
3. Per-city API port migration. Warn and ignore. Log "city 'X' has [api] port=N which is ignored under supervisor mode" at startup. Not an error — just unused config the user can clean up at their pace.
4. gc start behavior change. Option B: gc start auto-registers the city and starts the supervisor if not running. The supervisor should just exist, not be thought about. gc start --standalone is the escape hatch for legacy per-city mode.
5. Goroutine isolation. Wrap each city loop goroutine in defer recover(). On panic: log error, emit city.crashed event, mark city unhealthy, retry after backoff. Same pattern as Erlang supervisor restart.
6. Config reload atomicity. Use the same 200ms debounce strategy as per-city config watching. Already a solved problem.
7. Dashboard protocol. Minimum viable: dashboard queries the first registered city and updates all API URLs to include the city prefix. No city selector in v0 — just make it work with the new URL scheme. City switcher is follow-on work.
8. Global event sequence numbering. Wall-clock ordering is sufficient for the global stream. The {city}:{seq} composite cursor is for resumption, not total ordering. Cities are independent — total cross-city ordering is a non-goal.

​

Implementation Plan

​

Phase 0: Extract CityRuntime (small)

​

Phase 1: Registry and Supervisor Daemon (medium)

• Add ~/.gc/ global directory, cities.toml registry file.
• gc register / gc unregister CLI commands.
• gc supervisor start / gc supervisor stop.
• gc cities list command.
• Supervisor process: reads registry, starts one CityRuntime per city.
• Single API port from supervisor.toml.
• Backward compat: gc start still works (registers + starts supervisor).
• gc start --standalone for legacy per-city mode.

​

Phase 2: API City Namespace (medium)

• Add /v0/cities endpoint.
• Add /v0/city/{name}/... URL prefix routing.
• Single-city backward compat: /v0/agents routes to sole city.
• Dashboard minimum viable: query first registered city, update all API URLs to include city prefix. No city selector yet.

​

Phase 3: Global Event Stream (small)

• events.Multiplexer wrapping per-city providers.
• GET /v0/events/stream global SSE endpoint.
• Events tagged with city name.

​

Phase 4: Tenant Isolation (future, large)

• Tenant registry and configuration.
• Per-tenant API authentication.
• Resource limits.
• URL prefix: /v0/tenant/{tenant}/city/{city}/...