​

Orders

Last verified against code: 2026-03-01

​

Summary

​

Key Concepts

• Order: A parsed definition from an order.toml file with a Name (derived from subdirectory name), a dispatch action (Formula or Exec, mutually exclusive), a Gate type, gate-specific parameters, and optional Pool routing. Defined in the Order struct at internal/orders/order.go.
• Gate: The trigger condition that controls when an order fires. Five types exist: cooldown (minimum interval since last run), cron (5-field schedule matching), condition (shell command exits 0), event (matching events after a cursor position), and manual (explicit invocation only, never auto-fires). See internal/orders/gates.go.
• Exec Order: An order whose action is a shell command (exec field) run directly by the controller. No LLM, no agent, no wisp. The script receives ORDER_DIR in its environment, set to the directory containing the order.toml file. Default timeout: 60 seconds.
• Formula Order: An order whose action is a formula name (formula field). When the gate opens, the controller calls MolCook to instantiate a wisp and labels it for pool dispatch. Default timeout: 30 seconds.
• ScopedName: A rig-qualified key that creates unique identity for orders across rigs. City-level orders use the plain name (e.g., dolt-health). Rig-level orders append :rig:<rigName> (e.g., dolt-health:rig:demo-repo). ScopedName drives independent cooldown tracking, event cursors, and label scoping.
• Formula Layer: A directory scanned for orders/*/order.toml. Layers are ordered lowest to highest priority; a higher-priority layer’s order definition overrides a lower-priority one with the same subdirectory name (last-wins semantics).
• Tracking Bead: A bead created synchronously before each dispatch goroutine launches, labeled order-run:<scopedName>. Serves dual purpose: prevents the cooldown gate from re-firing on the next tick, and provides execution history for gc order history.

​

Architecture

• internal/orders/ — parsing, validation, scanning, and gate evaluation. Pure library code with no side effects beyond shell command execution for condition gates.
• cmd/gc/ — controller-side dispatch (order_dispatch.go) and CLI commands (cmd_order.go). Wires the library into the controller loop and the gc order command tree.

                ┌──────────────────────────────────────────────────┐
                │           Controller Tick                        │
                │           cmd/gc/controller.go                   │
                │                                                  │
                │  ┌────────────────────────────────────────────┐  │
                │  │  orderDispatcher.dispatch(ctx, now)    │  │
                │  │  cmd/gc/order_dispatch.go              │  │
                │  │                                             │  │
                │  │  for each order:                       │  │
                │  │    ┌───────────────────────────────────┐    │  │
                │  │    │ CheckGate(a, now, lastRunFn,      │    │  │
                │  │    │          ep, cursorFn)             │    │  │
                │  │    │ internal/orders/gates.go      │    │  │
                │  │    └───────────┬───────────────────────┘    │  │
                │  │                │                             │  │
                │  │        ┌───────▼────────┐                   │  │
                │  │        │ GateResult.Due? │                   │  │
                │  │        └──┬──────────┬──┘                   │  │
                │  │       no  │          │ yes                   │  │
                │  │       skip│          ▼                       │  │
                │  │           │  Create tracking bead (sync)    │  │
                │  │           │          │                       │  │
                │  │           │          ▼                       │  │
                │  │           │  go dispatchOne(a) ──────────┐  │  │
                │  │           │     ┌────────┬────────┐      │  │  │
                │  │           │     │ IsExec │Formula │      │  │  │
                │  │           │     ▼        ▼        │      │  │  │
                │  │           │  shell    MolCook     │      │  │  │
                │  │           │  script   + label     │      │  │  │
                │  │           │     │        │        │      │  │  │
                │  │           │     └────┬───┘        │      │  │  │
                │  │           │          ▼            │      │  │  │
                │  │           │   Record event       │      │  │  │
                │  │           │   (fired/completed/  │      │  │  │
                │  │           │    failed)            │      │  │  │
                │  │           │                       │      │  │  │
                │  └───────────┴───────────────────────┴──────┘  │
                └──────────────────────────────────────────────────┘

                ┌──────────────────────────────────────────────────┐
                │          Order Discovery (Startup)          │
                │                                                  │
                │  buildOrderDispatcher()                     │
                │    ├─ cityFormulaLayers(cfg)                     │
                │    ├─ orders.Scan(cityLayers)               │
                │    ├─ for each rig:                              │
                │    │   ├─ rigExclusiveLayers(rigLayers, city)    │
                │    │   ├─ orders.Scan(exclusive)            │
                │    │   └─ stamp Rig field on each order     │
                │    ├─ filter out manual-gate orders          │
                │    └─ return memoryOrderDispatcher           │
                └──────────────────────────────────────────────────┘

​

Data Flow

1. buildOrderDispatcher() resolves city-level formula layers via cityFormulaLayers() and calls orders.Scan() to find city orders.
2. For each rig, rigExclusiveLayers() strips the city prefix from the rig’s formula layers to avoid double-scanning city orders. The remaining rig-exclusive layers are scanned separately.
3. Rig orders get their Rig field stamped with the rig name.
4. Manual-gate orders are filtered out (they never auto-dispatch).
5. If no auto-dispatchable orders remain, the dispatcher is nil (nil-guard pattern — callers check before use).

1. dispatch() iterates all non-manual orders.
2. CheckGate() evaluates the gate condition against current time, last-run history (from bead store), and event state (from event bus).
3. For each due order, a tracking bead is created synchronously with label order-run:<scopedName>. This is critical: it prevents the cooldown gate from re-firing on the next tick.
4. A goroutine calls dispatchOne() with a context timeout derived from effectiveTimeout() (per-order timeout capped by global max_timeout).
5. dispatchOne() records an order.fired event, then branches:
   • Exec: dispatchExec() runs the shell command via ExecRunner, labels the tracking bead with exec (or exec-failed), and records order.completed or order.failed.
   • Formula: dispatchWisp() calls instantiateWisp() (which delegates to store.MolCook()), labels the wisp root bead with order-run:<scopedName> and pool:<qualifiedPool>, and records order.completed or order.failed.

1. For each formula layer (ordered lowest to highest priority), read <layer>/orders/*/order.toml.
2. Parse each TOML file into an Order struct. Set Name from the subdirectory name, Source from the absolute file path.
3. Higher-priority layers overwrite lower ones by name (map keyed on subdirectory name).
4. Exclude disabled orders (enabled = false) and those in the skip list.
5. Return the result slice preserving discovery order.

​

Key Types

• Order (internal/orders/order.go): The parsed order definition. Fields: Name, Description, Formula, Exec, Gate, Interval, Schedule, Check, On, Pool, Timeout, Enabled, Source, Rig.
• GateResult (internal/orders/gates.go): The outcome of a gate check. Fields: Due (bool), Reason (human-readable), LastRun (time.Time).
• orderDispatcher (cmd/gc/order_dispatch.go): Interface with a single method dispatch(ctx, cityPath, now). Production implementation is memoryOrderDispatcher.
• memoryOrderDispatcher (cmd/gc/order_dispatch.go): Holds the scanned order list, bead store, events provider, command runner, exec runner, events recorder, stderr writer, and max timeout.
• ExecRunner (cmd/gc/order_dispatch.go): Function type func(ctx, command, dir string, env []string) ([]byte, error) for running shell commands. Production implementation shellExecRunner uses os/exec.

​

Invariants

• Formula XOR Exec: Every order has exactly one of formula or exec set. Validate() rejects orders with both or neither.
• Exec orders have no pool: An exec order runs a shell script directly on the controller. It has no agent pipeline and therefore no pool. Validate() rejects exec + pool combinations.
• Gate type requires matching parameters: A cooldown gate requires interval, a cron gate requires schedule, a condition gate requires check, an event gate requires on. Validate() enforces these per-gate-type constraints.
• Tracking beads are created before dispatch goroutines: The tracking bead (labeled order-run:<scopedName>) is created synchronously in the main dispatch loop. This prevents the cooldown gate from re-firing on the next controller tick while the dispatch goroutine is still running.
• ScopedName provides rig isolation: The same order name deployed to multiple rigs produces independent scoped names (e.g., dolt-health:rig:rig-a vs dolt-health:rig:rig-b). Cooldown tracking, event cursors, and history queries all use ScopedName. Firing one rig’s order does not affect another rig’s gate evaluation.
• Higher-priority layers override lower by name: When the same order subdirectory name exists in multiple formula layers, Scan() uses the definition from the highest-priority layer (last in the layers slice). The override is total (the entire TOML definition replaces the lower one).
• Manual gates never auto-fire: CheckGate() for a manual gate always returns Due: false. Manual orders are filtered out of the dispatcher entirely during build. They can only be triggered via gc order run.
• Disabled orders are excluded from scan results: Scan() filters out orders with enabled = false. They do not appear in any CLI command output or dispatch evaluation.
• Cron gate fires at most once per minute: After matching the 5-field schedule, checkCron() verifies the last run was not in the same truncated minute. This prevents duplicate fires within a single cron window.
• Event gate uses cursor-based deduplication: Event orders track the highest processed event sequence number via seq:<N> labels on wisp beads. Subsequent gate checks use AfterSeq filtering to avoid reprocessing already-handled events.
• Dispatch is fire-and-forget: Once a goroutine is launched, the controller does not track its completion. Failed orders emit order.failed events but do not retry. The tracking bead prevents re-fire within the same cooldown window.
• No role names in Go code: The order subsystem operates on config-driven pool names and formula references. No line of Go references a specific role name.

​

Interactions

​

Code Map

​

Configuration

​

order.toml (per-order definition)

[order]
description = "Check database health"
formula = "mol-db-health"        # dispatch action (XOR with exec)
# exec = "scripts/check-db.sh"   # alternative: shell script dispatch
gate = "cooldown"                # cooldown | cron | condition | event | manual
interval = "5m"                  # required for cooldown gate
# schedule = "0 3 * * *"         # required for cron gate (5-field)
# check = "test -f /tmp/flag"    # required for condition gate
# on = "bead.closed"             # required for event gate
pool = "worker"                  # target pool for formula dispatch (optional)
timeout = "90s"                  # per-order timeout (optional)
enabled = true                   # default: true

​

city.toml (global settings)

[orders]
skip = ["noisy-order"]      # order names to exclude from scanning
max_timeout = "120s"             # hard cap on per-order timeout (default: uncapped)

​

Order layering (override priority, lowest to highest)

1. City pack formulas — from pack referenced in city.toml
2. City local formulas — from [formulas] section or .gc/formulas/
3. Rig pack formulas — from pack applied to a specific rig
4. Rig local formulas — from rig’s formulas_dir

​

Rig-scoped orders

• Same order deployed to rigs rig-a and rig-b tracks independently as db-health:rig:rig-a and db-health:rig:rig-b.
• Pool names are auto-qualified: pool = "worker" in rig demo-repo becomes pool:demo-repo/worker on the wisp label. Already-qualified names (containing /) are left unchanged.

​

Testing

​

Known Limitations

• No retry on dispatch failure: Failed orders emit events but are not retried. The tracking bead prevents re-fire within the same cooldown window, so a failed order must wait for the next gate opening.
• Cron granularity is minutes: The cron gate operates at minute-level granularity with simple field matching (*, exact integer, comma-separated values). It does not support ranges (1-5), steps (*/5), or sub-minute scheduling.
• Condition gate blocks the dispatch loop: checkCondition() runs sh -c <check> synchronously during gate evaluation. A slow check command blocks evaluation of subsequent orders on that tick.
• Event gate cursor is per-wisp, not per-dispatch: The cursor position is computed from seq:<N> labels on existing wisp beads via MaxSeqFromLabels(). If wisp creation fails, the cursor is not advanced, which may cause duplicate event processing on retry.
• No hot-add of orders: Order discovery runs on controller start and config reload (via fsnotify). Adding a new order.toml file requires the config directory watcher to trigger a reload; adding a new formula layer directory requires a city.toml change.
• Fire-and-forget goroutines: Dispatch goroutines are not tracked by the controller. On shutdown, in-flight dispatches may be interrupted mid-execution if the context is canceled.

​

See Also

• Architecture glossary — authoritative definitions of order, gate, wisp, formula, and other terms used in this document
• Health Patrol architecture — the controller loop that drives order dispatch on each tick
• Beads architecture — the bead store used for tracking beads, wisp instantiation via MolCook, and label-based queries
• Config architecture — FormulaLayers resolution, pack expansion, and OrdersConfig
• Gate evaluation logic — CheckGate implementation for all five gate types
• Order discovery — Scan function for formula layer traversal
• Controller dispatch — production dispatcher wiring exec and formula orders
• Event type constants — order.fired, order.completed, order.failed event types