slatepipe operational

Status › How it works

How it works

Three pieces, one direction of travel. Everything pushes forward; nothing polls backwards.

On this page

  1. The agent
  2. The collector
  3. The database
  4. One run end to end
  5. Collector down
  6. No agent inventory
Where the three pieces sityour schedulercron, timer, CI stepslatepipe-agentwraps the commandspool/var/lib/slatepipeslatepipe-collectoringest :7180PostgreSQL 13+runs, steps, eventsread-only UI:7181alertsSMTP or webhookHTTP+ tokenon failurereplayThe agent is the only piece that touches your job. Nothing in this picture reaches outside your network.Two processes, one database. The collector is stateless apart from PostgreSQL, so you can run a second one behind a load balancer.Agents push; the collector never calls out to an agent, which is why there is no host inventory to keep in sync.
Ports are defaults and both are configurable. The dashed box on the left is yours - Slatepipe neither replaces it nor needs to know what it is.

The agent

A single static Go binary, 11 MB, no shared libraries. In its usual mode it is a wrapper: it execs your command, passes standard input and output straight through, records the start and end time, and reads any step marks the job prints along the way. It adds roughly 9 ms to process startup and does not buffer your output, so interactive use is unaffected.

The agent never parses your data. It counts what the job tells it to count, which is why a step that stays silent about rows shows a duration and nothing else.

The collector

Accepts records over HTTP on :7180, validates them against a schema, writes them to PostgreSQL, and serves the read-only interface on :7181. It evaluates alert rules after each insert rather than on a timer, so a failed run reaches an inbox in about a second rather than at the top of the minute.

Two collectors can share one database. They do not coordinate: the only shared mutable state is the alert de-duplication table, and that is guarded by a unique constraint instead of a lock server.

The database

PostgreSQL 13 or newer, three tables and two views. Runs and steps are rows; raw event payloads are kept as jsonb for ninety days and then dropped by prune, while the aggregated numbers stay for as long as you keep them. A year of 200 pipelines running nightly is roughly 1.4 GB, most of it the stderr tails.

One run, start to finish

  1. Your scheduler runs slatepipe-agent --wrap --pipeline nightly-ledger -- ./load.sh at 02:00.
  2. The agent posts a run.started event with the pipeline name, host, and the exact command line it is about to execute. The collector answers with a run id.
  3. load.sh prints ##slatepipe step=extract rows=2410883 when its first stage ends. The agent turns that line into a step record and keeps it out of your own log stream.
  4. The script exits 0 at 02:12. The agent posts run.finished with the exit status, total duration, per-step timings and the last 64 KB of standard error.
  5. The collector compares 12 minutes against the trailing median for this pipeline, finds it unremarkable, writes the row and evaluates no rules.
  6. The next morning the same pipeline takes 41 minutes. That one crosses the ratio in the rule file, and an email goes out naming the step that grew.

While the collector is down

Events are appended to a spool directory, by default /var/lib/slatepipe/spool with a 256 MB ceiling. The agent retries with a jittered backoff up to ten minutes apart and replays in order once the collector answers again. If the ceiling is reached the oldest spool files are deleted first and a counter is reported with the next successful batch, so you can tell a gap in history from a quiet week.

The wrapped command is never blocked by any of this. An agent that cannot reach the collector still returns your job's own exit code, unchanged, and that behaviour is covered by a test we care about more than most.

Why there is no agent inventory

A pushing agent needs no registration, which means no host list to drift out of date and no credential to rotate on 300 machines at once. The price is honest: the collector cannot tell you that a machine stopped reporting unless you declare the pipeline's expected schedule in the rule file. We think that is the right trade for installs of this size, and it is the first thing we would revisit if anyone showed us a fleet where it hurts.

Reading the data directly is expected and supported. The two views v_run_summary and v_step_latest are the same ones the interface uses, and they will not change shape inside a minor version.

Related

InstallConfigureRun historyData & views