slatepipe operational

Status › Install

Install

Fifteen minutes to a running collector, including the database. Everything below is version 0.14.2; older agents talk to newer collectors within the same minor series.

What it needs

PieceMinimumNotes
Collector1 vCPU, 512 MB Comfortable to about 40 runs per minute on that. CPU is spent on JSON validation.
DatabasePostgreSQL 13+ SQLite is accepted for a single-user evaluation and refuses to start with two collectors.
Agent20 MB RSS Static binary for linux/amd64 and linux/arm64. No systemd dependency.
NetworkAgent → collector One outbound TCP port from each host that runs jobs. Nothing inbound to the agent.

With containers

Put this in /opt/slatepipe/compose.yml, change the two secrets, and bring it up.

services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: slatepipe
      POSTGRES_USER: slatepipe
      POSTGRES_PASSWORD: change-this-first
    volumes:
      - ./pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U slatepipe"]
      interval: 5s
      retries: 10

  collector:
    image: slatepipe/collector:0.14.2
    depends_on:
      db:
        condition: service_healthy
    environment:
      SLATEPIPE_DSN: postgres://slatepipe:change-this-first@db:5432/slatepipe?sslmode=disable
      SLATEPIPE_INGEST_TOKEN: 32-plus-random-characters-here
      SLATEPIPE_RETENTION_DAYS: "90"
    ports:
      - "7180:7180"
      - "127.0.0.1:7181:7181"
    volumes:
      - ./collector.yml:/etc/slatepipe/collector.yml:ro
    restart: unless-stopped

The interface is bound to localhost on purpose. It has no accounts and no login screen, so put it behind whatever already authenticates your internal tools, or read it over an SSH port forward.

docker compose -f /opt/slatepipe/compose.yml up -d
docker compose -f /opt/slatepipe/compose.yml exec collector slatepipe-collector migrate

migrate is idempotent and prints the schema version it ended on. It is safe to run on every deploy and that is how we run it.

Without containers

Download the archive for your architecture, verify the checksum shipped beside it, and drop two files into place.

tar -xzf slatepipe-0.14.2-linux-amd64.tar.gz
install -m 0755 slatepipe-collector slatepipe-agent /usr/local/bin/
install -d -m 0750 -o slatepipe -g slatepipe /etc/slatepipe /var/lib/slatepipe
[Unit]
Description=Slatepipe collector
After=network-online.target postgresql.service

[Service]
User=slatepipe
EnvironmentFile=/etc/slatepipe/env
ExecStart=/usr/local/bin/slatepipe-collector serve --config /etc/slatepipe/collector.yml
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target

Keep the DSN and the ingest token in /etc/slatepipe/env at mode 0640 rather than in the YAML file, which you will want to keep in version control.

First-run checklist

  1. slatepipe-collector check --config /etc/slatepipe/collector.yml - parses the file, connects to the database, and exits non-zero with a line number if anything is wrong.
  2. Confirm the schema is current: slatepipe-collector migrate --dry-run should report no pending steps.
  3. Send one synthetic run from the host that will do the real work: slatepipe-agent --wrap --pipeline smoke -- /bin/true. It should appear in the interface before you finish reading this line.
  4. Break it once on purpose with -- /bin/false, so you have seen what a non-zero exit looks like on the dashboard and in your inbox.
  5. Only then wrap a real job, and start with the least important one you have.

Upgrading

Replace the binary or the image tag, run migrate, restart. Migrations inside a minor series are additive, so a collector one patch behind the schema keeps serving reads. Agents are independent: we support agents up to two minor versions older than the collector, and the collector logs the agent version with every rejected record so a straggler is easy to find.

Downgrades are not supported, which is a polite way of saying we do not write reverse migrations. Snapshot the database before a minor-version jump.

Removing it

Stop both units, DROP SCHEMA slatepipe CASCADE, delete /etc/slatepipe and /var/lib/slatepipe, and take the wrapper out of your crontab. There is nothing else: no kernel module, no sidecar, no entry in a registry somewhere. Your jobs keep running because they never depended on us in the first place.