Incident history

Affected: alert delivery

• 05:41 UTC
  investigatingTwo failure alerts for the demo collector were acknowledged late; the relay queue was backing up.
• 05:44 UTC
  identifiedThe upstream SMTP relay was rejecting on a stale TLS session and the collector was retrying on the ninety-second schedule.
• 05:47 UTC
  resolvedThe relay recovered on its own; both queued messages left within one retry. Webhook delivery was never affected.

Affected: collector, interface

• 04:17 UTC
  investigatingThe dashboard went blank for the duration of the 04:17 prune; reads timed out against the statement limit.
• 04:22 UTC
  identifiedEvent-payload deletion was taking an exclusive lock on runs for roughly eighty seconds on a 1.4 GB database.
• 04:40 UTC
  monitoringPrune was moved off the hour and reads returned; a batched delete that does not hold the lock was staged for the next patch.
• 05:05 UTC
  resolvedThe 0.14.2 build ships the batched prune and the nightly pass now finishes in about eleven seconds.

Affected: interface

• 11:08 UTC
  investigatingA support customer on a satellite link reported the run page hanging before the error tail appeared.
• 11:40 UTC
  identifiedThe page requested the 64 KB tail once for the summary and once for the detail pane, doubling the transfer.
• 12:30 UTC
  resolvedFixed in 0.14.1; the tail is fetched once and shared between the two panes.

Affected: collector, database

• 02:00 UTC
  investigatingAgents began spooling to disk immediately after the collector was upgraded to 0.14.0.
• 02:03 UTC
  identifiedThe migration rewrites the baseline cache, and on this install that is about four minutes of write-heavy work that stalled new inserts.
• 02:09 UTC
  resolvedThe rewrite finished, the spool replayed in order and no run was lost. The upgrade note now warns that the pass runs about a minute per million runs.

Affected: collector, alert delivery

• 22:14 UTC
  investigatingA failing pipeline sent two of every alert after a second collector was added behind the load balancer.
• 23:10 UTC
  identifiedDe-duplication was kept in each collector's memory rather than shared, so both evaluated the same failure.
• 00:02 UTC
  resolved0.13.0 moves de-duplication into the database behind a unique constraint; one failure now sends one message.

Affected: agents

• 19:30 UTC
  investigatingThe collector host was down for a planned database move that ran long, and agents kept queuing events to disk.
• 21:00 UTC
  identifiedThree busy hosts crossed the 256 MiB spool ceiling and began dropping their oldest events first, as designed.
• 08:40 UTC
  resolvedThe collector came back, the spools replayed and the dropped-event counter reported the gap. The history shows a hole, not a silent loss.

Affected: interface

• 00:00 UTC
  monitoringThe single-page interface had become impossible to debug on customer networks and was scheduled for replacement.
• 00:00 UTC
  resolved0.10.0 serves plain server-rendered pages; the read-only interface has had no client framework since.