From 60d556b46edc1a76a2d5cf1b0897d1875a1c393b Mon Sep 17 00:00:00 2001 From: Niek Otten Date: Tue, 1 Sep 2026 09:45:28 +0200 Subject: [PATCH] docs(architecture): record why single-binary Tempo has its ingester health check off (refs #156) --- docs/architecture/adr-0023-observability-stack.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/architecture/adr-0023-observability-stack.md b/docs/architecture/adr-0023-observability-stack.md index 7adb034..da7f6cf 100644 --- a/docs/architecture/adr-0023-observability-stack.md +++ b/docs/architecture/adr-0023-observability-stack.md @@ -67,6 +67,14 @@ itself, so no in-image healthcheck tool is required. - Three more images built each CI run (kept small; not on the health-gate list). - Storage is ephemeral container fs — a demo backplane, not a retention target. Object storage for Tempo / remote-write for Prometheus is a later concern. +- Tempo runs **single-binary**, so its distributor and ingester are one process and + some of its distributed-mode machinery is not just redundant but harmful. Its + ingester-pool health check is disabled (`ingester_client.pool_config`) because with + a single in-process ingester the check can never route around a failure — a 1s + loopback-gRPC deadline missed under CI load only evicted the one ingester and made + Tempo drop spans, which is how `verify-tracing` flaked (#156). Expect the same + shape from other distributed-mode knobs if we tune them; the fix is to switch to + real multi-ingester Tempo, not to re-enable them here. ## Coupling rules touched (CLAUDE.md §8)