not
e54dbe9d5d
`verify-tracing` flaked on run 722: `FAIL — no single trace spanned ['bff',
'projection-api']`, green on a plain re-run of the same commit. Not a broken trace
chain — Tempo could not ingest:
removing distributor_pool failing healthcheck addr=127.0.0.1:9095
reason="rpc error: code = DeadlineExceeded"
pusher failed to consume trace data err="context canceled" (x18)
Root cause is the mechanism of the data loss, not whatever caused the stall. Tempo
runs single-binary, so distributor and ingester are the *same process* and the
distributor's ingester pool holds exactly one, in-process, member. dskit still
health-checks it over loopback gRPC with a 1s deadline; on the shared runner a
transient stall blows that, the only ingester is dropped from the pool, and every
push fails until the next 15s check interval — spans silently lost. With one
in-process ingester the check can never route around a failure, so it can only ever
discard data.
Fix: `ingester_client.pool_config.healthcheckenabled: false`. This addresses both
candidate triggers (GC pressure near `mem_limit`, CPU contention) at the point where
they turn into lost data, so `mem_limit: 400m` stays untouched — raising it would
risk reintroducing the verify-e2e OOM of #144 on a memory-tight runner.
Also print `tempo_distributor_ingester_clients` on the check's failure path: a
recurrence then names Tempo-dropped-spans instead of costing another container-log
dive, since from the check's side that is indistinguishable from missing
instrumentation.
Verified against the built image: config parses (`-config.verify`), the effective
`/status/config` reports `healthcheckenabled: false`, and the diagnostic reads the
metric off a live Tempo.
40 lines
1.5 KiB
YAML
40 lines
1.5 KiB
YAML
# Grafana Tempo — single-binary, all-in-one, local storage (S-16a, ADR-0023).
|
|
# Ingests OTLP directly (services export straight to Tempo; no collector hop).
|
|
# Storage is ephemeral container fs — this is a local/CI demo backplane, not a
|
|
# retention target. ponytail: local backend, swap for object storage if traces
|
|
# must outlive the stack.
|
|
server:
|
|
http_listen_port: 3200
|
|
|
|
distributor:
|
|
receivers:
|
|
otlp:
|
|
protocols:
|
|
grpc:
|
|
endpoint: 0.0.0.0:4317
|
|
http:
|
|
endpoint: 0.0.0.0:4318
|
|
|
|
ingester:
|
|
max_block_duration: 5m
|
|
|
|
storage:
|
|
trace:
|
|
backend: local
|
|
local:
|
|
path: /var/tempo/blocks
|
|
wal:
|
|
path: /var/tempo/wal
|
|
|
|
# #156: don't let the distributor evict its own ingester. Tempo runs single-binary here, so the
|
|
# distributor and the ingester are the same process and the "pool" holds exactly one, in-process,
|
|
# member. dskit still health-checks it over loopback gRPC with a 1s deadline (checkinterval 15s);
|
|
# on the shared CI runner a transient stall blows that deadline, the only ingester is dropped from
|
|
# the pool ("removing distributor_pool failing healthcheck"), and every push then fails ("pusher
|
|
# failed to consume trace data", err="context canceled") until the next check — silently losing
|
|
# spans, which is how verify-tracing flaked. With one in-process ingester the check can never route
|
|
# around a failure, so it can only ever discard data. Turn it off.
|
|
ingester_client:
|
|
pool_config:
|
|
healthcheckenabled: false
|