916d671d499400c48936109133b00842e16232a1
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 not
|
94720f0fcb |
fix(observability): stop single-binary Tempo evicting its only ingester (closes #156) (#157)
## What & why
`verify-tracing` flaked on `verify-stack` run 722 — `FAIL — no single trace spanned ['bff', 'projection-api']` — and went green on a plain re-run of the same commit. **The trace chain was not broken; Tempo could not ingest:**
```
removing distributor_pool failing healthcheck addr=127.0.0.1:9095
reason="rpc error: code = DeadlineExceeded"
pusher failed to consume trace data err="context canceled" (x18)
```
The root cause is the *mechanism* of the data loss, not whatever caused the stall. Tempo runs **single-binary**, so the distributor and the ingester are the same process and the distributor's ingester pool holds exactly one, in-process, member. dskit nevertheless health-checks that member over loopback gRPC with a **1 s** deadline (`checkinterval: 15s`, confirmed from the running image's `/status/config`). On the shared runner a transient stall blows the deadline, the only ingester is evicted from the pool, and every subsequent push fails until the next check interval — spans silently dropped.
With one in-process ingester the health check can **never** route around a failure. Its only possible effect is to discard data. So it is off:
```yaml
ingester_client:
pool_config:
healthcheckenabled: false
```
This lands at the point where *both* candidate triggers named in #156 (GC pressure near `mem_limit`, CPU contention from the grown stack) turn into lost spans, so **`mem_limit: 400m` is untouched** — raising it on a memory-tight runner risks reintroducing the `verify-e2e` OOM of #144. It also does not paper over anything the way a longer `TRACING_TIMEOUT` would (#156's own note).
Second change: `infra/tracing-check.py` prints `tempo_distributor_ingester_clients` on its failure path. From the check's side, Tempo-dropped-spans and missing instrumentation look identical — that ambiguity is what cost a container-log dive on run 722. A recurrence now names itself.
Closes #156
## Definition of Done
- [x] Linked Gitea issue (#156).
- [ ] **Failing test committed before the implementation — N/A, and deliberately so.** The trigger is runner load, so no deterministic red exists; the "red" is run 722's observed `verify-tracing` failure plus its Tempo logs. Same precedent as
|
||
|
not
|
88338396f6 |
feat(obs): distributed traces across the .NET services (S-16b, closes #123) (#126)
## What & why S-16b, second of the S-16 split, on top of the #125 backplane. The five .NET services now emit OpenTelemetry traces so a request is **one connected trace** across them. - Each host wires `AddOpenTelemetry().WithTracing(...)` with `AddAspNetCoreInstrumentation` (incoming) + `AddHttpClientInstrumentation` (outgoing) + `AddOtlpExporter` to **Tempo**. - Because every cross-service call already goes through a typed `HttpClient` (§8 boundaries), the W3C `traceparent` propagates with no manual code — bff → domain → acl → openzaak and bff → projection-api stitch into a single trace. - Service name + OTLP endpoint come from `OTEL_*` env set per app service in compose. `/health` is filtered out so liveness polls don't flood the traces. No new ADR — ADR-0023 already records the stack + the two documented gaps (browser-side tracing is out of scope, so the trace begins at the BFF; the async Flowable-poll boundary is a separate trace). Closes #123 ## Definition of Done - [x] Failing test committed first (`verify-tracing` fails with no instrumentation). - [x] Implementation makes it pass — **validated locally end to end**: a real connected trace spanning `bff` + `projection-api` was found in Tempo (BFF→projection→db + Tempo subset, no OpenZaak/egress). - [x] Conventional Commits referencing the issue (`refs #123`). - [ ] CI green — awaiting Gitea Actions (verify-tracing added to verify-stack after verify-bff). - [x] `docker compose up` health unaffected — services boot healthy even when Tempo is unreachable (exporter no-ops; verified). - [x] Docs — demo-script + BACKLOG. - [x] ADR — none needed (covered by ADR-0023). ## Notes for reviewers - **Per-service wiring, no shared lib:** the block is duplicated across the five hosts by design — services don't share code across boundaries here (§8), same as the duplicated typed clients. - **Packages:** OpenTelemetry.Extensions.Hosting / Instrumentation.AspNetCore / Instrumentation.Http / Exporter.OpenTelemetryProtocol, all 1.17.0, pinned per-csproj (no central props file). - **The check** generates anonymous BFF→projection traffic (no auth, no OpenZaak), then queries Tempo (TraceQL search → fetch trace → assert both service.names present) from a python:3-slim container in-network — same idiom as run-projection-check.sh. - **Next:** #124 (S-16c) adds `/metrics` + Prometheus scrape targets + golden-signal Grafana dashboards. Reviewed-on: #126 |