not
88338396f6
## What & why S-16b, second of the S-16 split, on top of the #125 backplane. The five .NET services now emit OpenTelemetry traces so a request is **one connected trace** across them. - Each host wires `AddOpenTelemetry().WithTracing(...)` with `AddAspNetCoreInstrumentation` (incoming) + `AddHttpClientInstrumentation` (outgoing) + `AddOtlpExporter` to **Tempo**. - Because every cross-service call already goes through a typed `HttpClient` (§8 boundaries), the W3C `traceparent` propagates with no manual code — bff → domain → acl → openzaak and bff → projection-api stitch into a single trace. - Service name + OTLP endpoint come from `OTEL_*` env set per app service in compose. `/health` is filtered out so liveness polls don't flood the traces. No new ADR — ADR-0023 already records the stack + the two documented gaps (browser-side tracing is out of scope, so the trace begins at the BFF; the async Flowable-poll boundary is a separate trace). Closes #123 ## Definition of Done - [x] Failing test committed first (`verify-tracing` fails with no instrumentation). - [x] Implementation makes it pass — **validated locally end to end**: a real connected trace spanning `bff` + `projection-api` was found in Tempo (BFF→projection→db + Tempo subset, no OpenZaak/egress). - [x] Conventional Commits referencing the issue (`refs #123`). - [ ] CI green — awaiting Gitea Actions (verify-tracing added to verify-stack after verify-bff). - [x] `docker compose up` health unaffected — services boot healthy even when Tempo is unreachable (exporter no-ops; verified). - [x] Docs — demo-script + BACKLOG. - [x] ADR — none needed (covered by ADR-0023). ## Notes for reviewers - **Per-service wiring, no shared lib:** the block is duplicated across the five hosts by design — services don't share code across boundaries here (§8), same as the duplicated typed clients. - **Packages:** OpenTelemetry.Extensions.Hosting / Instrumentation.AspNetCore / Instrumentation.Http / Exporter.OpenTelemetryProtocol, all 1.17.0, pinned per-csproj (no central props file). - **The check** generates anonymous BFF→projection traffic (no auth, no OpenZaak), then queries Tempo (TraceQL search → fetch trace → assert both service.names present) from a python:3-slim container in-network — same idiom as run-projection-check.sh. - **Next:** #124 (S-16c) adds `/metrics` + Prometheus scrape targets + golden-signal Grafana dashboards. Reviewed-on: #126
82 lines
2.5 KiB
Python
Executable File
82 lines
2.5 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""S-16b (#123): prove distributed tracing works end to end.
|
|
|
|
Generate anonymous BFF traffic (GET /openbaar/register, which the BFF serves by
|
|
calling projection-api — no auth, no OpenZaak egress), then query Tempo and assert
|
|
that ONE trace contains spans from both `bff` and `projection-api`. That proves the
|
|
services export OTLP to Tempo AND that the W3C traceparent propagates across the
|
|
HttpClient hop, stitching the request into a single connected trace.
|
|
|
|
Stdlib only (urllib/json) so it runs in a bare python:3-slim container in-network.
|
|
"""
|
|
import json
|
|
import os
|
|
import sys
|
|
import time
|
|
import urllib.error
|
|
import urllib.parse
|
|
import urllib.request
|
|
|
|
BFF = os.environ["BFF"] # http://<bff-ip>:8080
|
|
TEMPO = os.environ["TEMPO"] # http://<tempo-ip>:3200
|
|
TIMEOUT = int(os.environ.get("TRACING_TIMEOUT", "90"))
|
|
WANT = {"bff", "projection-api"} # the two services that must share one trace
|
|
|
|
|
|
def _get(url):
|
|
with urllib.request.urlopen(url, timeout=10) as r:
|
|
return r.read()
|
|
|
|
|
|
def generate_traffic():
|
|
# A non-2xx still produces spans; only total unreachability of the BFF is fatal.
|
|
for _ in range(3):
|
|
try:
|
|
_get(f"{BFF}/openbaar/register")
|
|
except urllib.error.HTTPError:
|
|
pass
|
|
|
|
|
|
def search_trace_ids():
|
|
q = urllib.parse.quote('{ resource.service.name = "bff" }')
|
|
try:
|
|
data = json.loads(_get(f"{TEMPO}/api/search?q={q}&limit=50"))
|
|
except Exception:
|
|
return []
|
|
return [t["traceID"] for t in data.get("traces", [])]
|
|
|
|
|
|
def services_in_trace(trace_id):
|
|
try:
|
|
data = json.loads(_get(f"{TEMPO}/api/traces/{trace_id}"))
|
|
except Exception:
|
|
return set()
|
|
names = set()
|
|
for batch in data.get("batches", []):
|
|
for attr in batch.get("resource", {}).get("attributes", []):
|
|
if attr.get("key") == "service.name":
|
|
names.add(attr.get("value", {}).get("stringValue"))
|
|
return names
|
|
|
|
|
|
def main():
|
|
deadline = time.time() + TIMEOUT
|
|
generate_traffic()
|
|
seen = set()
|
|
while time.time() < deadline:
|
|
for tid in search_trace_ids():
|
|
names = services_in_trace(tid)
|
|
seen |= names
|
|
if WANT.issubset(names):
|
|
print(f"OK — trace {tid} spans {sorted(names)}")
|
|
return 0
|
|
time.sleep(3)
|
|
generate_traffic()
|
|
print(f"FAIL — no single trace spanned {sorted(WANT)}; services seen: {sorted(seen)}",
|
|
file=sys.stderr)
|
|
return 1
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|