- Add dependency-free TF-IDF retrieval (src/lib/retrieval.js) with NL+EN
stopwords and a WeakMap-cached index.
- Rewrite buildKbContext to ship the top-K relevant topics + verbatim-
mentioned ids only, filter relations to the included set, and append a
[kb_hash: <8 hex>] suffix so the ephemeral prompt cache busts when the
graph changes. Returns { context, retrievedTopics, allTopics }.
- Add LOOKUP_TOPIC_TOOL and drive useChat through callLLM directly with a
multi-hop tool_result loop capped at 3 hops; preserve Anthropic-provided
tool_use ids through callLLM so the loop can echo correct tool_use_id.
- Truncate R42 history to the last 12 turns and prepend a single
"(earlier conversation truncated)" assistant message.
- Set R42 chat defaults: temperature 0.3, maxTokens 2048.
- Add pb_migrations/1780500002_created_llm_calls.js (the best-effort
logger in callLLM was already wired) and a new Admin → Diagnostics
view showing the last 100 calls with token usage, cache-hit rate, and
USD cost from a local Anthropic price table.
- Finalize AI_PIPELINE_HARDENING_PLAN.md: mark Phases 1–5 shipped and
Phase 6 (eval harness) explicitly out of scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace stateless one-shot extraction with a stateful, paced, cancellable
pipeline. Six subtasks:
- 3.1 Sentence-aware chunking with 800-char overlap (was paragraph-only
at 4000 chars). Hard-split fallback for runaway sentences.
- 3.2 Stateful extraction: chunks 2+ receive an "already-extracted topic
IDs" hint capped at 200 IDs, so the model reuses IDs instead of
inventing variants like software-developer vs software-engineer.
- 3.3 Token-bucket limiter in llmRetry.js (extractionLimiter, 5 req/min).
callLLM awaits the limiter before fetch; 429+Retry-After calls
pauseUntil. Replaces hard setTimeout(12000) and setTimeout(15000).
- 3.4 relevance_locked column on topics — admin edits to relevance are
sticky across re-extraction. Migration + merge respects the flag +
unlock checkbox in KnowledgeGraph edit form.
- 3.5 Unify relation vocabulary — handbook prompt no longer mentions
legacy "executes"; one-shot migration rewrites existing executes rows
to executed_by with source/target swapped.
- 3.6 Cancellation — Cancel button on UploadZone wired to an
AbortController threaded into callLLM; aborted runs persist status =
"cancelled" rather than "failed".
Tests: 16 new unit tests for chunkText, buildKnownIdsHint, and
createLimiter. All 61 tests pass, 0 lint errors, build clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
