AI_PIPELINE_HARDENING_PLAN.md

@@ -0,0 +1,569 @@
+# AI Pipeline Hardening — Implementation Plan
+
+> **Audience:** an AI agent executing this plan against the Respellion Learning Platform.
+> **Owner before this work:** Raymond Verhoef (rve@respellion.nl).
+> **Source of truth for repo conventions:** [`AI_AGENT.md`](AI_AGENT.md). Read it before starting.
+
+This plan upgrades the platform's interaction with the Anthropic API: how prompts are built, how responses are parsed, how the model is retried, and how outputs are validated. It is broken into six phases that can be implemented and shipped independently. Each phase ends with verifiable acceptance criteria.
+
+---
+
+## 0. Operating principles
+
+These rules govern every phase. Re-read them before you commit.
+
+1. **PocketBase is the source of truth.** No persistent state in localStorage (see [`AI_AGENT.md`](AI_AGENT.md) §2). The Anthropic API is proxied via Caddy; **never** add `x-api-key` headers in frontend code.
+2. **No behaviour regressions.** Existing UI flows (extraction, weekly learning, weekly quiz, R42 chat, handbook sync, analyze-graph) must keep working after every phase. Phases are additive.
+3. **Schema-first.** Where the model produces structured output, define a JSON Schema (Zod) and validate every response. Reject (don't paper over) malformed output.
+4. **Single LLM entry point.** After Phase 1 there is exactly one module that talks to `/api/anthropic/v1/messages`. All callers go through it.
+5. **No silent truncation.** If `stop_reason === 'max_tokens'` on a structured-output call, throw — never persist a partial parse.
+6. **Cache what is stable, vary what is dynamic.** Use prompt caching for system prompts and KB context. User messages are never cached.
+7. **Comments and docs:** follow the repo's terse style. Don't add explanatory comments unless the *why* is non-obvious.
+8. **Migrations:** when changing a PocketBase schema, add a migration in `pb_migrations/` following the existing timestamp prefix convention. Never edit shipped migrations.
+9. **Stop and ask** if you encounter a decision the plan doesn't cover (e.g. a model deprecation, a missing collection, a failing test that looks pre-existing).
+
+### Files you will touch (or create) across all phases
+
+| Path | Purpose |
+|---|---|
+| `src/lib/llm.js` *(new, Phase 1)* | Single Anthropic client wrapper |
+| `src/lib/llmSchemas.js` *(new, Phase 1)* | Zod schemas for every structured task |
+| `src/lib/llmRetry.js` *(new, Phase 1)* | Retry + backoff + abort policy |
+| `src/lib/random.js` *(new, Phase 4)* | Fisher–Yates shuffle + RNG helpers |
+| `src/lib/api.js` | Becomes a thin re-export from `llm.js` (back-compat) |
+| `src/lib/extractionPipeline.js` | Migrated to `llm.js` + tool use + overlap chunking |
+| `src/lib/learningService.js` | Migrated to `llm.js` + tool use + patch-refine |
+| `src/lib/testService.js` | Migrated to `llm.js` + tool use + dedup + shuffle fix |
+| `src/components/admin/KnowledgeGraph.jsx` | `analyzeGraph` → tool use + dry-run preview |
+| `src/components/chat/rag.js` | Retrieval (TF-IDF) + `lookup_topic` tool |
+| `src/components/chat/prompts.js` | Split system prompt into cacheable + dynamic |
+| `src/components/chat/useChat.js` | Wire retrieval + truncation |
+| `pb_migrations/*` | Schema additions for `llm_calls`, question `difficulty`, topic `relevance_locked` |
+| `evals/` *(new, Phase 6)* | Golden-set eval harness |
+
+---
+
+## Phase 1 — Foundation (single LLM client + robust parsing)
+
+**Goal:** every LLM call goes through one module that handles retry, timeout, abort, JSON extraction, and schema validation. No behaviour change visible to the user.
+
+### 1.1 Create `src/lib/llmRetry.js`
+
+Implements the retry policy used by `llm.js`.
+
+**Behaviour:**
+- Exponential backoff with full jitter, base 1000ms, cap 16000ms.
+- Retries only on these HTTP statuses: `408, 425, 429, 500, 502, 503, 504, 529`.
+- Honours `Retry-After` header (seconds or HTTP date). If present and ≤ 60s, use it; if > 60s, fail fast.
+- Default `maxRetries = 4`.
+- Does **not** retry on `AbortError`.
+
+**Exported interface:**
+
+```js
+// withRetry: (fn: (attempt:number) => Promise<T>, opts?) => Promise<T>
+// RetryableError(status, retryAfterMs)
+export async function withRetry(fn, { maxRetries = 4, signal } = {}) { ... }
+export class RetryableError extends Error { constructor(status, retryAfterMs) { ... } }
+```
+
+### 1.2 Create `src/lib/llmSchemas.js`
+
+One Zod schema per structured task. Install Zod (`npm i zod`).
+
+Required schemas (names + shape match what callers already produce — do not change field names):
+
+- `extractionResultSchema` — `{ topics: Topic[], relations: Relation[] }` matching the existing `SYSTEM_PROMPT` in [`extractionPipeline.js`](src/lib/extractionPipeline.js).
+- `handbookResultSchema` — same shape, but `relation.type` enum unified to `related_to | depends_on | part_of | executed_by` (see Phase 3 task 3.5 — for now the schema accepts both `executes` and `executed_by`, normalize `executes → executed_by` post-validation).
+- `learningArticleSchema`, `learningSlidesSchema`, `learningInfographicSchema`, `learningAllSchema` matching [`learningService.js`](src/lib/learningService.js).
+- `quizQuestionsSchema` — `{ questions: Question[] }` with `options.length === 4` and `correctIndex ∈ [0,3]`.
+- `customTopicSchema` — `{ label, type: 'concept'|'role'|'process', description }`.
+- `graphActionsSchema` — `{ merges, deletions, newRelations, relevanceUpdates }` matching [`KnowledgeGraph.jsx:329`](src/components/admin/KnowledgeGraph.jsx).
+- `proposeGraphDeltaSchema` — matches `PROPOSE_GRAPH_DELTA_TOOL.input_schema` in [`prompts.js`](src/components/chat/prompts.js).
+
+**Acceptance:** every schema has at least one happy-path Vitest test in `src/lib/__tests__/llmSchemas.test.js` (add `vitest` if not present).
+
+### 1.3 Create `src/lib/llm.js`
+
+The single Anthropic client. All other modules must call only this one.
+
+**Public interface:**
+
+```js
+// Task tier — used to pick a model from settings.
+// 'fast'      → admin:model:fast      (default: claude-haiku-4-5-20251001)
+// 'standard'  → admin:model:standard  (default: claude-sonnet-4-6)
+// 'reasoning' → admin:model:reasoning (default: claude-opus-4-7)
+export async function callLLM({
+  task,            // string, e.g. 'extract.source' — used for logging only
+  tier = 'standard',
+  system,          // string OR Array<{ type:'text', text:string, cache_control?:{type:'ephemeral'} }>
+  messages,        // [{ role, content }] OR omitted (use `user`)
+  user,            // shorthand for [{role:'user', content: user}]
+  tools,           // optional Anthropic tool definitions
+  toolChoice,      // optional, e.g. { type:'tool', name:'emit_knowledge_graph' }
+  schema,          // optional Zod schema for text→JSON path (used only when no tool)
+  maxTokens = 4096,
+  temperature = 0,
+  signal,          // AbortSignal
+}): Promise<{
+  text: string,
+  toolUses: Array<{ name, input }>,
+  stopReason: string,
+  usage: { input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens },
+  requestId: string | null,
+  model: string,
+  durationMs: number,
+}>
+```
+
+**Key requirements:**
+
+1. **Simulation mode** preserved: if `storage.get('admin:use_simulation') === true`, return a deterministic stub (use existing `simulateResponse` payload for backward compatibility — branch on `task` prefix to return a matching stub).
+2. **Fetch** with `AbortController`; default **60-second timeout** if caller didn't pass a signal.
+3. **Retry** through `withRetry` (Phase 1.1).
+4. **Auth-portal detection** preserved: if response is not `application/json`, throw `Your session has expired. Please refresh the page and log in again.` exactly as today.
+5. **No truncation acceptance**: if `stop_reason === 'max_tokens'` AND caller passed `schema` OR `toolChoice` requested a tool, throw `LLMTruncatedError`.
+6. **Robust JSON extraction** when caller passed `schema` (and no tool was used): use `parseStructuredText(text)` that
+   - strips ```` ```json ```` and ```` ``` ```` fences,
+   - finds the outermost balanced JSON value (object **or** array) via a tiny brace-matching scan, not regex,
+   - throws `LLMOutputError` if no balanced JSON found,
+   - runs Zod `schema.parse` on the result.
+7. **Tool path:** when `tools` is provided and the model emits `tool_use`, return them under `toolUses`. Validate each tool's input against the corresponding Zod schema if the caller wired one in (via `toolSchemas: { [toolName]: ZodSchema }`).
+8. **Logging:** after every call, append a row to a new PocketBase collection `llm_calls` (best-effort — never block on this; catch and console.debug failures). Fields: `task, model, tier, duration_ms, input_tokens, output_tokens, cache_read_tokens, cache_create_tokens, stop_reason, ok, error_msg`. See Phase 5 task 5.6 for the migration.
+9. **Custom errors**: `LLMHttpError`, `LLMTruncatedError`, `LLMOutputError`, `LLMValidationError`. All extend `Error` and set `name` for `instanceof` checks.
+
+### 1.4 Make `src/lib/api.js` a thin shim
+
+Replace the existing `anthropicApi.generateContent` and `anthropicApi.chat` implementations with calls into `llm.js`. Preserve the exact exported names and return shapes so no caller breaks.
+
+```js
+// api.js after Phase 1
+export const anthropicApi = {
+  async generateContent(systemPrompt, userMessage, maxRetries = 1) {
+    const { text } = await callLLM({
+      task: 'legacy.generateContent',
+      tier: 'standard',
+      system: systemPrompt,
+      user: userMessage,
+      maxTokens: 8192,
+      temperature: 0,
+    });
+    return text;
+  },
+  async chat(systemPrompt, messages, opts = {}) {
+    const r = await callLLM({
+      task: 'legacy.chat',
+      tier: 'standard',
+      system: systemPrompt,
+      messages,
+      tools: opts.tools,
+      maxTokens: 1024,
+      temperature: 0.3,   // chat default — see Phase 5
+    });
+    return {
+      content: [
+        ...(r.text ? [{ type:'text', text: r.text }] : []),
+        ...r.toolUses.map(tu => ({ type:'tool_use', name: tu.name, input: tu.input })),
+      ],
+      stop_reason: r.stopReason,
+    };
+  },
+};
+```
+
+### 1.5 Update default model + tiered settings
+
+- Replace `DEFAULT_MODEL = 'claude-sonnet-4-20250514'` with the three tier defaults above.
+- In **Admin → Settings**, add three model selects (`fast`, `standard`, `reasoning`). Read existing `admin:model` as a legacy fallback for `standard` (so existing users don't lose their override).
+
+### Phase 1 acceptance criteria
+
+- [ ] `npm run lint` passes; `npm run test` passes (Vitest).
+- [ ] Every existing user flow (extraction, weekly content, weekly quiz, R42, handbook sync, analyze graph) still works against the live API.
+- [ ] `grep -r "fetch.*anthropic" src/` returns only `src/lib/llm.js`.
+- [ ] Simulation mode toggle still returns stubbed responses for all flows.
+- [ ] Manually verify: kill the network mid-call → request aborts within 60s and surfaces a clear error message.
+- [ ] Manually verify: rate-limit the proxy (429 + `Retry-After: 5`) → call retries once after ~5s and then succeeds.
+
+---
+
+## Phase 2 — Prompt caching & tool-based structured outputs
+
+**Goal:** structured-output tasks no longer parse JSON out of prose. Large stable prompts are cached.
+
+### 2.1 Migrate extraction to tool use
+
+In [`extractionPipeline.js`](src/lib/extractionPipeline.js):
+
+- Replace the "Return JSON only" instruction with a tool: `emit_knowledge_graph` whose `input_schema` mirrors `extractionResultSchema`.
+- Replace `anthropicApi.generateContent(...)` with `callLLM({ ..., tools:[emitKnowledgeGraphTool], toolChoice:{ type:'tool', name:'emit_knowledge_graph' } })`.
+- Read the validated object from `toolUses[0].input`.
+- Same migration for `analyzeHandbookDelta` (tool `emit_handbook_delta`).
+- Delete every `responseText.match(/\{[\s\S]*\}/)` site.
+
+### 2.2 Migrate learning, quiz, custom-topic, graph-actions to tool use
+
+Same pattern, in:
+
+- [`learningService.js`](src/lib/learningService.js): tools `emit_learning_article`, `emit_learning_slides`, `emit_learning_infographic`, `emit_learning_all`, `emit_custom_topic`.
+- [`testService.js`](src/lib/testService.js): tool `emit_quiz_questions`.
+- [`KnowledgeGraph.jsx:297`](src/components/admin/KnowledgeGraph.jsx): tool `emit_graph_actions`.
+
+### 2.3 Prompt caching
+
+Pass `system` as an array of blocks so the stable parts can be cached:
+
+```js
+system: [
+  { type:'text', text: STABLE_SYSTEM_HEADER, cache_control: { type:'ephemeral' } },
+  { type:'text', text: dynamicPart }, // not cached
+],
+```
+
+Apply caching to:
+- Extraction `SYSTEM_PROMPT` and `HANDBOOK_SYSTEM_PROMPT` (both fully stable → cache the whole block).
+- R42 system prompt — split into three blocks: stable preamble (cached), KB context (cached *only* while the graph hasn't changed; bust by appending a short hash of the topic IDs+labels — Phase 5 details), and per-turn role line (not cached).
+
+### 2.4 Patch-based learning refinement
+
+Refactor `refineLearningContent` ([`learningService.js:147`](src/lib/learningService.js:147)) from "return the full updated JSON" to **patch operations** via tools:
+
+- `set_section(heading: string, body: string)` — replace one section by heading match.
+- `add_section(heading: string, body: string, position: 'start'|'end')`.
+- `remove_section(heading: string)`.
+- `replace_takeaways(items: string[])`.
+- `set_intro(intro: string)`.
+
+Apply patches client-side to the cached object. Re-validate against `learningArticleSchema` after patching; reject the whole turn if invalid.
+
+### Phase 2 acceptance criteria
+
+- [ ] No regex JSON extraction left in `src/`: `grep -rn "match(/\\\\{\\[\\\\s\\\\S\\]\\*\\\\}/)" src/` returns nothing.
+- [ ] Token usage telemetry shows `cache_read_input_tokens > 0` on the second extraction call within 5 minutes (cache hit).
+- [ ] Re-running extraction on a known source produces the same topic count ±10% as before this phase.
+- [ ] `refineLearningContent` round-trip ("make the intro shorter") produces only the changed section in the diff against the prior cached content.
+
+---
+
+## Phase 3 — Extraction quality
+
+**Goal:** fewer near-duplicate topics, no silent truncation, adaptive throttling, unified vocabulary.
+
+### 3.1 Sentence-aware chunking with overlap
+
+Replace `chunkText` in [`extractionPipeline.js:87`](src/lib/extractionPipeline.js:87):
+
+- Target **~2000 input tokens per chunk**. Approximate as `chars / 4`. Configurable via `MAX_CHUNK_CHARS = 8000`.
+- **200-token overlap** between chunks (`OVERLAP_CHARS = 800`).
+- Split on sentence boundaries (`/(?<=[.!?])\s+/`) first; fall back to paragraph boundary if a sentence is too long; never produce a chunk larger than `MAX_CHUNK_CHARS`.
+- Add a guard: if a single sentence exceeds `MAX_CHUNK_CHARS`, hard-split at character boundary and log a warning.
+
+### 3.2 Stateful extraction
+
+Before each chunk after the first, prepend to the user message:
+
+```text
+Already-extracted topic IDs (do NOT create new IDs for these — reuse them if the same concept appears here):
+- software-engineer
+- onboarding-buddy
+...
+```
+
+Cap the list at 200 IDs by recency to keep token cost bounded. The model will then reuse IDs instead of inventing variants like `software-developer`.
+
+### 3.3 Adaptive throttling
+
+Replace the hard `setTimeout(r, 12000)` in [`extractionPipeline.js:127`](src/lib/extractionPipeline.js:127) and the 15s sleep in [`KnowledgeGraph.jsx:274`](src/components/admin/KnowledgeGraph.jsx:274) with a shared token-bucket limiter in `src/lib/llmRetry.js`:
+
+```js
+export const extractionLimiter = createLimiter({ rps: 5/60, burst: 1 }); // 5 req/min
+// usage: await extractionLimiter.acquire();
+```
+
+`callLLM` accepts an optional `limiter` param that it `await`s before fetch. On 429 with `Retry-After`, the limiter is paused for that duration.
+
+### 3.4 Preserve admin-edited relevance
+
+Add a migration introducing `relevance_locked: bool` on `topics`. Set it to `true` whenever an admin edits `learning_relevance` via the UI ([`KnowledgeGraph.jsx`](src/components/admin/KnowledgeGraph.jsx) edit handler — locate by searching for `setLearningRelevance` or the relevance form field).
+
+In `mergeKnowledgeGraph` ([`extractionPipeline.js:167`](src/lib/extractionPipeline.js:167)), when `relevance_locked`, never overwrite `learning_relevance`.
+
+### 3.5 Unify relation vocabulary
+
+Pick **one** set: `related_to | depends_on | part_of | executed_by`. Migrate:
+
+- `HANDBOOK_SYSTEM_PROMPT` ([`extractionPipeline.js:42`](src/lib/extractionPipeline.js:42)) — change `executes` to `executed_by` and swap the source/target in the prompt example.
+- Write a one-shot migration script `pb_migrations/<timestamp>_normalize_relation_types.js` that rewrites any existing `executes` relation to `executed_by` and swaps `source ↔ target`.
+- Verify R42's `validateDelta` ([`rag.js:108`](src/components/chat/rag.js:108)) already enforces this set (it does) — no change needed there.
+
+### 3.6 Cancellation
+
+Add a "Cancel" button to the source-processing UI in `ContentManager.jsx` / `UploadZone.jsx` (locate the one that displays extraction progress). Wire it to abort the in-flight `callLLM` via the `signal` it receives. On cancel, set source status to `cancelled` (add to status enum migration).
+
+### Phase 3 acceptance criteria
+
+- [ ] Running extraction twice on the same `sources/ROLES.md` produces zero new topics on the second run (idempotency through reused IDs).
+- [ ] Locked-relevance topics survive re-extraction.
+- [ ] No fixed `setTimeout` ≥ 5s anywhere in `src/` (`grep -rn "setTimeout" src/`).
+- [ ] Cancelling an extraction mid-run leaves the source in `cancelled` state, not `processing`.
+- [ ] `pb_migrations` includes the relation-vocabulary normalization and the `relevance_locked` column.
+
+---
+
+## Phase 4 — Quiz & content quality
+
+**Goal:** quiz questions are positionally unbiased, deduped, and difficulty-tagged. Random helpers are correct.
+
+### 4.1 Random helpers
+
+Create `src/lib/random.js`:
+
+```js
+export function shuffle(arr) { /* Fisher–Yates, returns NEW array */ }
+export function sample(arr, n) { /* unbiased sample without replacement */ }
+export function pickInt(min, maxInclusive) { /* uniform integer */ }
+```
+
+Replace every `.sort(() => 0.5 - Math.random())` with `shuffle(arr)`:
+
+- [`testService.js:122`](src/lib/testService.js:122) and [`testService.js:163`](src/lib/testService.js:163).
+- Any other site found by `grep -rn "0.5 - Math.random()" src/`.
+
+### 4.2 Debias `correctIndex` in quiz prompt
+
+In [`testService.js:81`](src/lib/testService.js:81):
+
+- Change the example in the prompt to use `"correctIndex": 2` (not 0).
+- Add to the prompt: *"Distribute correctIndex roughly evenly across 0, 1, 2, and 3. Do not place the correct answer at the same position more than 4 out of 10 times."*
+- After parsing, run a check: if more than 50% of the batch share the same `correctIndex`, log a warning and re-roll up to 2 times.
+
+### 4.3 Difficulty field
+
+- Add to `quizQuestionsSchema`: `difficulty: 'easy'|'medium'|'hard'`.
+- Update the prompt to require difficulty on every question (current prompt says "4 easy, 4 medium, 2 hard" but never tagged — now tag it).
+- Migration: add `difficulty` to the `quiz_banks.questions[]` element. PocketBase stores `questions` as JSON, so the migration is a no-op at the column level; older records get `difficulty: 'medium'` on read (add a normalizer in `db.js`).
+
+### 4.4 Question dedup
+
+In `forceGenerateTopicQuestions` ([`testService.js:65`](src/lib/testService.js:65)):
+
+- Normalize question text (lowercase, strip punctuation, collapse whitespace) → `normKey`.
+- Before persisting, drop any new question whose `normKey` matches an existing bank question.
+- Log dropped duplicates with `console.debug('[quiz] dropped duplicate:', text)`.
+
+### 4.5 Quality gate
+
+In the same function, after schema validation:
+
+- Reject the whole batch if any question has fewer than 4 distinct options.
+- Reject if any option contains `"all of the above"`, `"none of the above"`, `"both A and B"` (case-insensitive).
+- Reject if `explanation.trim().length < 20`.
+- Surface the rejection to the admin UI with a "Retry" button.
+
+### 4.6 Custom topic ID hygiene
+
+In `generateCustomTopic` ([`learningService.js:177`](src/lib/learningService.js:177)):
+
+- Generate kebab-case ID from the polished label, not `Date.now()`.
+- Collision check against existing topics (append `-2`, `-3`, … if needed).
+- Default `learning_relevance: 'standard'` when the model omits it.
+
+### Phase 4 acceptance criteria
+
+- [ ] No `.sort(() => 0.5 - Math.random())` anywhere in `src/`.
+- [ ] Sample of 50 fresh quiz questions across 5 topics: no position holds >40% of correct answers.
+- [ ] Re-running quiz generation for the same topic does not grow the bank with semantic duplicates.
+- [ ] Custom topics created via R42 use kebab-case IDs and pass schema validation.
+
+---
+
+## Phase 5 — R42 retrieval & telemetry
+
+**Goal:** R42 stops shipping the entire KG every turn; conversations are bounded; every call is logged.
+
+### 5.1 TF-IDF retrieval in the browser
+
+Create `src/lib/retrieval.js`:
+
+```js
+export function buildIndex(topics) { /* TF-IDF over label + description */ }
+export function retrieveTopK(index, query, k = 10) { /* returns Topic[] */ }
+```
+
+Implementation: a small dependency-free TF-IDF — tokenize on `/[a-zA-Z0-9-]+/`, lowercase, drop stopwords (Dutch + English short list). Cache the index on the `topics` array reference. About 100 lines.
+
+### 5.2 Rewrite `buildKbContext`
+
+In [`rag.js:11`](src/components/chat/rag.js:11):
+
+- Use `retrieveTopK(index, userMessage, 10)` to pick which topics go into the system prompt.
+- Always include any topic whose ID or label is mentioned verbatim (existing behaviour).
+- Drop the full "every topic" dump.
+- Return `{ context, retrievedTopics, allTopics }` — `validateDelta` continues to use `allTopics`.
+
+### 5.3 `lookup_topic` tool
+
+Add a second R42 tool in [`prompts.js`](src/components/chat/prompts.js):
+
+```js
+export const LOOKUP_TOPIC_TOOL = {
+  name: 'lookup_topic',
+  description: 'Fetch the full description and any deeper learning content for a topic. Use when the retrieved context does not contain enough to answer.',
+  input_schema: { type:'object', properties: { id:{type:'string'} }, required:['id'] },
+};
+```
+
+In `useChat.js`, when the model emits a `lookup_topic`, fetch via `db.getTopics()` + `db.getContent(id)` and append a `tool_result` block, then call `callLLM` again with the extended messages. Cap to **3 lookup hops per turn** to avoid loops.
+
+### 5.4 R42 prompt cache busting
+
+KB-context block is cached as ephemeral. The cache key is automatic per text, so any change busts it. **Append a stable suffix** to the KB block: `"\n[kb_hash: <first-8-chars-of-sha256-of-sorted-topic-ids>]"`. This guarantees the block content changes the moment a topic is added/removed, even if the included topic list looks similar.
+
+### 5.5 Conversation truncation
+
+In `useChat.js`:
+
+- Keep last **12 turns** verbatim in `apiMessages`.
+- Older messages: if more than 12 turns exist, summarize the older ones with a cheap (`tier: 'fast'`) call and prepend a single `{ role:'system'-equivalent inside the user history is not allowed by Anthropic — instead prepend a user/assistant pair }` block, OR simply drop older turns. **Default: drop older turns and prepend a one-line `assistant` message saying "(earlier conversation truncated)".** Summarization is an optional follow-up.
+
+### 5.6 `llm_calls` collection
+
+Add a migration `pb_migrations/<timestamp>_created_llm_calls.js` for collection `llm_calls`:
+
+| Field | Type | Notes |
+|---|---|---|
+| `task` | text | e.g. `extract.source` |
+| `model` | text | resolved model ID |
+| `tier` | text | `fast` / `standard` / `reasoning` |
+| `duration_ms` | number | |
+| `input_tokens` | number | |
+| `output_tokens` | number | |
+| `cache_read_tokens` | number | |
+| `cache_create_tokens` | number | |
+| `stop_reason` | text | |
+| `ok` | bool | |
+| `error_msg` | text | nullable |
+
+Wire `callLLM` to write to it (best-effort, never throws). Add a minimal `Admin → Diagnostics` view that shows the last 100 calls and aggregate cost (using public Anthropic prices in a constant; refresh manually).
+
+### 5.7 R42 defaults
+
+- `temperature: 0.3` for R42 chat in `callLLM`.
+- `maxTokens: 2048` for R42 (text + tool budget).
+
+### Phase 5 acceptance criteria
+
+- [ ] R42 system prompt is ≤ 4000 tokens regardless of KG size (verify on a graph with 200+ topics).
+- [ ] Adding a topic in the admin UI causes the **next** R42 call to show `cache_read_tokens === 0` for the KB block, then subsequent calls to show non-zero.
+- [ ] R42 successfully answers a question about a topic whose ID was not in the user's message by emitting `lookup_topic` (manual verification).
+- [ ] `Admin → Diagnostics` shows recent LLM calls with model, tokens, duration.
+
+---
+
+## Phase 6 — Eval harness (optional, high-leverage)
+
+**Goal:** prompt or model changes can be measured before they ship.
+
+### 6.1 Golden sets
+
+Create `evals/` at the repo root:
+
+```
+evals/
+  extraction/
+    cases/
+      roles-handbook.txt
+      governance-handbook.txt
+    expected/
+      roles-handbook.json       # { mustContain: ['software-engineer', ...], minTopics: 20 }
+      governance-handbook.json
+  quiz/
+    cases/
+      software-engineer.json    # topic to generate quiz for
+    rubric.md                   # human-readable quality rubric
+  chat/
+    scripts/
+      ask-about-known-topic.json
+      ask-about-unknown-topic.json
+      propose-new-role.json
+```
+
+10 extraction cases, 5 quiz topics, 10 chat scripts is enough to start.
+
+### 6.2 Runner
+
+`evals/run.mjs` — Node script that:
+
+1. Loads each case.
+2. Invokes the same code path the app uses (import from `src/lib/*` directly; reuse the simulation toggle off).
+3. Compares against expectations:
+   - **Extraction:** `mustContain` IDs present? `minTopics` met? No `stop_reason: 'max_tokens'`?
+   - **Quiz:** distribution of `correctIndex`, mean explanation length, banned phrases.
+   - **Chat:** for each script, did the expected tool fire? Did the reply contain expected anchors?
+4. Writes `evals/results/<ISO-timestamp>.json` and a Markdown diff against the previous baseline.
+
+Add `npm run eval` to `package.json`.
+
+### 6.3 Prompt versioning
+
+In each prompt module, export `PROMPT_VERSION = '2026-05-20-001'`. Persist it on the artifact (`content.data.prompt_version`, `quiz_banks.questions[i].prompt_version`, `topics.metadata.prompt_version`). Add an admin button "Mark stale content for regeneration" that lists artifacts whose version is older than the current.
+
+### Phase 6 acceptance criteria
+
+- [ ] `npm run eval` runs end-to-end against the live API and produces a result file.
+- [ ] CI (or a manual check) runs evals on every change to `src/lib/llm.js`, prompts, or schemas.
+- [ ] Each AI-generated artifact in PocketBase carries a `prompt_version` and is filterable by it.
+
+---
+
+## Cross-phase verification checklist
+
+After every phase, run this short checklist before merging:
+
+1. **Build:** `npm run build` succeeds.
+2. **Lint:** `npm run lint` clean.
+3. **Tests:** `npm run test` green.
+4. **Smoke flows** in dev (simulation off, real API key):
+   - Add a source via `Admin → Sources`, extract, verify topics appear.
+   - Visit `Leren` for the current week, generate article. Then slides.
+   - Visit `Testen`, generate weekly quiz. Submit. Score lands on leaderboard.
+   - Open R42, ask a known and an unknown question; propose a new topic; admin accepts it.
+   - Run `Admin → Knowledge Graph → Analyze & Optimize Graph`.
+   - Run `Admin → Knowledge Graph → Sync Handbook` (small repo or mock).
+5. **Simulation toggle:** flip simulation mode on and confirm no real API calls happen (Network tab).
+
+---
+
+## Rollback strategy
+
+Every phase is shippable on its own. If a phase introduces a regression:
+
+- **Phase 1–2:** `git revert` the merge commit. `api.js` retains the legacy interface, so callers that haven't migrated still work.
+- **Phase 3:** revert + the relation-vocabulary migration is reversible (rerun the reverse swap).
+- **Phase 4:** revert; quiz schema additions are forward-compatible (older readers ignore `difficulty`).
+- **Phase 5:** revert + drop the `llm_calls` collection if undesired.
+- **Phase 6:** purely additive; remove the `evals/` folder if abandoned.
+
+Never `git push --force` to `main`. PR-per-phase.
+
+---
+
+## Out of scope (do not do as part of this plan)
+
+- Replacing PocketBase. Stays as-is.
+- Server-side embeddings or a vector store. Phase 5 deliberately uses in-browser TF-IDF.
+- Streaming responses. Mentioned as a future improvement; not in this plan.
+- Multi-tenant changes. The platform serves one company.
+- UI redesign of the Admin pages beyond what each phase requires.
+
+---
+
+## Glossary
+
+- **Tier** — coarse model class (`fast`/`standard`/`reasoning`) mapped to a concrete Anthropic model ID via admin settings.
+- **Tool use** — Anthropic's structured-output mechanism. The model emits a `tool_use` content block whose `input` is schema-valid JSON.
+- **Prompt caching** — Anthropic feature where blocks marked `cache_control: { type:'ephemeral' }` are reused across requests at lower input cost. 5-minute TTL.
+- **TF-IDF** — Term-frequency / inverse-document-frequency. A classic IR scoring function used here as a cheap retrieval signal.
+- **KB context** — The block in R42's system prompt that lists topics and relations from the knowledge graph.
+- **Delta** — A proposed addition to the knowledge graph emitted by R42 via the `propose_graph_delta` tool.

src/components/admin/KnowledgeGraph.jsx

@@ -4,7 +4,7 @@ import { Trash2, Edit2, Save, X, RefreshCw, AlertCircle, Plus, Link as LinkIcon
 import * as db from '../../lib/db';
 import { anthropicApi } from '../../lib/api';
 import { analyzeHandbookDelta } from '../../lib/extractionPipeline';
-import { getRepoFolder, getFileContent } from '../../lib/giteaService';
+import { getRepoFolder, getFileContent } from '../../lib/githubService';
 import Button from '../ui/Button';
 import SuggestionsQueue from './SuggestionsQueue';