feat: phase 2 of AI pipeline hardening — tool-based structured outputs + prompt caching

Every structured-output call now uses an Anthropic tool instead of
parsing JSON out of free-form prose, and stable system prompts are
sent as cacheable blocks. Behaviour-equivalent to phase 1 from the
caller's point of view; the savings show up in token usage and in the
absence of "AI returned non-JSON response" failure modes.

* src/lib/llmTools.js — single source of truth for tool definitions:
  emit_knowledge_graph, emit_handbook_delta, emit_learning_article /
  _slides / _infographic / _all, emit_custom_topic, emit_quiz_questions,
  emit_graph_actions, plus five article-patch tools (set_intro,
  set_section, add_section, remove_section, replace_takeaways).
* src/lib/articlePatches.js — pure applyArticlePatches +
  applyAndValidate; rebuilds the article from a sequence of patch tool
  calls and re-validates against learningArticleSchema. set_section
  falls back to appending when no matching heading exists so the
  model's intent is preserved rather than silently dropped.
* src/lib/llmSchemas.js — Zod schemas for the five patch ops,
  registered in toolSchemaRegistry so callLLM validates them
  automatically.
* src/lib/llm.js — simulation mode now returns a tool_use stub matching
  toolChoice.name, so the UI keeps working with Simulation Mode on
  after the structured-output migration.
* src/lib/extractionPipeline.js — processSourceText and
  analyzeHandbookDelta migrated to callLLM + tool use. System prompts
  sent as { cache_control: ephemeral } blocks. Handbook results pass
  through normalizeHandbookResult to collapse legacy "executes"
  relations into executed_by with swapped source/target.
* src/lib/learningService.js — generateLearningContent picks the right
  tool per selectedType; generateCustomTopic uses emit_custom_topic;
  refineLearningContent now drives the five patch tools with
  toolChoice 'any' and rejects the whole turn if the patched article
  fails validation. Article-only refinement is intentional for phase 2;
  refining a topic without an article surfaces a clear error.
* src/lib/testService.js — quiz generation via emit_quiz_questions.
* src/components/admin/KnowledgeGraph.jsx — analyzeGraph routed through
  the reasoning tier (Opus) since graph-wide consolidation benefits
  from a stronger reasoner.
* src/components/chat/prompts.js — buildSystemPrompt now returns three
  text blocks: stable preamble (cached), KB context (cached, hash-bust
  deferred to phase 5), per-turn user/admin tail (uncached).
* src/lib/__tests__/ — 13 new tests covering each patch op, multi-op
  sequencing, post-patch validation failure, and tool/registry shape.

Acceptance: lint and 45/45 tests green; build succeeds; no
`match(/\{[\s\S]*\}/)` JSON extraction left in src/. Live verification
of cache hits on a second extraction within 5 minutes is deferred to
manual smoke testing — needs real `/api/anthropic` traffic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

src/lib/extractionPipeline.js

@@ -1,85 +1,62 @@
-import { anthropicApi } from './api';
 import * as db from './db';
+import { callLLM } from './llm';
+import { EMIT_KNOWLEDGE_GRAPH_TOOL, EMIT_HANDBOOK_DELTA_TOOL } from './llmTools';
+import { normalizeHandbookResult } from './llmSchemas';
 
-const SYSTEM_PROMPT = `You are an AI knowledge extractor for Respellion, an IT company built on radical transparency.
-You receive a source text. Your task is to extract all core concepts, roles, and processes from the text, and return them as a structured JSON Knowledge Graph.
-Facts should be integrated into the descriptions of the other labels and NOT be extracted as unique topics.
+const EXTRACTION_SYSTEM_PROMPT = `You are an AI knowledge extractor for Respellion, an IT company built on radical transparency.
+You receive a source text. Extract every distinct concept, role, and process from it and emit them through the emit_knowledge_graph tool.
 
 CRITICAL INSTRUCTIONS FOR COMPLETENESS:
-- You must extract EVERY SINGLE distinct role, process, and concept described or mentioned in the source text. 
-- DO NOT summarize, skip, truncate, or omit any items. 
-- If the document contains 29 roles, your JSON topics array must contain exactly 29 role topics. 
-- Completeness is of paramount importance. Failing to extract all topics will result in loss of critical company knowledge.
-- Keep descriptions concise (max 3 sentences) to ensure you have enough output tokens to list everything.
+- Extract EVERY SINGLE distinct role, process, and concept described or mentioned in the source text.
+- DO NOT summarise, skip, truncate, or omit any items.
+- If the document contains 29 roles, the topics array must contain exactly 29 role topics.
+- Completeness is paramount. Failing to extract all topics loses critical company knowledge.
+- Facts should be integrated into the descriptions of other topics — never extracted as standalone topics.
+- Keep descriptions concise (max 3 sentences) so the response fits.
 
-You MUST assign a learning_relevance to each topic:
-- "core": Fundamental company knowledge.
-- "standard": Normal learning topics.
-- "peripheral": Good to know, but low priority.
-- "exclude": Pure operational reference material (e.g., printer guides, wifi passwords) that should NEVER be tested.
+Topic IDs are lowercase kebab-case slugs specific to the topic (e.g. "software-engineer", "data-quality-review"). Do not use generic IDs like "role-1" or "concept-2".
 
-ALWAYS return a valid JSON object in the following format:
-{
-  "topics": [
-    {
-      "id": "a-unique-lowercase-kebab-case-slug-specific-to-this-topic (e.g., 'software-engineer' or 'data-quality-review'). DO NOT use generic IDs like 'role-1' or 'concept-2'.",
-      "label": "Topic title",
-      "type": "concept | role | process",
-      "description": "A concise, clear explanation of max 3 sentences.",
-      "learning_relevance": "core | standard | peripheral | exclude"
-    }
-  ],
-  "relations": [
-    {
-      "source": "topic-id-1",
-      "target": "topic-id-2",
-      "type": "related_to | depends_on | part_of | executed_by"
-    }
-  ]
-}
-Return JSON only. No markdown blocks or other text.`;
+Assign a learning_relevance to every topic:
+- "core": fundamental company knowledge.
+- "standard": normal learning topics.
+- "peripheral": good to know, low priority.
+- "exclude": pure operational reference (printer guides, wifi passwords) that should never be tested.
 
-const HANDBOOK_SYSTEM_PROMPT = `You are analyzing an update to the Respellion Employee Handbook.
-Your task is to identify changes and extract structural knowledge.
+Relation types: related_to | depends_on | part_of | executed_by.
+`;
 
-CRITICAL INSTRUCTION:
-You must explicitly identify and create relations between Roles, Processes, and Concepts.
-Every Process must have a Role attached (who does it).
-Every Concept must have a relation to a Process or Role.
+const HANDBOOK_SYSTEM_PROMPT = `You are analysing an update to the Respellion Employee Handbook. Emit the extracted topics and relations through the emit_handbook_delta tool.
 
-You MUST assign a learning_relevance to each topic:
-- "core": Fundamental company knowledge.
-- "standard": Normal learning topics.
-- "peripheral": Good to know, but low priority.
-- "exclude": Pure operational reference material (e.g., printer guides, wifi passwords) that should NEVER be tested.
+CRITICAL INSTRUCTIONS:
+- Every process must have a role attached (the role that executes it).
+- Every concept must connect to a process or role.
+- Mark handbook topics with metadata.source = "github_handbook".
+- Assign learning_relevance using the same scale as extraction: core | standard | peripheral | exclude.
 
-Return a JSON object:
-{
-  "topics": [
-    { "id": "...", "label": "...", "type": "role | process | concept", "description": "...", "learning_relevance": "standard", "metadata": { "source": "github_handbook" } }
-  ],
-  "relations": [
-    { "source": "role-id", "target": "process-id", "type": "executes | related_to | depends_on | part_of", "description": "Brief metadata about this specific relation" }
-  ]
-}
-Return JSON only. No markdown blocks or other text.`;
+Relation types: related_to | depends_on | part_of | executed_by. (Legacy "executes" relations are normalised by the client into executed_by with source/target swapped.)
+`;
+
+const cachedSystem = (text) => [{ type: 'text', text, cache_control: { type: 'ephemeral' } }];
 
 export async function analyzeHandbookDelta(fileContent, filePath) {
-  const responseText = await anthropicApi.generateContent(HANDBOOK_SYSTEM_PROMPT, `Analyze the following handbook file update (${filePath}):\n\n${fileContent}`);
+  const result = await callLLM({
+    task: 'extract.handbook',
+    tier: 'standard',
+    system: cachedSystem(HANDBOOK_SYSTEM_PROMPT),
+    user: `Analyze the following handbook file update (${filePath}):\n\n${fileContent}`,
+    tools: [EMIT_HANDBOOK_DELTA_TOOL],
+    toolChoice: { type: 'tool', name: EMIT_HANDBOOK_DELTA_TOOL.name },
+    maxTokens: 8192,
+  });
 
-  let extractedData;
-  try {
-    const jsonMatch = responseText.match(/\{[\s\S]*\}/);
-    const jsonStr = jsonMatch ? jsonMatch[0] : responseText;
-    extractedData = JSON.parse(jsonStr);
-  } catch (e) {
-    console.error('[Pipeline] AI returned non-JSON response for handbook delta:', responseText?.substring(0, 500));
-    throw new Error(`AI response was not valid JSON. The model responded with: "${responseText?.substring(0, 120)}..."`, { cause: e });
-  }
+  const raw = result.toolUses[0]?.input;
+  if (!raw) throw new Error('Handbook extraction did not emit a tool result.');
+  const extractedData = normalizeHandbookResult(raw);
 
   await mergeKnowledgeGraph(extractedData);
   return { success: true, data: extractedData };
 }
+
 function chunkText(text, maxChunkSize = 4000) {
   const paragraphs = text.split(/\n+/);
   const chunks = [];
@@ -98,7 +75,6 @@ function chunkText(text, maxChunkSize = 4000) {
 }
 
 export async function processSourceText(textContent, sourceName) {
-  // Deduplicate: skip if a source with the same name was already successfully processed
   const existing = await db.getSources();
   const alreadyDone = existing.find(
     s => s.name === sourceName && s.status === 'completed'
@@ -124,21 +100,18 @@ export async function processSourceText(textContent, sourceName) {
       }
 
       console.log(`[Pipeline] Processing chunk ${i + 1}/${chunks.length} (${chunks[i].length} chars)...`);
-      const responseText = await anthropicApi.generateContent(
-        SYSTEM_PROMPT, 
-        `Analyze this part of the document (${i + 1}/${chunks.length}):\n\n${chunks[i]}`
-      );
-      console.log(`[Pipeline] Raw AI response for chunk ${i + 1}:`, responseText);
+      const result = await callLLM({
+        task: 'extract.source',
+        tier: 'standard',
+        system: cachedSystem(EXTRACTION_SYSTEM_PROMPT),
+        user: `Analyze this part of the document (${i + 1}/${chunks.length}):\n\n${chunks[i]}`,
+        tools: [EMIT_KNOWLEDGE_GRAPH_TOOL],
+        toolChoice: { type: 'tool', name: EMIT_KNOWLEDGE_GRAPH_TOOL.name },
+        maxTokens: 8192,
+      });
 
-      let extractedData;
-      try {
-        const jsonMatch = responseText.match(/\{[\s\S]*\}/);
-        const jsonStr = jsonMatch ? jsonMatch[0] : responseText;
-        extractedData = JSON.parse(jsonStr);
-      } catch (e) {
-        console.error(`[Pipeline] AI returned non-JSON response for chunk ${i + 1}:`, responseText?.substring(0, 500));
-        throw new Error(`AI response for chunk ${i + 1} was not valid JSON.`, { cause: e });
-      }
+      const extractedData = result.toolUses[0]?.input;
+      if (!extractedData) throw new Error(`Extraction did not emit a tool result for chunk ${i + 1}.`);
 
       if (extractedData.topics && Array.isArray(extractedData.topics)) {
         allExtractedTopics.push(...extractedData.topics);
@@ -148,7 +121,6 @@ export async function processSourceText(textContent, sourceName) {
       }
     }
 
-    // Merge everything together
     await mergeKnowledgeGraph({ topics: allExtractedTopics, relations: allExtractedRelations });
     await db.updateSourceStatus(sourceId, 'completed');
 
@@ -169,13 +141,11 @@ async function mergeKnowledgeGraph(newData) {
   if (newData.topics && Array.isArray(newData.topics)) {
     for (const t of newData.topics) {
       if (topicsMap.has(t.id)) {
-        // Upsert: merge new data into existing topic
         const existing = topicsMap.get(t.id);
-        topicsMap.set(t.id, { 
-          ...existing, 
-          ...t, 
-          // Keep existing description if new one is empty, or combine them if needed. Here we prefer the new one.
-          description: t.description || existing.description 
+        topicsMap.set(t.id, {
+          ...existing,
+          ...t,
+          description: t.description || existing.description,
         });
       } else {
         topicsMap.set(t.id, t);