feat: phase 2 of AI pipeline hardening — tool-based structured outputs + prompt caching
Every structured-output call now uses an Anthropic tool instead of
parsing JSON out of free-form prose, and stable system prompts are
sent as cacheable blocks. Behaviour-equivalent to phase 1 from the
caller's point of view; the savings show up in token usage and in the
absence of "AI returned non-JSON response" failure modes.
* src/lib/llmTools.js — single source of truth for tool definitions:
emit_knowledge_graph, emit_handbook_delta, emit_learning_article /
_slides / _infographic / _all, emit_custom_topic, emit_quiz_questions,
emit_graph_actions, plus five article-patch tools (set_intro,
set_section, add_section, remove_section, replace_takeaways).
* src/lib/articlePatches.js — pure applyArticlePatches +
applyAndValidate; rebuilds the article from a sequence of patch tool
calls and re-validates against learningArticleSchema. set_section
falls back to appending when no matching heading exists so the
model's intent is preserved rather than silently dropped.
* src/lib/llmSchemas.js — Zod schemas for the five patch ops,
registered in toolSchemaRegistry so callLLM validates them
automatically.
* src/lib/llm.js — simulation mode now returns a tool_use stub matching
toolChoice.name, so the UI keeps working with Simulation Mode on
after the structured-output migration.
* src/lib/extractionPipeline.js — processSourceText and
analyzeHandbookDelta migrated to callLLM + tool use. System prompts
sent as { cache_control: ephemeral } blocks. Handbook results pass
through normalizeHandbookResult to collapse legacy "executes"
relations into executed_by with swapped source/target.
* src/lib/learningService.js — generateLearningContent picks the right
tool per selectedType; generateCustomTopic uses emit_custom_topic;
refineLearningContent now drives the five patch tools with
toolChoice 'any' and rejects the whole turn if the patched article
fails validation. Article-only refinement is intentional for phase 2;
refining a topic without an article surfaces a clear error.
* src/lib/testService.js — quiz generation via emit_quiz_questions.
* src/components/admin/KnowledgeGraph.jsx — analyzeGraph routed through
the reasoning tier (Opus) since graph-wide consolidation benefits
from a stronger reasoner.
* src/components/chat/prompts.js — buildSystemPrompt now returns three
text blocks: stable preamble (cached), KB context (cached, hash-bust
deferred to phase 5), per-turn user/admin tail (uncached).
* src/lib/__tests__/ — 13 new tests covering each patch op, multi-op
sequencing, post-patch validation failure, and tool/registry shape.
Acceptance: lint and 45/45 tests green; build succeeds; no
`match(/\{[\s\S]*\}/)` JSON extraction left in src/. Live verification
of cache hits on a second extraction within 5 minutes is deferred to
manual smoke testing — needs real `/api/anthropic` traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,11 +1,15 @@
|
||||
import { anthropicApi } from './api';
|
||||
import * as db from './db';
|
||||
import { callLLM } from './llm';
|
||||
import { EMIT_QUIZ_QUESTIONS_TOOL } from './llmTools';
|
||||
import { getCurriculumTopic, getQuarterForWeek } from './curriculumService';
|
||||
|
||||
const QUIZ_SYSTEM = `You are a quiz generator for Respellion, an internal IT company learning platform.
|
||||
You generate multiple-choice questions to test employee knowledge on specific topics.
|
||||
Always write in clear, professional English.
|
||||
ALWAYS return valid JSON only — no markdown code blocks, no extra text.`;
|
||||
|
||||
Emit questions through the emit_quiz_questions tool. Each question has exactly four options; correctIndex is 0-based; mix difficulty roughly 4 easy / 4 medium / 2 hard.`;
|
||||
|
||||
const cachedSystem = (text) => [{ type: 'text', text, cache_control: { type: 'ephemeral' } }];
|
||||
|
||||
async function selectTestTopics(userId, weekNumber) {
|
||||
const allTopics = await db.getTopics();
|
||||
@@ -66,45 +70,31 @@ export async function getCachedQuiz(userId, weekNumber) {
|
||||
export async function forceGenerateTopicQuestions(topic, count = 10) {
|
||||
let bank = await db.getQuizBank(topic.id);
|
||||
|
||||
const prompt = `Generate exactly ${count} multiple-choice quiz questions based on this knowledge topic:
|
||||
const prompt = `Generate exactly ${count} multiple-choice quiz questions for this knowledge topic and emit them via the emit_quiz_questions tool:
|
||||
|
||||
Topic: ${topic.label}
|
||||
Type: ${topic.type}
|
||||
Description: ${topic.description}
|
||||
|
||||
Return ONLY a JSON object with this structure:
|
||||
{
|
||||
"questions": [
|
||||
{
|
||||
"id": "unique-id-string",
|
||||
"question": "The question text",
|
||||
"topicLabel": "${topic.label}",
|
||||
"options": ["A) First option", "B) Second option", "C) Third option", "D) Fourth option"],
|
||||
"correctIndex": 0,
|
||||
"explanation": "A clear 1-2 sentence explanation of why the correct answer is correct."
|
||||
}
|
||||
]
|
||||
}
|
||||
Options must be prefixed "A) ", "B) ", "C) ", "D) ". Make questions specific and practical, not trivial.`;
|
||||
|
||||
Rules:
|
||||
- Each question must have exactly 4 options.
|
||||
- correctIndex is 0-based (0=A, 1=B, 2=C, 3=D).
|
||||
- Mix difficulty: 4 easy, 4 medium, 2 hard.
|
||||
- Make questions specific and practical, not trivial.`;
|
||||
const result = await callLLM({
|
||||
task: 'quiz.generate',
|
||||
tier: 'standard',
|
||||
system: cachedSystem(QUIZ_SYSTEM),
|
||||
user: prompt,
|
||||
tools: [EMIT_QUIZ_QUESTIONS_TOOL],
|
||||
toolChoice: { type: 'tool', name: EMIT_QUIZ_QUESTIONS_TOOL.name },
|
||||
maxTokens: 4096,
|
||||
});
|
||||
|
||||
const responseText = await anthropicApi.generateContent(QUIZ_SYSTEM, prompt);
|
||||
let newQuestions;
|
||||
try {
|
||||
const jsonMatch = responseText.match(/\{[\s\S]*\}/);
|
||||
const parsed = JSON.parse(jsonMatch ? jsonMatch[0] : responseText);
|
||||
newQuestions = parsed.questions || [];
|
||||
newQuestions.forEach(q => {
|
||||
q.id = `${topic.id}-${Math.random().toString(36).substr(2, 9)}`;
|
||||
});
|
||||
} catch (e) {
|
||||
console.error('Failed to generate questions for topic', topic.label, e);
|
||||
throw new Error(`Could not generate questions for ${topic.label}`, { cause: e });
|
||||
}
|
||||
const emitted = result.toolUses[0]?.input;
|
||||
if (!emitted) throw new Error(`Could not generate questions for ${topic.label}`);
|
||||
|
||||
const newQuestions = (emitted.questions || []).map(q => ({
|
||||
...q,
|
||||
id: `${topic.id}-${Math.random().toString(36).slice(2, 11)}`,
|
||||
}));
|
||||
|
||||
bank = [...bank, ...newQuestions];
|
||||
await db.setQuizBank(topic.id, bank);
|
||||
|
||||
Reference in New Issue
Block a user