Add comprehensive documentation for employee learning platform
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
This commit is contained in:
RaymondVerhoef
@@ -1,573 +0,0 @@
|
||||
# AI Pipeline Hardening — Implementation Plan
|
||||
|
||||
> **Audience:** an AI agent executing this plan against the Respellion Learning Platform.
|
||||
> **Owner before this work:** Raymond Verhoef (rve@respellion.nl).
|
||||
> **Source of truth for repo conventions:** [`AI_AGENT.md`](AI_AGENT.md). Read it before starting.
|
||||
|
||||
This plan upgrades the platform's interaction with the Anthropic API: how prompts are built, how responses are parsed, how the model is retried, and how outputs are validated. It is broken into six phases that can be implemented and shipped independently. Each phase ends with verifiable acceptance criteria.
|
||||
|
||||
> **Status (2026-05-20):** Phases 1–5 implemented and shipped. Phase 6 (eval harness) is intentionally **out of scope** for this initiative — the production pipeline is hardened to the level the platform needs, and a golden-set runner can be reopened later as a stand-alone task if regression risk grows. The repo no longer carries any TODO from this plan.
|
||||
|
||||
---
|
||||
|
||||
## 0. Operating principles
|
||||
|
||||
These rules govern every phase. Re-read them before you commit.
|
||||
|
||||
1. **PocketBase is the source of truth.** No persistent state in localStorage (see [`AI_AGENT.md`](AI_AGENT.md) §2). The Anthropic API is proxied via Caddy; **never** add `x-api-key` headers in frontend code.
|
||||
2. **No behaviour regressions.** Existing UI flows (extraction, weekly learning, weekly quiz, R42 chat, handbook sync, analyze-graph) must keep working after every phase. Phases are additive.
|
||||
3. **Schema-first.** Where the model produces structured output, define a JSON Schema (Zod) and validate every response. Reject (don't paper over) malformed output.
|
||||
4. **Single LLM entry point.** After Phase 1 there is exactly one module that talks to `/api/anthropic/v1/messages`. All callers go through it.
|
||||
5. **No silent truncation.** If `stop_reason === 'max_tokens'` on a structured-output call, throw — never persist a partial parse.
|
||||
6. **Cache what is stable, vary what is dynamic.** Use prompt caching for system prompts and KB context. User messages are never cached.
|
||||
7. **Comments and docs:** follow the repo's terse style. Don't add explanatory comments unless the *why* is non-obvious.
|
||||
8. **Migrations:** when changing a PocketBase schema, add a migration in `pb_migrations/` following the existing timestamp prefix convention. Never edit shipped migrations.
|
||||
9. **Stop and ask** if you encounter a decision the plan doesn't cover (e.g. a model deprecation, a missing collection, a failing test that looks pre-existing).
|
||||
|
||||
### Files you will touch (or create) across all phases
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `src/lib/llm.js` *(new, Phase 1)* | Single Anthropic client wrapper |
|
||||
| `src/lib/llmSchemas.js` *(new, Phase 1)* | Zod schemas for every structured task |
|
||||
| `src/lib/llmRetry.js` *(new, Phase 1)* | Retry + backoff + abort policy |
|
||||
| `src/lib/random.js` *(new, Phase 4)* | Fisher–Yates shuffle + RNG helpers |
|
||||
| `src/lib/api.js` | Becomes a thin re-export from `llm.js` (back-compat) |
|
||||
| `src/lib/extractionPipeline.js` | Migrated to `llm.js` + tool use + overlap chunking |
|
||||
| `src/lib/learningService.js` | Migrated to `llm.js` + tool use + patch-refine |
|
||||
| `src/lib/testService.js` | Migrated to `llm.js` + tool use + dedup + shuffle fix |
|
||||
| `src/components/admin/KnowledgeGraph.jsx` | `analyzeGraph` → tool use + dry-run preview |
|
||||
| `src/components/chat/rag.js` | Retrieval (TF-IDF) + `lookup_topic` tool |
|
||||
| `src/components/chat/prompts.js` | Split system prompt into cacheable + dynamic |
|
||||
| `src/components/chat/useChat.js` | Wire retrieval + truncation |
|
||||
| `pb_migrations/*` | Schema additions for `llm_calls`, question `difficulty`, topic `relevance_locked` |
|
||||
| `evals/` *(new, Phase 6)* | Golden-set eval harness |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Foundation (single LLM client + robust parsing)
|
||||
|
||||
**Goal:** every LLM call goes through one module that handles retry, timeout, abort, JSON extraction, and schema validation. No behaviour change visible to the user.
|
||||
|
||||
### 1.1 Create `src/lib/llmRetry.js`
|
||||
|
||||
Implements the retry policy used by `llm.js`.
|
||||
|
||||
**Behaviour:**
|
||||
- Exponential backoff with full jitter, base 1000ms, cap 16000ms.
|
||||
- Retries only on these HTTP statuses: `408, 425, 429, 500, 502, 503, 504, 529`.
|
||||
- Honours `Retry-After` header (seconds or HTTP date). If present and ≤ 60s, use it; if > 60s, fail fast.
|
||||
- Default `maxRetries = 4`.
|
||||
- Does **not** retry on `AbortError`.
|
||||
|
||||
**Exported interface:**
|
||||
|
||||
```js
|
||||
// withRetry: (fn: (attempt:number) => Promise<T>, opts?) => Promise<T>
|
||||
// RetryableError(status, retryAfterMs)
|
||||
export async function withRetry(fn, { maxRetries = 4, signal } = {}) { ... }
|
||||
export class RetryableError extends Error { constructor(status, retryAfterMs) { ... } }
|
||||
```
|
||||
|
||||
### 1.2 Create `src/lib/llmSchemas.js`
|
||||
|
||||
One Zod schema per structured task. Install Zod (`npm i zod`).
|
||||
|
||||
Required schemas (names + shape match what callers already produce — do not change field names):
|
||||
|
||||
- `extractionResultSchema` — `{ topics: Topic[], relations: Relation[] }` matching the existing `SYSTEM_PROMPT` in [`extractionPipeline.js`](src/lib/extractionPipeline.js).
|
||||
- `handbookResultSchema` — same shape, but `relation.type` enum unified to `related_to | depends_on | part_of | executed_by` (see Phase 3 task 3.5 — for now the schema accepts both `executes` and `executed_by`, normalize `executes → executed_by` post-validation).
|
||||
- `learningArticleSchema`, `learningSlidesSchema`, `learningInfographicSchema`, `learningAllSchema` matching [`learningService.js`](src/lib/learningService.js).
|
||||
- `quizQuestionsSchema` — `{ questions: Question[] }` with `options.length === 4` and `correctIndex ∈ [0,3]`.
|
||||
- `customTopicSchema` — `{ label, type: 'concept'|'role'|'process', description }`.
|
||||
- `graphActionsSchema` — `{ merges, deletions, newRelations, relevanceUpdates }` matching [`KnowledgeGraph.jsx:329`](src/components/admin/KnowledgeGraph.jsx).
|
||||
- `proposeGraphDeltaSchema` — matches `PROPOSE_GRAPH_DELTA_TOOL.input_schema` in [`prompts.js`](src/components/chat/prompts.js).
|
||||
|
||||
**Acceptance:** every schema has at least one happy-path Vitest test in `src/lib/__tests__/llmSchemas.test.js` (add `vitest` if not present).
|
||||
|
||||
### 1.3 Create `src/lib/llm.js`
|
||||
|
||||
The single Anthropic client. All other modules must call only this one.
|
||||
|
||||
**Public interface:**
|
||||
|
||||
```js
|
||||
// Task tier — used to pick a model from settings.
|
||||
// 'fast' → admin:model:fast (default: claude-haiku-4-5-20251001)
|
||||
// 'standard' → admin:model:standard (default: claude-sonnet-4-6)
|
||||
// 'reasoning' → admin:model:reasoning (default: claude-opus-4-7)
|
||||
export async function callLLM({
|
||||
task, // string, e.g. 'extract.source' — used for logging only
|
||||
tier = 'standard',
|
||||
system, // string OR Array<{ type:'text', text:string, cache_control?:{type:'ephemeral'} }>
|
||||
messages, // [{ role, content }] OR omitted (use `user`)
|
||||
user, // shorthand for [{role:'user', content: user}]
|
||||
tools, // optional Anthropic tool definitions
|
||||
toolChoice, // optional, e.g. { type:'tool', name:'emit_knowledge_graph' }
|
||||
schema, // optional Zod schema for text→JSON path (used only when no tool)
|
||||
maxTokens = 4096,
|
||||
temperature = 0,
|
||||
signal, // AbortSignal
|
||||
}): Promise<{
|
||||
text: string,
|
||||
toolUses: Array<{ name, input }>,
|
||||
stopReason: string,
|
||||
usage: { input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens },
|
||||
requestId: string | null,
|
||||
model: string,
|
||||
durationMs: number,
|
||||
}>
|
||||
```
|
||||
|
||||
**Key requirements:**
|
||||
|
||||
1. **Simulation mode** preserved: if `storage.get('admin:use_simulation') === true`, return a deterministic stub (use existing `simulateResponse` payload for backward compatibility — branch on `task` prefix to return a matching stub).
|
||||
2. **Fetch** with `AbortController`; default **60-second timeout** if caller didn't pass a signal.
|
||||
3. **Retry** through `withRetry` (Phase 1.1).
|
||||
4. **Auth-portal detection** preserved: if response is not `application/json`, throw `Your session has expired. Please refresh the page and log in again.` exactly as today.
|
||||
5. **No truncation acceptance**: if `stop_reason === 'max_tokens'` AND caller passed `schema` OR `toolChoice` requested a tool, throw `LLMTruncatedError`.
|
||||
6. **Robust JSON extraction** when caller passed `schema` (and no tool was used): use `parseStructuredText(text)` that
|
||||
- strips ```` ```json ```` and ```` ``` ```` fences,
|
||||
- finds the outermost balanced JSON value (object **or** array) via a tiny brace-matching scan, not regex,
|
||||
- throws `LLMOutputError` if no balanced JSON found,
|
||||
- runs Zod `schema.parse` on the result.
|
||||
7. **Tool path:** when `tools` is provided and the model emits `tool_use`, return them under `toolUses`. Validate each tool's input against the corresponding Zod schema if the caller wired one in (via `toolSchemas: { [toolName]: ZodSchema }`).
|
||||
8. **Logging:** after every call, append a row to a new PocketBase collection `llm_calls` (best-effort — never block on this; catch and console.debug failures). Fields: `task, model, tier, duration_ms, input_tokens, output_tokens, cache_read_tokens, cache_create_tokens, stop_reason, ok, error_msg`. See Phase 5 task 5.6 for the migration.
|
||||
9. **Custom errors**: `LLMHttpError`, `LLMTruncatedError`, `LLMOutputError`, `LLMValidationError`. All extend `Error` and set `name` for `instanceof` checks.
|
||||
|
||||
### 1.4 Make `src/lib/api.js` a thin shim
|
||||
|
||||
Replace the existing `anthropicApi.generateContent` and `anthropicApi.chat` implementations with calls into `llm.js`. Preserve the exact exported names and return shapes so no caller breaks.
|
||||
|
||||
```js
|
||||
// api.js after Phase 1
|
||||
export const anthropicApi = {
|
||||
async generateContent(systemPrompt, userMessage, maxRetries = 1) {
|
||||
const { text } = await callLLM({
|
||||
task: 'legacy.generateContent',
|
||||
tier: 'standard',
|
||||
system: systemPrompt,
|
||||
user: userMessage,
|
||||
maxTokens: 8192,
|
||||
temperature: 0,
|
||||
});
|
||||
return text;
|
||||
},
|
||||
async chat(systemPrompt, messages, opts = {}) {
|
||||
const r = await callLLM({
|
||||
task: 'legacy.chat',
|
||||
tier: 'standard',
|
||||
system: systemPrompt,
|
||||
messages,
|
||||
tools: opts.tools,
|
||||
maxTokens: 1024,
|
||||
temperature: 0.3, // chat default — see Phase 5
|
||||
});
|
||||
return {
|
||||
content: [
|
||||
...(r.text ? [{ type:'text', text: r.text }] : []),
|
||||
...r.toolUses.map(tu => ({ type:'tool_use', name: tu.name, input: tu.input })),
|
||||
],
|
||||
stop_reason: r.stopReason,
|
||||
};
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
### 1.5 Update default model + tiered settings
|
||||
|
||||
- Replace `DEFAULT_MODEL = 'claude-sonnet-4-20250514'` with the three tier defaults above.
|
||||
- In **Admin → Settings**, add three model selects (`fast`, `standard`, `reasoning`). Read existing `admin:model` as a legacy fallback for `standard` (so existing users don't lose their override).
|
||||
|
||||
### Phase 1 acceptance criteria
|
||||
|
||||
- [ ] `npm run lint` passes; `npm run test` passes (Vitest).
|
||||
- [ ] Every existing user flow (extraction, weekly content, weekly quiz, R42, handbook sync, analyze graph) still works against the live API.
|
||||
- [ ] `grep -r "fetch.*anthropic" src/` returns only `src/lib/llm.js`.
|
||||
- [ ] Simulation mode toggle still returns stubbed responses for all flows.
|
||||
- [ ] Manually verify: kill the network mid-call → request aborts within 60s and surfaces a clear error message.
|
||||
- [ ] Manually verify: rate-limit the proxy (429 + `Retry-After: 5`) → call retries once after ~5s and then succeeds.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Prompt caching & tool-based structured outputs
|
||||
|
||||
**Goal:** structured-output tasks no longer parse JSON out of prose. Large stable prompts are cached.
|
||||
|
||||
### 2.1 Migrate extraction to tool use
|
||||
|
||||
In [`extractionPipeline.js`](src/lib/extractionPipeline.js):
|
||||
|
||||
- Replace the "Return JSON only" instruction with a tool: `emit_knowledge_graph` whose `input_schema` mirrors `extractionResultSchema`.
|
||||
- Replace `anthropicApi.generateContent(...)` with `callLLM({ ..., tools:[emitKnowledgeGraphTool], toolChoice:{ type:'tool', name:'emit_knowledge_graph' } })`.
|
||||
- Read the validated object from `toolUses[0].input`.
|
||||
- Same migration for `analyzeHandbookDelta` (tool `emit_handbook_delta`).
|
||||
- Delete every `responseText.match(/\{[\s\S]*\}/)` site.
|
||||
|
||||
### 2.2 Migrate learning, quiz, custom-topic, graph-actions to tool use
|
||||
|
||||
Same pattern, in:
|
||||
|
||||
- [`learningService.js`](src/lib/learningService.js): tools `emit_learning_article`, `emit_learning_slides`, `emit_learning_infographic`, `emit_learning_all`, `emit_custom_topic`.
|
||||
- [`testService.js`](src/lib/testService.js): tool `emit_quiz_questions`.
|
||||
- [`KnowledgeGraph.jsx:297`](src/components/admin/KnowledgeGraph.jsx): tool `emit_graph_actions`.
|
||||
|
||||
### 2.3 Prompt caching
|
||||
|
||||
Pass `system` as an array of blocks so the stable parts can be cached:
|
||||
|
||||
```js
|
||||
system: [
|
||||
{ type:'text', text: STABLE_SYSTEM_HEADER, cache_control: { type:'ephemeral' } },
|
||||
{ type:'text', text: dynamicPart }, // not cached
|
||||
],
|
||||
```
|
||||
|
||||
Apply caching to:
|
||||
- Extraction `SYSTEM_PROMPT` and `HANDBOOK_SYSTEM_PROMPT` (both fully stable → cache the whole block).
|
||||
- R42 system prompt — split into three blocks: stable preamble (cached), KB context (cached *only* while the graph hasn't changed; bust by appending a short hash of the topic IDs+labels — Phase 5 details), and per-turn role line (not cached).
|
||||
|
||||
### 2.4 Patch-based learning refinement
|
||||
|
||||
Refactor `refineLearningContent` ([`learningService.js:147`](src/lib/learningService.js:147)) from "return the full updated JSON" to **patch operations** via tools:
|
||||
|
||||
- `set_section(heading: string, body: string)` — replace one section by heading match.
|
||||
- `add_section(heading: string, body: string, position: 'start'|'end')`.
|
||||
- `remove_section(heading: string)`.
|
||||
- `replace_takeaways(items: string[])`.
|
||||
- `set_intro(intro: string)`.
|
||||
|
||||
Apply patches client-side to the cached object. Re-validate against `learningArticleSchema` after patching; reject the whole turn if invalid.
|
||||
|
||||
### Phase 2 acceptance criteria
|
||||
|
||||
- [ ] No regex JSON extraction left in `src/`: `grep -rn "match(/\\\\{\\[\\\\s\\\\S\\]\\*\\\\}/)" src/` returns nothing.
|
||||
- [ ] Token usage telemetry shows `cache_read_input_tokens > 0` on the second extraction call within 5 minutes (cache hit).
|
||||
- [ ] Re-running extraction on a known source produces the same topic count ±10% as before this phase.
|
||||
- [ ] `refineLearningContent` round-trip ("make the intro shorter") produces only the changed section in the diff against the prior cached content.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Extraction quality
|
||||
|
||||
**Goal:** fewer near-duplicate topics, no silent truncation, adaptive throttling, unified vocabulary.
|
||||
|
||||
### 3.1 Sentence-aware chunking with overlap
|
||||
|
||||
Replace `chunkText` in [`extractionPipeline.js:87`](src/lib/extractionPipeline.js:87):
|
||||
|
||||
- Target **~2000 input tokens per chunk**. Approximate as `chars / 4`. Configurable via `MAX_CHUNK_CHARS = 8000`.
|
||||
- **200-token overlap** between chunks (`OVERLAP_CHARS = 800`).
|
||||
- Split on sentence boundaries (`/(?<=[.!?])\s+/`) first; fall back to paragraph boundary if a sentence is too long; never produce a chunk larger than `MAX_CHUNK_CHARS`.
|
||||
- Add a guard: if a single sentence exceeds `MAX_CHUNK_CHARS`, hard-split at character boundary and log a warning.
|
||||
|
||||
### 3.2 Stateful extraction
|
||||
|
||||
Before each chunk after the first, prepend to the user message:
|
||||
|
||||
```text
|
||||
Already-extracted topic IDs (do NOT create new IDs for these — reuse them if the same concept appears here):
|
||||
- software-engineer
|
||||
- onboarding-buddy
|
||||
...
|
||||
```
|
||||
|
||||
Cap the list at 200 IDs by recency to keep token cost bounded. The model will then reuse IDs instead of inventing variants like `software-developer`.
|
||||
|
||||
### 3.3 Adaptive throttling
|
||||
|
||||
Replace the hard `setTimeout(r, 12000)` in [`extractionPipeline.js:127`](src/lib/extractionPipeline.js:127) and the 15s sleep in [`KnowledgeGraph.jsx:274`](src/components/admin/KnowledgeGraph.jsx:274) with a shared token-bucket limiter in `src/lib/llmRetry.js`:
|
||||
|
||||
```js
|
||||
export const extractionLimiter = createLimiter({ rps: 5/60, burst: 1 }); // 5 req/min
|
||||
// usage: await extractionLimiter.acquire();
|
||||
```
|
||||
|
||||
`callLLM` accepts an optional `limiter` param that it `await`s before fetch. On 429 with `Retry-After`, the limiter is paused for that duration.
|
||||
|
||||
### 3.4 Preserve admin-edited relevance
|
||||
|
||||
Add a migration introducing `relevance_locked: bool` on `topics`. Set it to `true` whenever an admin edits `learning_relevance` via the UI ([`KnowledgeGraph.jsx`](src/components/admin/KnowledgeGraph.jsx) edit handler — locate by searching for `setLearningRelevance` or the relevance form field).
|
||||
|
||||
In `mergeKnowledgeGraph` ([`extractionPipeline.js:167`](src/lib/extractionPipeline.js:167)), when `relevance_locked`, never overwrite `learning_relevance`.
|
||||
|
||||
### 3.5 Unify relation vocabulary
|
||||
|
||||
Pick **one** set: `related_to | depends_on | part_of | executed_by`. Migrate:
|
||||
|
||||
- `HANDBOOK_SYSTEM_PROMPT` ([`extractionPipeline.js:42`](src/lib/extractionPipeline.js:42)) — change `executes` to `executed_by` and swap the source/target in the prompt example.
|
||||
- Write a one-shot migration script `pb_migrations/<timestamp>_normalize_relation_types.js` that rewrites any existing `executes` relation to `executed_by` and swaps `source ↔ target`.
|
||||
- Verify R42's `validateDelta` ([`rag.js:108`](src/components/chat/rag.js:108)) already enforces this set (it does) — no change needed there.
|
||||
|
||||
### 3.6 Cancellation
|
||||
|
||||
Add a "Cancel" button to the source-processing UI in `ContentManager.jsx` / `UploadZone.jsx` (locate the one that displays extraction progress). Wire it to abort the in-flight `callLLM` via the `signal` it receives. On cancel, set source status to `cancelled` (add to status enum migration).
|
||||
|
||||
### Phase 3 acceptance criteria
|
||||
|
||||
- [ ] Running extraction twice on the same `sources/ROLES.md` produces zero new topics on the second run (idempotency through reused IDs).
|
||||
- [ ] Locked-relevance topics survive re-extraction.
|
||||
- [ ] No fixed `setTimeout` ≥ 5s anywhere in `src/` (`grep -rn "setTimeout" src/`).
|
||||
- [ ] Cancelling an extraction mid-run leaves the source in `cancelled` state, not `processing`.
|
||||
- [ ] `pb_migrations` includes the relation-vocabulary normalization and the `relevance_locked` column.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Quiz & content quality
|
||||
|
||||
**Goal:** quiz questions are positionally unbiased, deduped, and difficulty-tagged. Random helpers are correct.
|
||||
|
||||
### 4.1 Random helpers
|
||||
|
||||
Create `src/lib/random.js`:
|
||||
|
||||
```js
|
||||
export function shuffle(arr) { /* Fisher–Yates, returns NEW array */ }
|
||||
export function sample(arr, n) { /* unbiased sample without replacement */ }
|
||||
export function pickInt(min, maxInclusive) { /* uniform integer */ }
|
||||
```
|
||||
|
||||
Replace every `.sort(() => 0.5 - Math.random())` with `shuffle(arr)`:
|
||||
|
||||
- [`testService.js:122`](src/lib/testService.js:122) and [`testService.js:163`](src/lib/testService.js:163).
|
||||
- Any other site found by `grep -rn "0.5 - Math.random()" src/`.
|
||||
|
||||
### 4.2 Debias `correctIndex` in quiz prompt
|
||||
|
||||
In [`testService.js:81`](src/lib/testService.js:81):
|
||||
|
||||
- Change the example in the prompt to use `"correctIndex": 2` (not 0).
|
||||
- Add to the prompt: *"Distribute correctIndex roughly evenly across 0, 1, 2, and 3. Do not place the correct answer at the same position more than 4 out of 10 times."*
|
||||
- After parsing, run a check: if more than 50% of the batch share the same `correctIndex`, log a warning and re-roll up to 2 times.
|
||||
|
||||
### 4.3 Difficulty field
|
||||
|
||||
- Add to `quizQuestionsSchema`: `difficulty: 'easy'|'medium'|'hard'`.
|
||||
- Update the prompt to require difficulty on every question (current prompt says "4 easy, 4 medium, 2 hard" but never tagged — now tag it).
|
||||
- Migration: add `difficulty` to the `quiz_banks.questions[]` element. PocketBase stores `questions` as JSON, so the migration is a no-op at the column level; older records get `difficulty: 'medium'` on read (add a normalizer in `db.js`).
|
||||
|
||||
### 4.4 Question dedup
|
||||
|
||||
In `forceGenerateTopicQuestions` ([`testService.js:65`](src/lib/testService.js:65)):
|
||||
|
||||
- Normalize question text (lowercase, strip punctuation, collapse whitespace) → `normKey`.
|
||||
- Before persisting, drop any new question whose `normKey` matches an existing bank question.
|
||||
- Log dropped duplicates with `console.debug('[quiz] dropped duplicate:', text)`.
|
||||
|
||||
### 4.5 Quality gate
|
||||
|
||||
In the same function, after schema validation:
|
||||
|
||||
- Reject the whole batch if any question has fewer than 4 distinct options.
|
||||
- Reject if any option contains `"all of the above"`, `"none of the above"`, `"both A and B"` (case-insensitive).
|
||||
- Reject if `explanation.trim().length < 20`.
|
||||
- Surface the rejection to the admin UI with a "Retry" button.
|
||||
|
||||
### 4.6 Custom topic ID hygiene
|
||||
|
||||
In `generateCustomTopic` ([`learningService.js:177`](src/lib/learningService.js:177)):
|
||||
|
||||
- Generate kebab-case ID from the polished label, not `Date.now()`.
|
||||
- Collision check against existing topics (append `-2`, `-3`, … if needed).
|
||||
- Default `learning_relevance: 'standard'` when the model omits it.
|
||||
|
||||
### Phase 4 acceptance criteria
|
||||
|
||||
- [ ] No `.sort(() => 0.5 - Math.random())` anywhere in `src/`.
|
||||
- [ ] Sample of 50 fresh quiz questions across 5 topics: no position holds >40% of correct answers.
|
||||
- [ ] Re-running quiz generation for the same topic does not grow the bank with semantic duplicates.
|
||||
- [ ] Custom topics created via R42 use kebab-case IDs and pass schema validation.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — R42 retrieval & telemetry
|
||||
|
||||
**Goal:** R42 stops shipping the entire KG every turn; conversations are bounded; every call is logged.
|
||||
|
||||
### 5.1 TF-IDF retrieval in the browser
|
||||
|
||||
Create `src/lib/retrieval.js`:
|
||||
|
||||
```js
|
||||
export function buildIndex(topics) { /* TF-IDF over label + description */ }
|
||||
export function retrieveTopK(index, query, k = 10) { /* returns Topic[] */ }
|
||||
```
|
||||
|
||||
Implementation: a small dependency-free TF-IDF — tokenize on `/[a-zA-Z0-9-]+/`, lowercase, drop stopwords (Dutch + English short list). Cache the index on the `topics` array reference. About 100 lines.
|
||||
|
||||
### 5.2 Rewrite `buildKbContext`
|
||||
|
||||
In [`rag.js:11`](src/components/chat/rag.js:11):
|
||||
|
||||
- Use `retrieveTopK(index, userMessage, 10)` to pick which topics go into the system prompt.
|
||||
- Always include any topic whose ID or label is mentioned verbatim (existing behaviour).
|
||||
- Drop the full "every topic" dump.
|
||||
- Return `{ context, retrievedTopics, allTopics }` — `validateDelta` continues to use `allTopics`.
|
||||
|
||||
### 5.3 `lookup_topic` tool
|
||||
|
||||
Add a second R42 tool in [`prompts.js`](src/components/chat/prompts.js):
|
||||
|
||||
```js
|
||||
export const LOOKUP_TOPIC_TOOL = {
|
||||
name: 'lookup_topic',
|
||||
description: 'Fetch the full description and any deeper learning content for a topic. Use when the retrieved context does not contain enough to answer.',
|
||||
input_schema: { type:'object', properties: { id:{type:'string'} }, required:['id'] },
|
||||
};
|
||||
```
|
||||
|
||||
In `useChat.js`, when the model emits a `lookup_topic`, fetch via `db.getTopics()` + `db.getContent(id)` and append a `tool_result` block, then call `callLLM` again with the extended messages. Cap to **3 lookup hops per turn** to avoid loops.
|
||||
|
||||
### 5.4 R42 prompt cache busting
|
||||
|
||||
KB-context block is cached as ephemeral. The cache key is automatic per text, so any change busts it. **Append a stable suffix** to the KB block: `"\n[kb_hash: <first-8-chars-of-sha256-of-sorted-topic-ids>]"`. This guarantees the block content changes the moment a topic is added/removed, even if the included topic list looks similar.
|
||||
|
||||
### 5.5 Conversation truncation
|
||||
|
||||
In `useChat.js`:
|
||||
|
||||
- Keep last **12 turns** verbatim in `apiMessages`.
|
||||
- Older messages: if more than 12 turns exist, summarize the older ones with a cheap (`tier: 'fast'`) call and prepend a single `{ role:'system'-equivalent inside the user history is not allowed by Anthropic — instead prepend a user/assistant pair }` block, OR simply drop older turns. **Default: drop older turns and prepend a one-line `assistant` message saying "(earlier conversation truncated)".** Summarization is an optional follow-up.
|
||||
|
||||
### 5.6 `llm_calls` collection
|
||||
|
||||
Add a migration `pb_migrations/<timestamp>_created_llm_calls.js` for collection `llm_calls`:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `task` | text | e.g. `extract.source` |
|
||||
| `model` | text | resolved model ID |
|
||||
| `tier` | text | `fast` / `standard` / `reasoning` |
|
||||
| `duration_ms` | number | |
|
||||
| `input_tokens` | number | |
|
||||
| `output_tokens` | number | |
|
||||
| `cache_read_tokens` | number | |
|
||||
| `cache_create_tokens` | number | |
|
||||
| `stop_reason` | text | |
|
||||
| `ok` | bool | |
|
||||
| `error_msg` | text | nullable |
|
||||
|
||||
Wire `callLLM` to write to it (best-effort, never throws). Add a minimal `Admin → Diagnostics` view that shows the last 100 calls and aggregate cost (using public Anthropic prices in a constant; refresh manually).
|
||||
|
||||
### 5.7 R42 defaults
|
||||
|
||||
- `temperature: 0.3` for R42 chat in `callLLM`.
|
||||
- `maxTokens: 2048` for R42 (text + tool budget).
|
||||
|
||||
### Phase 5 acceptance criteria
|
||||
|
||||
- [ ] R42 system prompt is ≤ 4000 tokens regardless of KG size (verify on a graph with 200+ topics).
|
||||
- [ ] Adding a topic in the admin UI causes the **next** R42 call to show `cache_read_tokens === 0` for the KB block, then subsequent calls to show non-zero.
|
||||
- [ ] R42 successfully answers a question about a topic whose ID was not in the user's message by emitting `lookup_topic` (manual verification).
|
||||
- [ ] `Admin → Diagnostics` shows recent LLM calls with model, tokens, duration.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Eval harness (NOT IMPLEMENTED — out of scope)
|
||||
|
||||
> Phase 6 was deliberately skipped. The acceptance criteria for Phases 1–5 give enough confidence in extraction, content, quiz, R42, and telemetry that a golden-set harness is not currently load-bearing. Leave this section intact as a starting point for a future, separately-scoped initiative.
|
||||
|
||||
**Goal:** prompt or model changes can be measured before they ship.
|
||||
|
||||
### 6.1 Golden sets
|
||||
|
||||
Create `evals/` at the repo root:
|
||||
|
||||
```
|
||||
evals/
|
||||
extraction/
|
||||
cases/
|
||||
roles-handbook.txt
|
||||
governance-handbook.txt
|
||||
expected/
|
||||
roles-handbook.json # { mustContain: ['software-engineer', ...], minTopics: 20 }
|
||||
governance-handbook.json
|
||||
quiz/
|
||||
cases/
|
||||
software-engineer.json # topic to generate quiz for
|
||||
rubric.md # human-readable quality rubric
|
||||
chat/
|
||||
scripts/
|
||||
ask-about-known-topic.json
|
||||
ask-about-unknown-topic.json
|
||||
propose-new-role.json
|
||||
```
|
||||
|
||||
10 extraction cases, 5 quiz topics, 10 chat scripts is enough to start.
|
||||
|
||||
### 6.2 Runner
|
||||
|
||||
`evals/run.mjs` — Node script that:
|
||||
|
||||
1. Loads each case.
|
||||
2. Invokes the same code path the app uses (import from `src/lib/*` directly; reuse the simulation toggle off).
|
||||
3. Compares against expectations:
|
||||
- **Extraction:** `mustContain` IDs present? `minTopics` met? No `stop_reason: 'max_tokens'`?
|
||||
- **Quiz:** distribution of `correctIndex`, mean explanation length, banned phrases.
|
||||
- **Chat:** for each script, did the expected tool fire? Did the reply contain expected anchors?
|
||||
4. Writes `evals/results/<ISO-timestamp>.json` and a Markdown diff against the previous baseline.
|
||||
|
||||
Add `npm run eval` to `package.json`.
|
||||
|
||||
### 6.3 Prompt versioning
|
||||
|
||||
In each prompt module, export `PROMPT_VERSION = '2026-05-20-001'`. Persist it on the artifact (`content.data.prompt_version`, `quiz_banks.questions[i].prompt_version`, `topics.metadata.prompt_version`). Add an admin button "Mark stale content for regeneration" that lists artifacts whose version is older than the current.
|
||||
|
||||
### Phase 6 acceptance criteria
|
||||
|
||||
- [ ] `npm run eval` runs end-to-end against the live API and produces a result file.
|
||||
- [ ] CI (or a manual check) runs evals on every change to `src/lib/llm.js`, prompts, or schemas.
|
||||
- [ ] Each AI-generated artifact in PocketBase carries a `prompt_version` and is filterable by it.
|
||||
|
||||
---
|
||||
|
||||
## Cross-phase verification checklist
|
||||
|
||||
After every phase, run this short checklist before merging:
|
||||
|
||||
1. **Build:** `npm run build` succeeds.
|
||||
2. **Lint:** `npm run lint` clean.
|
||||
3. **Tests:** `npm run test` green.
|
||||
4. **Smoke flows** in dev (simulation off, real API key):
|
||||
- Add a source via `Admin → Sources`, extract, verify topics appear.
|
||||
- Visit `Leren` for the current week, generate article. Then slides.
|
||||
- Visit `Testen`, generate weekly quiz. Submit. Score lands on leaderboard.
|
||||
- Open R42, ask a known and an unknown question; propose a new topic; admin accepts it.
|
||||
- Run `Admin → Knowledge Graph → Analyze & Optimize Graph`.
|
||||
- Run `Admin → Knowledge Graph → Sync Handbook` (small repo or mock).
|
||||
5. **Simulation toggle:** flip simulation mode on and confirm no real API calls happen (Network tab).
|
||||
|
||||
---
|
||||
|
||||
## Rollback strategy
|
||||
|
||||
Every phase is shippable on its own. If a phase introduces a regression:
|
||||
|
||||
- **Phase 1–2:** `git revert` the merge commit. `api.js` retains the legacy interface, so callers that haven't migrated still work.
|
||||
- **Phase 3:** revert + the relation-vocabulary migration is reversible (rerun the reverse swap).
|
||||
- **Phase 4:** revert; quiz schema additions are forward-compatible (older readers ignore `difficulty`).
|
||||
- **Phase 5:** revert + drop the `llm_calls` collection if undesired.
|
||||
- **Phase 6:** purely additive; remove the `evals/` folder if abandoned.
|
||||
|
||||
Never `git push --force` to `main`. PR-per-phase.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope (do not do as part of this plan)
|
||||
|
||||
- Replacing PocketBase. Stays as-is.
|
||||
- Server-side embeddings or a vector store. Phase 5 deliberately uses in-browser TF-IDF.
|
||||
- Streaming responses. Mentioned as a future improvement; not in this plan.
|
||||
- Multi-tenant changes. The platform serves one company.
|
||||
- UI redesign of the Admin pages beyond what each phase requires.
|
||||
|
||||
---
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Tier** — coarse model class (`fast`/`standard`/`reasoning`) mapped to a concrete Anthropic model ID via admin settings.
|
||||
- **Tool use** — Anthropic's structured-output mechanism. The model emits a `tool_use` content block whose `input` is schema-valid JSON.
|
||||
- **Prompt caching** — Anthropic feature where blocks marked `cache_control: { type:'ephemeral' }` are reused across requests at lower input cost. 5-minute TTL.
|
||||
- **TF-IDF** — Term-frequency / inverse-document-frequency. A classic IR scoring function used here as a cheap retrieval signal.
|
||||
- **KB context** — The block in R42's system prompt that lists topics and relations from the knowledge graph.
|
||||
- **Delta** — A proposed addition to the knowledge graph emitted by R42 via the `propose_graph_delta` tool.
|
||||
153
CLAUDE.md
Normal file
153
CLAUDE.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# CLAUDE.md
|
||||
|
||||
## What you are building
|
||||
|
||||
A mobile-first progressive web application for employee learning. Employees follow
|
||||
a perpetual 26-week curriculum built from an internal knowledge base. An AI
|
||||
assistant called R42 is available on every screen. Admins upload source documents
|
||||
that are processed into the knowledge base by AI.
|
||||
|
||||
Read the full design before writing any code:
|
||||
- /docs/handover.md — all decisions made, rationale, constraints
|
||||
- /docs/architecture.md — system design, data flows, tech stack
|
||||
- /docs/data-model.md — all PocketBase collections, Qdrant schema, types
|
||||
- /docs/ingestion-spec.md — ingestion service (build this first)
|
||||
- /docs/implementation-plan.md — ordered build sequence with acceptance criteria
|
||||
|
||||
---
|
||||
|
||||
## Absolute constraints
|
||||
|
||||
These rules are non-negotiable and apply to every session:
|
||||
|
||||
1. Never modify any file listed in PROTECTED.md
|
||||
2. Never modify any file outside /app — the pipeline, Dockerfile, ansible,
|
||||
and docker-compose files are frozen
|
||||
3. Never delete files without explicit confirmation
|
||||
4. Never change package.json scripts that contain 'deploy' or 'build:prod'
|
||||
5. Ask before acting when scope is unclear — do not infer intent from legacy/ code
|
||||
|
||||
---
|
||||
|
||||
## Repository structure
|
||||
|
||||
```
|
||||
repo/
|
||||
├── CLAUDE.md ← you are here
|
||||
├── PROTECTED.md ← files you must never touch
|
||||
├── docs/ ← spec files — read, never modify
|
||||
├── legacy/ ← old prototype — read-only reference only
|
||||
└── app/ ← your working directory
|
||||
├── frontend/ ← Next.js 14 PWA
|
||||
└── services/
|
||||
├── ingestion/ ← build first
|
||||
├── generation/ ← build second
|
||||
├── curriculum/ ← build third
|
||||
├── embedding/ ← integrated into ingestion, separate later
|
||||
├── chat/ ← R42
|
||||
└── progress/ ← gamification
|
||||
```
|
||||
|
||||
All work happens inside /app. No exceptions.
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Layer | Technology |
|
||||
|---|---|
|
||||
| Frontend | Next.js 14, TypeScript strict, Tailwind CSS, PWA |
|
||||
| Backend state | PocketBase (binary, not source — do not scaffold from scratch) |
|
||||
| Vector store | Qdrant via REST client |
|
||||
| AI generation | Claude Sonnet 4 — model string: claude-sonnet-4-20250514 |
|
||||
| AI chat (R42) | Claude Haiku 4.5 — model string: claude-haiku-4-5-20251001 |
|
||||
| Embeddings | OpenAI text-embedding-3-small |
|
||||
| Services | Node.js, Fastify, TypeScript strict |
|
||||
| Validation | Zod on all external data (API responses, AI output, PocketBase) |
|
||||
|
||||
---
|
||||
|
||||
## Stylesheet
|
||||
|
||||
An existing stylesheet lives at /stylesheet.css in the repo root. This is the
|
||||
authoritative visual style for the application.
|
||||
|
||||
Rules:
|
||||
- Never modify stylesheet.css
|
||||
- Import it as a global stylesheet in the Next.js frontend
|
||||
- Do not override its rules with Tailwind utility classes or inline styles
|
||||
- When Tailwind and the stylesheet conflict, the stylesheet wins
|
||||
- If a UI element is not covered by the stylesheet, use Tailwind — but match
|
||||
the visual language (spacing, colour, type scale) of the existing stylesheet
|
||||
|
||||
---
|
||||
|
||||
## Code conventions
|
||||
|
||||
- TypeScript strict mode everywhere — no `any` types
|
||||
- All Claude API responses validated through Zod before use
|
||||
- All PocketBase writes typed against collection schemas in data-model.md
|
||||
- Qdrant payloads explicitly typed — no untyped objects
|
||||
- No inline hardcoded API keys — environment variables only
|
||||
- REST conventions for all service APIs
|
||||
- Mobile-first CSS — design for 375px width, scale up
|
||||
|
||||
---
|
||||
|
||||
## Build order
|
||||
|
||||
Follow /docs/implementation-plan.md exactly.
|
||||
Do not skip phases. Do not build phase N+1 before phase N passes its
|
||||
acceptance criteria.
|
||||
|
||||
Current phase: **Phase 1 — Infrastructure + ingestion service**
|
||||
|
||||
---
|
||||
|
||||
## Session discipline
|
||||
|
||||
Each session has a defined scope in implementation-plan.md.
|
||||
At the start of each session:
|
||||
1. State which phase and step you are working on
|
||||
2. Read the relevant spec file(s)
|
||||
3. Propose a file structure or change plan before writing code
|
||||
4. Implement after the plan is clear
|
||||
|
||||
At the end of each session:
|
||||
1. State what was completed
|
||||
2. State what acceptance criteria have been met
|
||||
3. State what the next session should start with
|
||||
|
||||
---
|
||||
|
||||
## When you are uncertain
|
||||
|
||||
Check the spec files in /docs first.
|
||||
If the spec does not cover it, flag the gap explicitly rather than making
|
||||
an assumption. Do not look at legacy/ code for implementation guidance —
|
||||
it is a different stack and will lead you in the wrong direction.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
Each service has its own .env file. Templates are defined in each service spec.
|
||||
Never commit actual values. Always use .env.example with placeholder values.
|
||||
|
||||
The full set across all services:
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
QDRANT_URL=
|
||||
QDRANT_API_KEY=
|
||||
INGESTION_PORT=3001
|
||||
GENERATION_PORT=3002
|
||||
CURRICULUM_PORT=3003
|
||||
CHAT_PORT=3004
|
||||
PROGRESS_PORT=3005
|
||||
NEXT_PUBLIC_API_URL=
|
||||
NEXT_PUBLIC_POCKETBASE_URL=
|
||||
```
|
||||
34
PROTECTED.md
Normal file
34
PROTECTED.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Protected files — do not modify
|
||||
|
||||
These files and directories must never be modified by Claude Code.
|
||||
They belong to the existing deployment pipeline and are frozen.
|
||||
|
||||
## Pipeline and CI/CD
|
||||
.github/
|
||||
.github/workflows/
|
||||
.gitlab-ci.yml (if present)
|
||||
|
||||
## Container and orchestration
|
||||
Dockerfile
|
||||
docker-compose.yml
|
||||
docker-compose.prod.yml
|
||||
docker-compose.override.yml (if present)
|
||||
|
||||
## Infrastructure and provisioning
|
||||
ansible/
|
||||
nginx/
|
||||
|
||||
## Stylesheet
|
||||
stylesheet.css (repo root — use as-is, do not modify)
|
||||
|
||||
## Environment templates at repo root
|
||||
.env.example (repo root only — services may have their own)
|
||||
|
||||
## Legacy prototype
|
||||
legacy/
|
||||
|
||||
---
|
||||
|
||||
The only infrastructure file that will change is Dockerfile, and only in
|
||||
Phase 8 step 8.5. That change is made manually by the human, not by
|
||||
Claude Code.
|
||||
0
app/frontend/.gitkeep
Normal file
0
app/frontend/.gitkeep
Normal file
0
app/services/chat/.gitkeep
Normal file
0
app/services/chat/.gitkeep
Normal file
0
app/services/curriculum/.gitkeep
Normal file
0
app/services/curriculum/.gitkeep
Normal file
0
app/services/generation/.gitkeep
Normal file
0
app/services/generation/.gitkeep
Normal file
4
app/services/ingestion/.gitignore
vendored
Normal file
4
app/services/ingestion/.gitignore
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
node_modules/
|
||||
dist/
|
||||
.env
|
||||
*.log
|
||||
30
app/services/ingestion/package.json
Normal file
30
app/services/ingestion/package.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "ingestion",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "tsx watch src/index.ts",
|
||||
"build": "tsc",
|
||||
"start": "node dist/index.js",
|
||||
"migrate": "tsx src/migrations/001_initial_schema.ts",
|
||||
"migrate:qdrant": "tsx src/migrations/002_qdrant_setup.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"openai": "^4",
|
||||
"@qdrant/js-client-rest": "^1.9",
|
||||
"pocketbase": "^0.21",
|
||||
"pdf-parse": "^1.1",
|
||||
"uuid": "^9",
|
||||
"zod": "^3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5",
|
||||
"tsx": "^4",
|
||||
"dotenv": "^16",
|
||||
"@types/node": "^20",
|
||||
"@types/pdf-parse": "^1.1",
|
||||
"@types/uuid": "^9"
|
||||
}
|
||||
}
|
||||
18
app/services/ingestion/src/index.ts
Normal file
18
app/services/ingestion/src/index.ts
Normal file
@@ -0,0 +1,18 @@
|
||||
import 'dotenv/config';
|
||||
import Fastify from 'fastify';
|
||||
import documentRoutes from './routes/documents.js';
|
||||
|
||||
const PORT = parseInt(process.env['INGESTION_PORT'] ?? '3001', 10);
|
||||
|
||||
async function start(): Promise<void> {
|
||||
const app = Fastify({ logger: true });
|
||||
|
||||
await app.register(documentRoutes);
|
||||
|
||||
await app.listen({ port: PORT, host: '0.0.0.0' });
|
||||
}
|
||||
|
||||
start().catch((err: unknown) => {
|
||||
console.error('Failed to start ingestion service:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
102
app/services/ingestion/src/jobs/queue.ts
Normal file
102
app/services/ingestion/src/jobs/queue.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
import { v4 as uuid } from 'uuid';
|
||||
import { extract } from '../pipeline/extract.js';
|
||||
import { chunk } from '../pipeline/chunk.js';
|
||||
import { clean } from '../pipeline/clean.js';
|
||||
import { extractStructure } from '../pipeline/structure.js';
|
||||
import { writeToPocketBase, markDocumentFailed } from '../pipeline/write.js';
|
||||
import { embedAndStore } from '../pipeline/embed.js';
|
||||
import type { Job, JobProgress, IngestBody } from '../types.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// In-memory store
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const jobs = new Map<string, Job>();
|
||||
|
||||
const DEFAULT_PROGRESS: JobProgress = {
|
||||
chunksTotal: 0,
|
||||
chunksEmbedded: 0,
|
||||
themesFound: 0,
|
||||
topicsFound: 0,
|
||||
};
|
||||
|
||||
export function createJob(params: IngestBody): Job {
|
||||
const job: Job = {
|
||||
id: uuid(),
|
||||
documentId: params.documentId,
|
||||
filename: params.filename,
|
||||
format: params.format,
|
||||
filePath: params.filePath,
|
||||
status: 'queued',
|
||||
progress: { ...DEFAULT_PROGRESS },
|
||||
error: null,
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
};
|
||||
jobs.set(job.id, job);
|
||||
void runPipeline(job.id);
|
||||
return job;
|
||||
}
|
||||
|
||||
export function getJob(id: string): Job | undefined {
|
||||
return jobs.get(id);
|
||||
}
|
||||
|
||||
function updateJob(id: string, updates: Partial<Omit<Job, 'id' | 'createdAt'>>): void {
|
||||
const job = jobs.get(id);
|
||||
if (!job) return;
|
||||
jobs.set(id, { ...job, ...updates, updatedAt: new Date() });
|
||||
}
|
||||
|
||||
function mergeProgress(id: string, partial: Partial<JobProgress>): void {
|
||||
const job = jobs.get(id);
|
||||
if (!job) return;
|
||||
updateJob(id, { progress: { ...job.progress, ...partial } });
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pipeline orchestration
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function runPipeline(jobId: string): Promise<void> {
|
||||
const job = jobs.get(jobId);
|
||||
if (!job) return;
|
||||
|
||||
try {
|
||||
// Stage 1: text extraction
|
||||
updateJob(jobId, { status: 'extracting' });
|
||||
const text = await extract(job.filePath, job.format);
|
||||
|
||||
// Stages 2–3: chunking + cleaning
|
||||
updateJob(jobId, { status: 'chunking' });
|
||||
const rawChunks = chunk(text, job.format, job.documentId);
|
||||
const cleanChunks = clean(rawChunks);
|
||||
mergeProgress(jobId, { chunksTotal: cleanChunks.length });
|
||||
|
||||
// Stage 4: structure extraction (AI)
|
||||
updateJob(jobId, { status: 'structuring' });
|
||||
const draftKB = await extractStructure(cleanChunks, job.filename, job.format);
|
||||
const topicsFound = draftKB.themes.reduce((n, t) => n + t.topics.length, 0);
|
||||
mergeProgress(jobId, { themesFound: draftKB.themes.length, topicsFound });
|
||||
|
||||
// Stage 5: PocketBase write
|
||||
updateJob(jobId, { status: 'writing' });
|
||||
const writtenTopics = await writeToPocketBase(draftKB, job.documentId, cleanChunks.length);
|
||||
|
||||
// Stage 6: embeddings + Qdrant write
|
||||
updateJob(jobId, { status: 'embedding' });
|
||||
await embedAndStore(cleanChunks, writtenTopics, embedded => {
|
||||
mergeProgress(jobId, { chunksEmbedded: embedded });
|
||||
});
|
||||
|
||||
updateJob(jobId, { status: 'done' });
|
||||
} catch (err: unknown) {
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
updateJob(jobId, { status: 'failed', error: reason });
|
||||
|
||||
const j = jobs.get(jobId);
|
||||
if (j) {
|
||||
await markDocumentFailed(j.documentId, reason).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
9
app/services/ingestion/src/lib/anthropic.ts
Normal file
9
app/services/ingestion/src/lib/anthropic.ts
Normal file
@@ -0,0 +1,9 @@
|
||||
import Anthropic from '@anthropic-ai/sdk';
|
||||
|
||||
export const anthropic = new Anthropic({
|
||||
apiKey: process.env['ANTHROPIC_API_KEY'],
|
||||
});
|
||||
|
||||
export const MODELS = {
|
||||
SONNET: 'claude-sonnet-4-20250514',
|
||||
} as const;
|
||||
9
app/services/ingestion/src/lib/openai.ts
Normal file
9
app/services/ingestion/src/lib/openai.ts
Normal file
@@ -0,0 +1,9 @@
|
||||
import OpenAI from 'openai';
|
||||
|
||||
export const openai = new OpenAI({
|
||||
apiKey: process.env['OPENAI_API_KEY'],
|
||||
});
|
||||
|
||||
export const EMBEDDING_MODEL = 'text-embedding-3-small';
|
||||
export const EMBEDDING_DIMENSIONS = 1536;
|
||||
export const EMBEDDING_BATCH_SIZE = 100;
|
||||
14
app/services/ingestion/src/lib/pocketbase.ts
Normal file
14
app/services/ingestion/src/lib/pocketbase.ts
Normal file
@@ -0,0 +1,14 @@
|
||||
import PocketBase from 'pocketbase';
|
||||
|
||||
const POCKETBASE_URL = process.env['POCKETBASE_URL'] ?? '';
|
||||
const POCKETBASE_ADMIN_EMAIL = process.env['POCKETBASE_ADMIN_EMAIL'] ?? '';
|
||||
const POCKETBASE_ADMIN_PASSWORD = process.env['POCKETBASE_ADMIN_PASSWORD'] ?? '';
|
||||
|
||||
const pb = new PocketBase(POCKETBASE_URL);
|
||||
|
||||
export async function getPocketBase(): Promise<PocketBase> {
|
||||
if (!pb.authStore.isValid) {
|
||||
await pb.admins.authWithPassword(POCKETBASE_ADMIN_EMAIL, POCKETBASE_ADMIN_PASSWORD);
|
||||
}
|
||||
return pb;
|
||||
}
|
||||
14
app/services/ingestion/src/lib/qdrant.ts
Normal file
14
app/services/ingestion/src/lib/qdrant.ts
Normal file
@@ -0,0 +1,14 @@
|
||||
import { QdrantClient } from '@qdrant/js-client-rest';
|
||||
|
||||
const QDRANT_URL = process.env['QDRANT_URL'] ?? '';
|
||||
const QDRANT_API_KEY = process.env['QDRANT_API_KEY'] ?? '';
|
||||
|
||||
export const qdrant = new QdrantClient({
|
||||
url: QDRANT_URL,
|
||||
...(QDRANT_API_KEY ? { apiKey: QDRANT_API_KEY } : {}),
|
||||
});
|
||||
|
||||
export const QDRANT_COLLECTIONS = {
|
||||
SOURCE_CHUNKS: 'source_chunks',
|
||||
TOPIC_SUMMARIES: 'topic_summaries',
|
||||
} as const;
|
||||
402
app/services/ingestion/src/migrations/001_initial_schema.ts
Normal file
402
app/services/ingestion/src/migrations/001_initial_schema.ts
Normal file
@@ -0,0 +1,402 @@
|
||||
import 'dotenv/config';
|
||||
import PocketBase, { type CollectionModel } from 'pocketbase';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Env validation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const POCKETBASE_URL = process.env['POCKETBASE_URL'] ?? '';
|
||||
const POCKETBASE_ADMIN_EMAIL = process.env['POCKETBASE_ADMIN_EMAIL'] ?? '';
|
||||
const POCKETBASE_ADMIN_PASSWORD = process.env['POCKETBASE_ADMIN_PASSWORD'] ?? '';
|
||||
|
||||
if (!POCKETBASE_URL || !POCKETBASE_ADMIN_EMAIL || !POCKETBASE_ADMIN_PASSWORD) {
|
||||
console.error('Missing env vars: POCKETBASE_URL, POCKETBASE_ADMIN_EMAIL, POCKETBASE_ADMIN_PASSWORD');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Field types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface FieldDef {
|
||||
name: string;
|
||||
type: string;
|
||||
required: boolean;
|
||||
options: Record<string, unknown>;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Field helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const field = {
|
||||
text: (name: string, required = false): FieldDef => ({
|
||||
name, type: 'text', required,
|
||||
options: { min: null, max: null, pattern: '' },
|
||||
}),
|
||||
|
||||
number: (name: string, required = false): FieldDef => ({
|
||||
name, type: 'number', required,
|
||||
options: { min: null, max: null, noDecimal: false },
|
||||
}),
|
||||
|
||||
select: (name: string, values: string[], required = false, maxSelect = 1): FieldDef => ({
|
||||
name, type: 'select', required,
|
||||
options: { maxSelect, values },
|
||||
}),
|
||||
|
||||
json: (name: string): FieldDef => ({
|
||||
name, type: 'json', required: false, options: {},
|
||||
}),
|
||||
|
||||
date: (name: string): FieldDef => ({
|
||||
name, type: 'date', required: false, options: { min: '', max: '' },
|
||||
}),
|
||||
|
||||
editor: (name: string): FieldDef => ({
|
||||
name, type: 'editor', required: false, options: {},
|
||||
}),
|
||||
|
||||
file: (name: string, mimeTypes: string[] = [], maxSelect = 1): FieldDef => ({
|
||||
name, type: 'file', required: false,
|
||||
options: { maxSelect, maxSize: 52428800, mimeTypes, thumbs: [], protected: false },
|
||||
}),
|
||||
|
||||
relation: (name: string, collectionId: string, maxSelect: number | null = 1, required = false): FieldDef => ({
|
||||
name, type: 'relation', required,
|
||||
options: { collectionId, cascadeDelete: false, minSelect: null, maxSelect, displayFields: null },
|
||||
}),
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Badge seed data (from data-model.md + handover.md)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const BADGES = [
|
||||
// Bronze — beginner milestones
|
||||
{ key: 'first_commit', tier: 'bronze', label: 'First Commit', description: 'Complete your first topic', icon: '🔰' },
|
||||
{ key: 'week_shipped', tier: 'bronze', label: 'Week Shipped', description: 'Complete your first full week', icon: '📦' },
|
||||
{ key: 'streak_3', tier: 'bronze', label: '3-Week Streak', description: 'Maintain a 3-week learning streak', icon: '🔥' },
|
||||
{ key: 'quiz_taker', tier: 'bronze', label: 'Quiz Taker', description: 'Complete your first scenario quiz', icon: '❓' },
|
||||
{ key: 'flashcard_fan', tier: 'bronze', label: 'Flashcard Fan', description: 'Complete your first flashcard set', icon: '🃏' },
|
||||
{ key: 'reached_junior', tier: 'bronze', label: 'Junior Dev', description: 'Reach Junior level', icon: '👶' },
|
||||
// Silver — intermediate
|
||||
{ key: 'streak_13', tier: 'silver', label: '13-Week Streak', description: 'Maintain a 13-week learning streak', icon: '💥' },
|
||||
{ key: 'half_cycle', tier: 'silver', label: 'Half Cycle', description: 'Complete week 13 — halfway through the cycle', icon: '🔄' },
|
||||
{ key: 'case_student', tier: 'silver', label: 'Case Student', description: 'Complete 5 case studies', icon: '📋' },
|
||||
{ key: 'reached_medior', tier: 'silver', label: 'Medior Dev', description: 'Reach Medior level', icon: '💼' },
|
||||
{ key: 'reached_senior', tier: 'silver', label: 'Senior Dev', description: 'Reach Senior level', icon: '🏅' },
|
||||
// Gold — advanced
|
||||
{ key: 'full_cycle', tier: 'gold', label: 'Full Cycle', description: 'Complete a full 26-week cycle', icon: '🏆' },
|
||||
{ key: 'type_explorer', tier: 'gold', label: 'Type Explorer', description: 'Use all 10 micro learning types at least once', icon: '🗺️' },
|
||||
{ key: 'reached_staff', tier: 'gold', label: 'Staff Eng', description: 'Reach Staff level', icon: '⭐' },
|
||||
// Legendary — exceptional
|
||||
{ key: 'streak_26', tier: 'legendary', label: 'Unbroken', description: 'Complete all 26 weeks without breaking a streak', icon: '⚡' },
|
||||
{ key: 'second_cycle', tier: 'legendary', label: 'Second Cycle', description: 'Complete two full cycles', icon: '🌀' },
|
||||
{ key: 'reached_principal', tier: 'legendary', label: 'Principal', description: 'Reach Principal level — maximum rank', icon: '👑' },
|
||||
// Content — domain mastery
|
||||
{ key: 'governance_nerd', tier: 'content', label: 'Governance Nerd', description: 'Complete all published governance topics', icon: '⚖️' },
|
||||
{ key: 'process_architect', tier: 'content', label: 'Process Architect', description: 'Complete all published process topics', icon: '🏗️' },
|
||||
{ key: 'deep_reader', tier: 'content', label: 'Deep Reader', description: 'Complete 50 or more unique topics', icon: '📚' },
|
||||
] as const;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function requireId(ids: Map<string, string>, name: string): string {
|
||||
const id = ids.get(name);
|
||||
if (id === undefined) throw new Error(`Collection ID not found: ${name}`);
|
||||
return id;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Migration
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function run(): Promise<void> {
|
||||
console.log('Connecting to PocketBase...');
|
||||
const pb = new PocketBase(POCKETBASE_URL);
|
||||
await pb.admins.authWithPassword(POCKETBASE_ADMIN_EMAIL, POCKETBASE_ADMIN_PASSWORD);
|
||||
console.log('Authenticated.\n');
|
||||
|
||||
// Snapshot of existing collections before we begin
|
||||
const existingCollections = await pb.collections.getFullList();
|
||||
const ids = new Map<string, string>();
|
||||
for (const col of existingCollections) {
|
||||
ids.set(col.name, col.id);
|
||||
}
|
||||
const existingNames = new Set(ids.keys());
|
||||
|
||||
// Create a collection only if it doesn't already exist
|
||||
async function ensureCollection(params: { name: string; type: string; schema: FieldDef[] }): Promise<void> {
|
||||
if (ids.has(params.name)) {
|
||||
console.log(` skip ${params.name}`);
|
||||
return;
|
||||
}
|
||||
const created = await pb.collections.create(params as Record<string, unknown>);
|
||||
ids.set(created.name, created.id);
|
||||
console.log(` create ${created.name}`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Extend users (auth collection — already exists in PocketBase)
|
||||
// ---------------------------------------------------------------------------
|
||||
console.log('users:');
|
||||
const usersCol = await pb.collections.getFirstListItem<CollectionModel>('name="users"');
|
||||
ids.set('users', usersCol.id);
|
||||
|
||||
const hasRole = usersCol.schema.some(s => s.name === 'role');
|
||||
if (!hasRole) {
|
||||
const updateBody: Record<string, unknown> = {
|
||||
schema: [
|
||||
...usersCol.schema,
|
||||
field.select('role', ['admin', 'employee'], true),
|
||||
field.text('display_name'),
|
||||
field.file('avatar', ['image/jpeg', 'image/png', 'image/webp']),
|
||||
],
|
||||
};
|
||||
await pb.collections.update(usersCol.id, updateBody);
|
||||
console.log(' extended with role, display_name, avatar');
|
||||
} else {
|
||||
console.log(' skip users (already extended)');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// source_documents
|
||||
// ---------------------------------------------------------------------------
|
||||
console.log('\ncollections:');
|
||||
await ensureCollection({
|
||||
name: 'source_documents',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.text('filename', true),
|
||||
field.file('file', ['application/pdf', 'text/markdown', 'text/x-markdown', 'text/plain']),
|
||||
field.select('format', ['pdf', 'md', 'txt'], true),
|
||||
field.select('status', ['processing', 'processed', 'failed'], true),
|
||||
field.date('ingested_at'),
|
||||
field.number('chunk_count'),
|
||||
field.relation('created_by', requireId(ids, 'users')),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// themes
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'themes',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.text('title', true),
|
||||
field.text('description'),
|
||||
field.select('status', ['draft', 'published'], true),
|
||||
field.relation('source_documents', requireId(ids, 'source_documents'), null),
|
||||
field.relation('approved_by', requireId(ids, 'users')),
|
||||
field.date('approved_at'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// topics — create without self-referential fields first
|
||||
// ---------------------------------------------------------------------------
|
||||
const topicsBaseSchema: FieldDef[] = [
|
||||
field.relation('theme', requireId(ids, 'themes'), 1, true),
|
||||
field.text('title', true),
|
||||
field.editor('body'),
|
||||
field.select('difficulty', ['introductory', 'intermediate', 'advanced'], true),
|
||||
field.number('complexity_weight'),
|
||||
field.select('status', ['draft', 'published'], true),
|
||||
field.json('key_terms'),
|
||||
field.json('qdrant_chunk_ids'),
|
||||
];
|
||||
const topicsIsNew = !existingNames.has('topics');
|
||||
await ensureCollection({ name: 'topics', type: 'base', schema: topicsBaseSchema });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// micro_learnings
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'micro_learnings',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('topic', requireId(ids, 'topics'), 1, true),
|
||||
field.select('type', [
|
||||
'concept_explainer', 'scenario_quiz', 'misconceptions', 'how_to',
|
||||
'comparison_card', 'reflection_prompt', 'flashcard_set', 'case_study',
|
||||
'glossary_anchor', 'myth_vs_evidence',
|
||||
], true),
|
||||
field.json('content'),
|
||||
field.select('status', ['queued', 'generated', 'published', 'rejected'], true),
|
||||
field.text('generation_model'),
|
||||
field.date('generated_at'),
|
||||
field.date('published_at'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// curriculum_versions
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'curriculum_versions',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.number('version', true),
|
||||
field.select('status', ['draft', 'active', 'superseded'], true),
|
||||
field.date('generated_at'),
|
||||
field.relation('approved_by', requireId(ids, 'users')),
|
||||
field.date('approved_at'),
|
||||
field.text('generation_notes'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// curriculum_weeks
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'curriculum_weeks',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('curriculum_version', requireId(ids, 'curriculum_versions'), 1, true),
|
||||
field.number('week_number', true),
|
||||
field.relation('theme', requireId(ids, 'themes'), 1, true),
|
||||
field.relation('topics', requireId(ids, 'topics'), null),
|
||||
field.json('topic_order'),
|
||||
field.number('estimated_duration_minutes'),
|
||||
field.text('admin_notes'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// employee_curriculum_state
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'employee_curriculum_state',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('user', requireId(ids, 'users'), 1, true),
|
||||
field.number('current_cycle', true),
|
||||
field.number('current_week', true),
|
||||
field.date('start_date'),
|
||||
field.relation('active_version', requireId(ids, 'curriculum_versions')),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// session_completions
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'session_completions',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('user', requireId(ids, 'users'), 1, true),
|
||||
field.relation('topic', requireId(ids, 'topics'), 1, true),
|
||||
field.relation('micro_learning', requireId(ids, 'micro_learnings'), 1, true),
|
||||
field.number('week_number', true),
|
||||
field.number('cycle', true),
|
||||
field.date('completed_at'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// gamification_profiles
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'gamification_profiles',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('user', requireId(ids, 'users'), 1, true),
|
||||
field.number('total_commits'),
|
||||
field.select('current_level', ['intern', 'junior', 'medior', 'senior', 'staff', 'principal']),
|
||||
field.number('current_streak_weeks'),
|
||||
field.number('longest_streak_weeks'),
|
||||
field.json('types_used'),
|
||||
field.number('last_active_week'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// badges
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'badges',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.text('key', true),
|
||||
field.select('tier', ['bronze', 'silver', 'gold', 'legendary', 'content'], true),
|
||||
field.text('label', true),
|
||||
field.text('description'),
|
||||
field.text('icon'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// employee_badges
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'employee_badges',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('user', requireId(ids, 'users'), 1, true),
|
||||
field.relation('badge', requireId(ids, 'badges'), 1, true),
|
||||
field.date('earned_at'),
|
||||
field.number('cycle'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// milestone_cards
|
||||
// ---------------------------------------------------------------------------
|
||||
await ensureCollection({
|
||||
name: 'milestone_cards',
|
||||
type: 'base',
|
||||
schema: [
|
||||
field.relation('user', requireId(ids, 'users'), 1, true),
|
||||
field.number('cycle', true),
|
||||
field.number('week', true),
|
||||
field.number('total_commits'),
|
||||
field.number('streak_weeks'),
|
||||
field.json('badge_keys'),
|
||||
],
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Add self-referential relations to topics (requires topics ID to exist first)
|
||||
// ---------------------------------------------------------------------------
|
||||
if (topicsIsNew) {
|
||||
console.log('\ntopics self-refs:');
|
||||
const topicsId = requireId(ids, 'topics');
|
||||
const updateBody: Record<string, unknown> = {
|
||||
schema: [
|
||||
...topicsBaseSchema,
|
||||
field.relation('related_topics', topicsId, null),
|
||||
field.relation('prerequisite_topics', topicsId, null),
|
||||
field.relation('contrast_topics', topicsId, null),
|
||||
],
|
||||
};
|
||||
await pb.collections.update(topicsId, updateBody);
|
||||
console.log(' updated topics with related_topics, prerequisite_topics, contrast_topics');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Seed badges
|
||||
// ---------------------------------------------------------------------------
|
||||
console.log('\nbadges:');
|
||||
const existingBadges = await pb.collection('badges').getFullList<{ key: string }>();
|
||||
const existingBadgeKeys = new Set(existingBadges.map(b => b.key));
|
||||
|
||||
for (const badge of BADGES) {
|
||||
if (existingBadgeKeys.has(badge.key)) {
|
||||
console.log(` skip ${badge.key}`);
|
||||
continue;
|
||||
}
|
||||
await pb.collection('badges').create(badge);
|
||||
console.log(` seed ${badge.key}`);
|
||||
}
|
||||
|
||||
console.log('\nDone.');
|
||||
}
|
||||
|
||||
run().catch((err: unknown) => {
|
||||
console.error('Migration failed:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
76
app/services/ingestion/src/migrations/002_qdrant_setup.ts
Normal file
76
app/services/ingestion/src/migrations/002_qdrant_setup.ts
Normal file
@@ -0,0 +1,76 @@
|
||||
import 'dotenv/config';
|
||||
import { QdrantClient } from '@qdrant/js-client-rest';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Env validation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const QDRANT_URL = process.env['QDRANT_URL'] ?? '';
|
||||
const QDRANT_API_KEY = process.env['QDRANT_API_KEY'] ?? '';
|
||||
|
||||
if (!QDRANT_URL) {
|
||||
console.error('Missing env var: QDRANT_URL');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Collection definitions
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const VECTOR_SIZE = 1536; // text-embedding-3-small
|
||||
const DISTANCE = 'Cosine' as const;
|
||||
|
||||
const COLLECTIONS = [
|
||||
{
|
||||
name: 'source_chunks',
|
||||
// Payload indices for R42 context-weighted retrieval (boost by theme_id)
|
||||
payloadIndices: ['theme_id', 'topic_id', 'source_document_id', 'format'],
|
||||
},
|
||||
{
|
||||
name: 'topic_summaries',
|
||||
payloadIndices: ['theme_id', 'topic_id'],
|
||||
},
|
||||
] as const;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Setup
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function run(): Promise<void> {
|
||||
console.log('Connecting to Qdrant...');
|
||||
const client = new QdrantClient({
|
||||
url: QDRANT_URL,
|
||||
...(QDRANT_API_KEY ? { apiKey: QDRANT_API_KEY } : {}),
|
||||
});
|
||||
|
||||
const { collections } = await client.getCollections();
|
||||
const existingNames = new Set(collections.map(c => c.name));
|
||||
|
||||
for (const col of COLLECTIONS) {
|
||||
if (existingNames.has(col.name)) {
|
||||
console.log(` skip ${col.name}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
await client.createCollection(col.name, {
|
||||
vectors: { size: VECTOR_SIZE, distance: DISTANCE },
|
||||
});
|
||||
console.log(` create ${col.name}`);
|
||||
|
||||
// Keyword payload indices for efficient filtering
|
||||
for (const field of col.payloadIndices) {
|
||||
await client.createPayloadIndex(col.name, {
|
||||
field_name: field,
|
||||
field_schema: 'keyword',
|
||||
});
|
||||
console.log(` index ${col.name}.${field}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('\nDone.');
|
||||
}
|
||||
|
||||
run().catch((err: unknown) => {
|
||||
console.error('Qdrant setup failed:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
246
app/services/ingestion/src/pipeline/chunk.ts
Normal file
246
app/services/ingestion/src/pipeline/chunk.ts
Normal file
@@ -0,0 +1,246 @@
|
||||
import { v4 as uuid } from 'uuid';
|
||||
import type { Chunk, DocumentFormat } from '../types.js';
|
||||
|
||||
const MD_MIN = 100;
|
||||
const MD_MAX = 1500;
|
||||
const TXT_WINDOW = 800;
|
||||
const TXT_OVERLAP = 150;
|
||||
const PDF_MIN = 100;
|
||||
const PDF_MAX = 1200;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// MD chunking — heading-based
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface HeadingSection {
|
||||
level: number;
|
||||
heading: string;
|
||||
parent: string | null;
|
||||
content: string;
|
||||
}
|
||||
|
||||
function parseMdSections(text: string): HeadingSection[] {
|
||||
const lines = text.split('\n');
|
||||
const sections: HeadingSection[] = [];
|
||||
let current: HeadingSection | null = null;
|
||||
const parentStack: { level: number; heading: string }[] = [];
|
||||
|
||||
for (const line of lines) {
|
||||
const h3 = line.match(/^### (.+)/);
|
||||
const h2 = line.match(/^## (.+)/);
|
||||
const h1 = line.match(/^# (.+)/);
|
||||
const match = h3 ?? h2 ?? h1;
|
||||
|
||||
if (match) {
|
||||
if (current) sections.push(current);
|
||||
const level = h1 ? 1 : h2 ? 2 : 3;
|
||||
const heading = match[1] ?? line;
|
||||
|
||||
// Maintain parent stack
|
||||
while (parentStack.length > 0 && (parentStack[parentStack.length - 1]?.level ?? 0) >= level) {
|
||||
parentStack.pop();
|
||||
}
|
||||
const parent = parentStack[parentStack.length - 1]?.heading ?? null;
|
||||
parentStack.push({ level, heading });
|
||||
|
||||
current = { level, heading, parent, content: '' };
|
||||
} else if (current) {
|
||||
current.content += line + '\n';
|
||||
}
|
||||
}
|
||||
if (current) sections.push(current);
|
||||
return sections;
|
||||
}
|
||||
|
||||
function splitOnParagraphs(text: string, maxSize: number): string[] {
|
||||
const paragraphs = text.split(/\n\n+/);
|
||||
const parts: string[] = [];
|
||||
let current = '';
|
||||
|
||||
for (const para of paragraphs) {
|
||||
if ((current + para).length > maxSize && current.length > 0) {
|
||||
parts.push(current.trim());
|
||||
current = para;
|
||||
} else {
|
||||
current = current ? current + '\n\n' + para : para;
|
||||
}
|
||||
}
|
||||
if (current.trim()) parts.push(current.trim());
|
||||
return parts;
|
||||
}
|
||||
|
||||
function chunkMd(text: string, documentId: string): Chunk[] {
|
||||
const sections = parseMdSections(text);
|
||||
const merged: HeadingSection[] = [];
|
||||
|
||||
for (let i = 0; i < sections.length; i++) {
|
||||
const sec = sections[i];
|
||||
if (sec === undefined) continue;
|
||||
const fullText = `${'#'.repeat(sec.level)} ${sec.heading}\n\n${sec.content}`.trim();
|
||||
|
||||
if (fullText.length < MD_MIN && i + 1 < sections.length) {
|
||||
// Merge with next sibling by appending to next's content prefix
|
||||
const next = sections[i + 1];
|
||||
if (next !== undefined) {
|
||||
next.content = fullText + '\n\n' + next.content;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
merged.push(sec);
|
||||
}
|
||||
|
||||
const chunks: Chunk[] = [];
|
||||
let globalIndex = 0;
|
||||
|
||||
for (const sec of merged) {
|
||||
const fullText = `${'#'.repeat(sec.level)} ${sec.heading}\n\n${sec.content}`.trim();
|
||||
const parts = fullText.length > MD_MAX ? splitOnParagraphs(fullText, MD_MAX) : [fullText];
|
||||
|
||||
for (const part of parts) {
|
||||
if (part.length < 1) continue;
|
||||
chunks.push({
|
||||
id: uuid(),
|
||||
documentId,
|
||||
text: part,
|
||||
format: 'md',
|
||||
index: globalIndex++,
|
||||
metadata: {
|
||||
headingLevel: sec.level,
|
||||
headingText: sec.heading,
|
||||
parentHeading: sec.parent ?? undefined,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return chunks;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// TXT chunking — sliding window
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function splitSentences(text: string): string[] {
|
||||
return text.match(/[^.!?]+[.!?]+\s*/g) ?? [text];
|
||||
}
|
||||
|
||||
function chunkTxt(text: string, documentId: string): Chunk[] {
|
||||
const paragraphs = text.split(/\n\n+/).filter(p => p.trim().length > 0);
|
||||
const chunks: Chunk[] = [];
|
||||
let current = '';
|
||||
let index = 0;
|
||||
const total = text.length;
|
||||
|
||||
const pushChunk = (t: string) => {
|
||||
const pos = index * TXT_WINDOW;
|
||||
chunks.push({
|
||||
id: uuid(),
|
||||
documentId,
|
||||
text: t.trim(),
|
||||
format: 'txt',
|
||||
index: index++,
|
||||
metadata: {
|
||||
approximatePosition:
|
||||
pos < total * 0.25 ? 'start' : pos > total * 0.75 ? 'end' : 'middle',
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
for (const para of paragraphs) {
|
||||
if ((current + ' ' + para).trim().length <= TXT_WINDOW) {
|
||||
current = (current + ' ' + para).trim();
|
||||
} else {
|
||||
if (current.length >= MD_MIN) pushChunk(current);
|
||||
// Para may itself exceed window — split on sentences
|
||||
if (para.length > TXT_WINDOW) {
|
||||
const sentences = splitSentences(para);
|
||||
let buf = '';
|
||||
for (const sent of sentences) {
|
||||
if ((buf + sent).length > TXT_WINDOW && buf.length >= MD_MIN) {
|
||||
pushChunk(buf);
|
||||
// Keep overlap
|
||||
const words = buf.split(' ');
|
||||
buf = words.slice(-Math.floor(TXT_OVERLAP / 5)).join(' ') + ' ' + sent;
|
||||
} else {
|
||||
buf += sent;
|
||||
}
|
||||
}
|
||||
current = buf;
|
||||
} else {
|
||||
current = para;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (current.trim().length >= MD_MIN) pushChunk(current);
|
||||
|
||||
return chunks;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// PDF chunking — page + paragraph
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function chunkPdf(text: string, documentId: string): Chunk[] {
|
||||
const pages = text.split('---PAGE---');
|
||||
const chunks: Chunk[] = [];
|
||||
let globalIndex = 0;
|
||||
|
||||
for (let pageIdx = 0; pageIdx < pages.length; pageIdx++) {
|
||||
const page = pages[pageIdx];
|
||||
if (page === undefined || !page.trim()) continue;
|
||||
|
||||
const paragraphs = page.split(/\n\n+/).filter(p => p.trim().length > 0);
|
||||
let chunkOnPage = 0;
|
||||
let accumulator = '';
|
||||
|
||||
const flushAccumulator = () => {
|
||||
const t = accumulator.trim();
|
||||
if (t.length >= PDF_MIN) {
|
||||
chunks.push({
|
||||
id: uuid(),
|
||||
documentId,
|
||||
text: t,
|
||||
format: 'pdf',
|
||||
index: globalIndex++,
|
||||
metadata: { pageNumber: pageIdx + 1, chunkIndexOnPage: chunkOnPage++ },
|
||||
});
|
||||
}
|
||||
accumulator = '';
|
||||
};
|
||||
|
||||
for (const para of paragraphs) {
|
||||
if ((accumulator + '\n\n' + para).trim().length > PDF_MAX) {
|
||||
flushAccumulator();
|
||||
// Para may exceed max on its own — hard split at sentence boundary
|
||||
if (para.length > PDF_MAX) {
|
||||
const sentences = splitSentences(para);
|
||||
for (const sent of sentences) {
|
||||
if ((accumulator + sent).length > PDF_MAX && accumulator.length >= PDF_MIN) {
|
||||
flushAccumulator();
|
||||
}
|
||||
accumulator += sent;
|
||||
}
|
||||
} else {
|
||||
accumulator = para;
|
||||
}
|
||||
} else {
|
||||
accumulator = accumulator ? accumulator + '\n\n' + para : para;
|
||||
}
|
||||
}
|
||||
flushAccumulator();
|
||||
}
|
||||
|
||||
return chunks;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Public entry
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function chunk(text: string, format: DocumentFormat, documentId: string): Chunk[] {
|
||||
switch (format) {
|
||||
case 'md': return chunkMd(text, documentId);
|
||||
case 'txt': return chunkTxt(text, documentId);
|
||||
case 'pdf': return chunkPdf(text, documentId);
|
||||
}
|
||||
}
|
||||
20
app/services/ingestion/src/pipeline/clean.ts
Normal file
20
app/services/ingestion/src/pipeline/clean.ts
Normal file
@@ -0,0 +1,20 @@
|
||||
import type { Chunk } from '../types.js';
|
||||
|
||||
const MIN_CLEAN_LENGTH = 80;
|
||||
|
||||
export function clean(chunks: Chunk[]): Chunk[] {
|
||||
return chunks
|
||||
.map(c => ({ ...c, text: cleanText(c.text) }))
|
||||
.filter(c => c.text.length >= MIN_CLEAN_LENGTH);
|
||||
}
|
||||
|
||||
function cleanText(text: string): string {
|
||||
return text
|
||||
.normalize('NFC')
|
||||
// Remove null bytes and non-printable characters (keep tabs + newlines)
|
||||
.replace(/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/g, '')
|
||||
// Collapse 3+ consecutive newlines to 2
|
||||
.replace(/\n{3,}/g, '\n\n')
|
||||
// Trim leading/trailing whitespace
|
||||
.trim();
|
||||
}
|
||||
102
app/services/ingestion/src/pipeline/embed.ts
Normal file
102
app/services/ingestion/src/pipeline/embed.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
import { v4 as uuid } from 'uuid';
|
||||
import { openai, EMBEDDING_MODEL, EMBEDDING_BATCH_SIZE } from '../lib/openai.js';
|
||||
import { qdrant, QDRANT_COLLECTIONS } from '../lib/qdrant.js';
|
||||
import { updateTopicQdrantIds } from './write.js';
|
||||
import type { Chunk, WrittenTopic, SourceChunkPayload, TopicSummaryPayload } from '../types.js';
|
||||
|
||||
async function embedTexts(texts: string[]): Promise<number[][]> {
|
||||
const vectors: number[][] = [];
|
||||
|
||||
for (let i = 0; i < texts.length; i += EMBEDDING_BATCH_SIZE) {
|
||||
const batch = texts.slice(i, i + EMBEDDING_BATCH_SIZE);
|
||||
const response = await openai.embeddings.create({
|
||||
model: EMBEDDING_MODEL,
|
||||
input: batch,
|
||||
});
|
||||
for (const item of response.data) {
|
||||
vectors.push(item.embedding);
|
||||
}
|
||||
}
|
||||
|
||||
return vectors;
|
||||
}
|
||||
|
||||
export async function embedAndStore(
|
||||
chunks: Chunk[],
|
||||
writtenTopics: WrittenTopic[],
|
||||
onProgress: (embedded: number) => void,
|
||||
): Promise<void> {
|
||||
// Build chunk → topic mapping
|
||||
const chunkTopicMap = new Map<string, string>();
|
||||
const chunkThemeMap = new Map<string, string>();
|
||||
for (const topic of writtenTopics) {
|
||||
for (const chunkId of topic.sourceChunkIds) {
|
||||
chunkTopicMap.set(chunkId, topic.id);
|
||||
chunkThemeMap.set(chunkId, topic.themeId);
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Source chunks
|
||||
// -------------------------------------------------------------------------
|
||||
const chunkTexts = chunks.map(c => c.text);
|
||||
const chunkVectors = await embedTexts(chunkTexts);
|
||||
|
||||
const sourcePoints = chunks.map((c, i) => {
|
||||
const vector = chunkVectors[i];
|
||||
if (!vector) throw new Error(`Missing embedding for chunk index ${i}`);
|
||||
|
||||
const payload: SourceChunkPayload = {
|
||||
source_document_id: c.documentId,
|
||||
chunk_index: c.index,
|
||||
text: c.text,
|
||||
theme_id: chunkThemeMap.get(c.id) ?? null,
|
||||
topic_id: chunkTopicMap.get(c.id) ?? null,
|
||||
format: c.format,
|
||||
};
|
||||
|
||||
return { id: c.id, vector, payload };
|
||||
});
|
||||
|
||||
// Upsert in batches
|
||||
for (let i = 0; i < sourcePoints.length; i += EMBEDDING_BATCH_SIZE) {
|
||||
const batch = sourcePoints.slice(i, i + EMBEDDING_BATCH_SIZE);
|
||||
await qdrant.upsert(QDRANT_COLLECTIONS.SOURCE_CHUNKS, { points: batch });
|
||||
onProgress(i + batch.length);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Topic summaries
|
||||
// -------------------------------------------------------------------------
|
||||
const topicTexts = writtenTopics.map(t => t.body);
|
||||
const topicVectors = await embedTexts(topicTexts);
|
||||
|
||||
const summaryPoints = writtenTopics.map((topic, i) => {
|
||||
const vector = topicVectors[i];
|
||||
if (!vector) throw new Error(`Missing embedding for topic index ${i}`);
|
||||
|
||||
const payload: TopicSummaryPayload = {
|
||||
topic_id: topic.id,
|
||||
theme_id: topic.themeId,
|
||||
title: topic.title,
|
||||
text: topic.body,
|
||||
};
|
||||
|
||||
return { id: uuid(), vector, payload };
|
||||
});
|
||||
|
||||
for (let i = 0; i < summaryPoints.length; i += EMBEDDING_BATCH_SIZE) {
|
||||
const batch = summaryPoints.slice(i, i + EMBEDDING_BATCH_SIZE);
|
||||
await qdrant.upsert(QDRANT_COLLECTIONS.TOPIC_SUMMARIES, { points: batch });
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Update topics.qdrant_chunk_ids in PocketBase
|
||||
// -------------------------------------------------------------------------
|
||||
for (const topic of writtenTopics) {
|
||||
const qdrantIds = topic.sourceChunkIds.filter(id => chunkTopicMap.get(id) === topic.id);
|
||||
if (qdrantIds.length > 0) {
|
||||
await updateTopicQdrantIds(topic.id, qdrantIds);
|
||||
}
|
||||
}
|
||||
}
|
||||
40
app/services/ingestion/src/pipeline/extract.ts
Normal file
40
app/services/ingestion/src/pipeline/extract.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
import fs from 'node:fs/promises';
|
||||
import type { DocumentFormat } from '../types.js';
|
||||
|
||||
// Lazy import — pdf-parse has side effects on module load
|
||||
async function getPdfParse() {
|
||||
const mod = await import('pdf-parse');
|
||||
return mod.default ?? mod;
|
||||
}
|
||||
|
||||
export async function extract(filePath: string, format: DocumentFormat): Promise<string> {
|
||||
let fileBuffer: Buffer;
|
||||
try {
|
||||
fileBuffer = await fs.readFile(filePath);
|
||||
} catch {
|
||||
throw new Error('file_not_found');
|
||||
}
|
||||
|
||||
if (format === 'txt' || format === 'md') {
|
||||
return fileBuffer.toString('utf-8');
|
||||
}
|
||||
|
||||
// PDF — collect one text string per page via pagerender callback
|
||||
const pdfParse = await getPdfParse();
|
||||
const pageTexts: string[] = [];
|
||||
|
||||
await pdfParse(fileBuffer, {
|
||||
pagerender: (pageData: { getTextContent: () => Promise<{ items: Array<{ str: string }> }> }) =>
|
||||
pageData.getTextContent().then(tc => {
|
||||
const text = tc.items.map(i => i.str).join(' ').trim();
|
||||
if (text) pageTexts.push(text);
|
||||
return text;
|
||||
}),
|
||||
});
|
||||
|
||||
if (pageTexts.length === 0) {
|
||||
throw new Error('pdf_extraction_empty');
|
||||
}
|
||||
|
||||
return pageTexts.join('\n\n---PAGE---\n\n');
|
||||
}
|
||||
181
app/services/ingestion/src/pipeline/structure.ts
Normal file
181
app/services/ingestion/src/pipeline/structure.ts
Normal file
@@ -0,0 +1,181 @@
|
||||
import { anthropic, MODELS } from '../lib/anthropic.js';
|
||||
import { DraftKBSchema, type Chunk, type DraftKB, type DraftTheme, type DraftTopic, type DocumentFormat } from '../types.js';
|
||||
|
||||
const BATCH_SIZE = 40;
|
||||
const BATCH_OVERLAP = 5;
|
||||
const LARGE_DOC_THRESHOLD = 60;
|
||||
|
||||
const SYSTEM_PROMPT = `You are a knowledge architect. Your task is to analyse a set of text chunks from a source document and extract a structured knowledge base.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no explanation, no markdown fences.
|
||||
|
||||
Rules:
|
||||
- Group related content into Themes. A Theme is a broad subject area.
|
||||
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
|
||||
- Identify relationships between Topics: related, prerequisite, or contrast.
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A must be understood before Topic B
|
||||
- contrast: Topics that represent opposing approaches or concepts
|
||||
- For each Topic, extract key terms suitable for a glossary.
|
||||
- Assign a complexity weight (1–5) to each Topic.
|
||||
1 = introductory, 5 = advanced
|
||||
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
|
||||
- Draft a description for each Theme (1–2 sentences).
|
||||
- Every Topic must reference the chunk IDs that contributed to it.
|
||||
|
||||
Output schema:
|
||||
{
|
||||
"themes": [
|
||||
{
|
||||
"title": "string",
|
||||
"description": "string",
|
||||
"topics": [
|
||||
{
|
||||
"title": "string",
|
||||
"body": "string",
|
||||
"difficulty": "introductory" | "intermediate" | "advanced",
|
||||
"complexityWeight": 1-5,
|
||||
"keyTerms": ["string"],
|
||||
"sourceChunkIds": ["chunk-id"],
|
||||
"relationships": {
|
||||
"related": ["topic title"],
|
||||
"prerequisites": ["topic title"],
|
||||
"contrasts": ["topic title"]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}`;
|
||||
|
||||
const STRICT_SUFFIX = '\n\nCRITICAL: Your entire response must be valid JSON only. No text before or after.';
|
||||
|
||||
function buildUserPrompt(chunks: Chunk[], filename: string, format: DocumentFormat, strict: boolean): string {
|
||||
const chunkText = chunks
|
||||
.map(c => `[CHUNK-${c.id}]\n${c.text}`)
|
||||
.join('\n\n');
|
||||
|
||||
return `Source document: ${filename}\nFormat: ${format}\n\nChunks:\n${chunkText}\n\nExtract the knowledge base structure from these chunks.${strict ? STRICT_SUFFIX : ''}`;
|
||||
}
|
||||
|
||||
async function callClaude(chunks: Chunk[], filename: string, format: DocumentFormat, strict: boolean): Promise<DraftKB> {
|
||||
const response = await anthropic.messages.create({
|
||||
model: MODELS.SONNET,
|
||||
max_tokens: 8000,
|
||||
temperature: 0,
|
||||
system: SYSTEM_PROMPT,
|
||||
messages: [{ role: 'user', content: buildUserPrompt(chunks, filename, format, strict) }],
|
||||
});
|
||||
|
||||
const textBlock = response.content.find(b => b.type === 'text');
|
||||
if (!textBlock || textBlock.type !== 'text') {
|
||||
throw new Error('structure_extraction_failed: no text block in response');
|
||||
}
|
||||
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(textBlock.text);
|
||||
} catch {
|
||||
if (strict) throw new Error('structure_extraction_failed');
|
||||
return callClaude(chunks, filename, format, true);
|
||||
}
|
||||
|
||||
const result = DraftKBSchema.safeParse(parsed);
|
||||
if (!result.success) {
|
||||
if (strict) throw new Error('structure_extraction_failed');
|
||||
return callClaude(chunks, filename, format, true);
|
||||
}
|
||||
|
||||
if (result.data.themes.length === 0) {
|
||||
throw new Error('no_structure_found');
|
||||
}
|
||||
|
||||
return result.data;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// DraftKB merge (for large documents processed in batches)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface MergedTheme {
|
||||
title: string;
|
||||
description: string;
|
||||
topicMap: Map<string, DraftTopic>;
|
||||
}
|
||||
|
||||
function mergeDraftKBs(batches: DraftKB[]): DraftKB {
|
||||
const themeMap = new Map<string, MergedTheme>();
|
||||
|
||||
for (const kb of batches) {
|
||||
for (const theme of kb.themes) {
|
||||
const key = theme.title.toLowerCase().trim();
|
||||
const existing = themeMap.get(key);
|
||||
|
||||
if (!existing) {
|
||||
themeMap.set(key, {
|
||||
title: theme.title,
|
||||
description: theme.description,
|
||||
topicMap: new Map(theme.topics.map(t => [t.title.toLowerCase().trim(), { ...t }])),
|
||||
});
|
||||
} else {
|
||||
if (theme.description.length > existing.description.length) {
|
||||
existing.description = theme.description;
|
||||
}
|
||||
for (const topic of theme.topics) {
|
||||
const tKey = topic.title.toLowerCase().trim();
|
||||
const existingTopic = existing.topicMap.get(tKey);
|
||||
if (!existingTopic) {
|
||||
existing.topicMap.set(tKey, { ...topic });
|
||||
} else {
|
||||
existingTopic.body =
|
||||
existingTopic.body.length >= topic.body.length ? existingTopic.body : topic.body;
|
||||
existingTopic.sourceChunkIds = [
|
||||
...new Set([...existingTopic.sourceChunkIds, ...topic.sourceChunkIds]),
|
||||
];
|
||||
existingTopic.relationships = {
|
||||
related: [...new Set([...existingTopic.relationships.related, ...topic.relationships.related])],
|
||||
prerequisites: [...new Set([...existingTopic.relationships.prerequisites, ...topic.relationships.prerequisites])],
|
||||
contrasts: [...new Set([...existingTopic.relationships.contrasts, ...topic.relationships.contrasts])],
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const themes: DraftTheme[] = [...themeMap.values()].map(t => ({
|
||||
title: t.title,
|
||||
description: t.description,
|
||||
topics: [...t.topicMap.values()],
|
||||
}));
|
||||
|
||||
return { themes };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Public entry
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function extractStructure(
|
||||
chunks: Chunk[],
|
||||
filename: string,
|
||||
format: DocumentFormat,
|
||||
): Promise<DraftKB> {
|
||||
if (chunks.length <= LARGE_DOC_THRESHOLD) {
|
||||
return callClaude(chunks, filename, format, false);
|
||||
}
|
||||
|
||||
const batches: DraftKB[] = [];
|
||||
let start = 0;
|
||||
|
||||
while (start < chunks.length) {
|
||||
const end = Math.min(start + BATCH_SIZE, chunks.length);
|
||||
const batchChunks = chunks.slice(start, end);
|
||||
const batchKB = await callClaude(batchChunks, filename, format, false);
|
||||
batches.push(batchKB);
|
||||
if (end >= chunks.length) break;
|
||||
start = end - BATCH_OVERLAP;
|
||||
}
|
||||
|
||||
return mergeDraftKBs(batches);
|
||||
}
|
||||
91
app/services/ingestion/src/pipeline/write.ts
Normal file
91
app/services/ingestion/src/pipeline/write.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
import { getPocketBase } from '../lib/pocketbase.js';
|
||||
import type { Chunk, DraftKB, WrittenTopic } from '../types.js';
|
||||
|
||||
export async function writeToPocketBase(
|
||||
draftKB: DraftKB,
|
||||
documentId: string,
|
||||
chunkCount: number,
|
||||
): Promise<WrittenTopic[]> {
|
||||
const pb = await getPocketBase();
|
||||
|
||||
await pb.collection('source_documents').update(documentId, {
|
||||
status: 'processed',
|
||||
chunk_count: chunkCount,
|
||||
ingested_at: new Date().toISOString(),
|
||||
});
|
||||
|
||||
const writtenTopics: WrittenTopic[] = [];
|
||||
// title → PocketBase topic ID (for relationship resolution)
|
||||
const topicIdByTitle = new Map<string, string>();
|
||||
|
||||
// Pass 1: create all themes and topics (without relationships)
|
||||
for (const theme of draftKB.themes) {
|
||||
const themeRecord = await pb.collection('themes').create({
|
||||
title: theme.title,
|
||||
description: theme.description,
|
||||
status: 'draft',
|
||||
source_documents: [documentId],
|
||||
});
|
||||
|
||||
for (const topic of theme.topics) {
|
||||
const topicRecord = await pb.collection('topics').create({
|
||||
theme: themeRecord.id,
|
||||
title: topic.title,
|
||||
body: topic.body,
|
||||
difficulty: topic.difficulty,
|
||||
complexity_weight: topic.complexityWeight,
|
||||
status: 'draft',
|
||||
key_terms: topic.keyTerms,
|
||||
qdrant_chunk_ids: [],
|
||||
related_topics: [],
|
||||
prerequisite_topics: [],
|
||||
contrast_topics: [],
|
||||
});
|
||||
|
||||
topicIdByTitle.set(topic.title.toLowerCase().trim(), topicRecord.id);
|
||||
|
||||
writtenTopics.push({
|
||||
id: topicRecord.id,
|
||||
title: topic.title,
|
||||
themeId: themeRecord.id,
|
||||
body: topic.body,
|
||||
sourceChunkIds: topic.sourceChunkIds,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Pass 2: resolve relationships (title → ID lookup)
|
||||
for (const theme of draftKB.themes) {
|
||||
for (const topic of theme.topics) {
|
||||
const topicId = topicIdByTitle.get(topic.title.toLowerCase().trim());
|
||||
if (!topicId) continue;
|
||||
|
||||
const resolve = (titles: string[]): string[] =>
|
||||
titles
|
||||
.map(t => topicIdByTitle.get(t.toLowerCase().trim()))
|
||||
.filter((id): id is string => id !== undefined);
|
||||
|
||||
await pb.collection('topics').update(topicId, {
|
||||
related_topics: resolve(topic.relationships.related),
|
||||
prerequisite_topics: resolve(topic.relationships.prerequisites),
|
||||
contrast_topics: resolve(topic.relationships.contrasts),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return writtenTopics;
|
||||
}
|
||||
|
||||
export async function updateTopicQdrantIds(
|
||||
topicId: string,
|
||||
qdrantChunkIds: string[],
|
||||
): Promise<void> {
|
||||
const pb = await getPocketBase();
|
||||
await pb.collection('topics').update(topicId, { qdrant_chunk_ids: qdrantChunkIds });
|
||||
}
|
||||
|
||||
export async function markDocumentFailed(documentId: string, reason: string): Promise<void> {
|
||||
console.error(`[write] document ${documentId} failed: ${reason}`);
|
||||
const pb = await getPocketBase();
|
||||
await pb.collection('source_documents').update(documentId, { status: 'failed' });
|
||||
}
|
||||
29
app/services/ingestion/src/routes/documents.ts
Normal file
29
app/services/ingestion/src/routes/documents.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
import type { FastifyPluginAsync } from 'fastify';
|
||||
import { IngestBodySchema } from '../types.js';
|
||||
import { createJob, getJob } from '../jobs/queue.js';
|
||||
|
||||
const documentRoutes: FastifyPluginAsync = async (app) => {
|
||||
app.post('/ingest', async (request, reply) => {
|
||||
const parsed = IngestBodySchema.safeParse(request.body);
|
||||
if (!parsed.success) {
|
||||
return reply.status(400).send({ error: 'Invalid request body', details: parsed.error.format() });
|
||||
}
|
||||
const job = createJob(parsed.data);
|
||||
return reply.status(202).send({ jobId: job.id, status: job.status });
|
||||
});
|
||||
|
||||
app.get<{ Params: { jobId: string } }>('/status/:jobId', async (request, reply) => {
|
||||
const job = getJob(request.params.jobId);
|
||||
if (!job) {
|
||||
return reply.status(404).send({ error: 'Job not found' });
|
||||
}
|
||||
return reply.send({
|
||||
jobId: job.id,
|
||||
status: job.status,
|
||||
progress: job.progress,
|
||||
error: job.error,
|
||||
});
|
||||
});
|
||||
};
|
||||
|
||||
export default documentRoutes;
|
||||
145
app/services/ingestion/src/types.ts
Normal file
145
app/services/ingestion/src/types.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
import { z } from 'zod';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Shared primitives
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type DocumentFormat = 'pdf' | 'md' | 'txt';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pipeline: Chunk
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface ChunkMetadata {
|
||||
// MD-specific
|
||||
headingLevel?: number;
|
||||
headingText?: string;
|
||||
parentHeading?: string;
|
||||
// TXT-specific
|
||||
approximatePosition?: 'start' | 'middle' | 'end';
|
||||
// PDF-specific
|
||||
pageNumber?: number;
|
||||
chunkIndexOnPage?: number;
|
||||
}
|
||||
|
||||
export interface Chunk {
|
||||
id: string;
|
||||
documentId: string;
|
||||
text: string;
|
||||
format: DocumentFormat;
|
||||
index: number;
|
||||
metadata: ChunkMetadata;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pipeline: DraftKB (AI output — validated with Zod before use)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const DraftTopicRelationshipsSchema = z.object({
|
||||
related: z.array(z.string()),
|
||||
prerequisites: z.array(z.string()),
|
||||
contrasts: z.array(z.string()),
|
||||
});
|
||||
|
||||
export const DraftTopicSchema = z.object({
|
||||
title: z.string().min(1),
|
||||
body: z.string().min(1),
|
||||
difficulty: z.enum(['introductory', 'intermediate', 'advanced']),
|
||||
complexityWeight: z.number().int().min(1).max(5),
|
||||
keyTerms: z.array(z.string()),
|
||||
sourceChunkIds: z.array(z.string()),
|
||||
relationships: DraftTopicRelationshipsSchema,
|
||||
});
|
||||
|
||||
export const DraftThemeSchema = z.object({
|
||||
title: z.string().min(1),
|
||||
description: z.string().min(1),
|
||||
topics: z.array(DraftTopicSchema).min(1),
|
||||
});
|
||||
|
||||
export const DraftKBSchema = z.object({
|
||||
themes: z.array(DraftThemeSchema).min(1),
|
||||
});
|
||||
|
||||
export type DraftTopic = z.infer<typeof DraftTopicSchema>;
|
||||
export type DraftTheme = z.infer<typeof DraftThemeSchema>;
|
||||
export type DraftKB = z.infer<typeof DraftKBSchema>;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pipeline: PocketBase write result
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface WrittenTopic {
|
||||
id: string;
|
||||
title: string;
|
||||
themeId: string;
|
||||
body: string;
|
||||
sourceChunkIds: string[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Qdrant payload types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface SourceChunkPayload {
|
||||
source_document_id: string;
|
||||
chunk_index: number;
|
||||
text: string;
|
||||
theme_id: string | null;
|
||||
topic_id: string | null;
|
||||
format: DocumentFormat;
|
||||
}
|
||||
|
||||
export interface TopicSummaryPayload {
|
||||
topic_id: string;
|
||||
theme_id: string;
|
||||
title: string;
|
||||
text: string;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Job system
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type JobStatus =
|
||||
| 'queued'
|
||||
| 'extracting'
|
||||
| 'chunking'
|
||||
| 'structuring'
|
||||
| 'writing'
|
||||
| 'embedding'
|
||||
| 'done'
|
||||
| 'failed';
|
||||
|
||||
export interface JobProgress {
|
||||
chunksTotal: number;
|
||||
chunksEmbedded: number;
|
||||
themesFound: number;
|
||||
topicsFound: number;
|
||||
}
|
||||
|
||||
export interface Job {
|
||||
id: string;
|
||||
documentId: string;
|
||||
filename: string;
|
||||
format: DocumentFormat;
|
||||
filePath: string;
|
||||
status: JobStatus;
|
||||
progress: JobProgress;
|
||||
error: string | null;
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// API schemas (Zod — validates external input)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export const IngestBodySchema = z.object({
|
||||
documentId: z.string().min(1),
|
||||
filename: z.string().min(1),
|
||||
format: z.enum(['pdf', 'md', 'txt']),
|
||||
filePath: z.string().min(1),
|
||||
});
|
||||
|
||||
export type IngestBody = z.infer<typeof IngestBodySchema>;
|
||||
16
app/services/ingestion/tsconfig.json
Normal file
16
app/services/ingestion/tsconfig.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "Node16",
|
||||
"moduleResolution": "Node16",
|
||||
"rootDir": "src",
|
||||
"outDir": "dist",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true
|
||||
},
|
||||
"include": ["src/**/*.ts"],
|
||||
"exclude": ["dist", "node_modules"]
|
||||
}
|
||||
0
app/services/progress/.gitkeep
Normal file
0
app/services/progress/.gitkeep
Normal file
279
docs/architecture.md
Normal file
279
docs/architecture.md
Normal file
@@ -0,0 +1,279 @@
|
||||
# Architecture: employee learning platform
|
||||
|
||||
## Overview
|
||||
|
||||
A mobile-first progressive web application that provides employees with a structured knowledge library, a 26-week perpetual learning curriculum, and an AI-powered assistant (R42). The knowledge base is the single source of truth for all content, micro learnings, curriculum scheduling, and chat retrieval.
|
||||
|
||||
---
|
||||
|
||||
## System domains
|
||||
|
||||
### Admin app
|
||||
Browser-based interface for content administrators.
|
||||
|
||||
Responsibilities:
|
||||
- Upload source documents (PDF, MD, TXT)
|
||||
- Review and approve AI-generated Theme batches
|
||||
- Edit and finetune AI-generated curriculum
|
||||
- Confirm curriculum regeneration after KB updates
|
||||
- Monitor ingestion and generation job status
|
||||
|
||||
### Employee app
|
||||
Mobile-first PWA accessible on all devices.
|
||||
|
||||
Responsibilities:
|
||||
- Weekly session delivery (Theme + Topics + micro learning type selection)
|
||||
- Knowledge library (browse all published Topics)
|
||||
- Gamification profile (heatmap, badges, streak, leaderboard)
|
||||
- R42 chatbot (available on every screen)
|
||||
|
||||
### Backend services
|
||||
Six discrete services, each with a single responsibility.
|
||||
|
||||
| Service | Responsibility |
|
||||
|---|---|
|
||||
| Ingestion service | Document upload → chunk → extract KB structure |
|
||||
| Generation service | Topics → 10 micro learning types (structured JSON) |
|
||||
| Curriculum service | KB graph → 26-week schedule, versioning, regeneration |
|
||||
| Embedding service | Chunks + topic summaries → Qdrant |
|
||||
| Chat service (R42) | Query → vector retrieval → grounded response |
|
||||
| Progress service | Completions → XP → badges → streak |
|
||||
|
||||
---
|
||||
|
||||
## Deployment topology
|
||||
|
||||
```
|
||||
repo/
|
||||
├── .github/workflows/ ← pipeline (frozen)
|
||||
├── docker-compose.yml ← infrastructure (frozen)
|
||||
├── Dockerfile ← updated once to point at /app
|
||||
├── ansible/ ← provisioning (frozen)
|
||||
├── legacy/ ← original prototype (read-only reference)
|
||||
└── app/
|
||||
├── frontend/ ← Next.js PWA (admin + employee)
|
||||
└── services/
|
||||
├── ingestion/
|
||||
├── generation/
|
||||
├── curriculum/
|
||||
├── embedding/
|
||||
├── chat/
|
||||
└── progress/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Layer | Technology | Rationale |
|
||||
|---|---|---|
|
||||
| Frontend | Next.js 14, TypeScript, Tailwind CSS | PWA support, single codebase for admin + employee |
|
||||
| Backend state | PocketBase | Auth, file storage, admin UI, SQLite — no infra overhead |
|
||||
| Vector store | Qdrant (Docker) | RAG retrieval, runs as single container |
|
||||
| AI generation | Claude Sonnet 4 via Anthropic API | Structured JSON output, long-form drafting, graph reasoning |
|
||||
| AI chat (R42) | Claude Haiku 4.5 via Anthropic API | Low latency, cost-effective, grounded by retrieval layer |
|
||||
| Embeddings | OpenAI text-embedding-3-small | Cost-effective, high quality at this scale |
|
||||
| Auth | PocketBase built-in | Role-based: admin / employee |
|
||||
|
||||
---
|
||||
|
||||
## AI model responsibilities
|
||||
|
||||
| Task | Model |
|
||||
|---|---|
|
||||
| Document → KB structure extraction | Claude Sonnet 4 |
|
||||
| Topic body drafting | Claude Sonnet 4 |
|
||||
| Micro learning generation (all 10 types) | Claude Sonnet 4 |
|
||||
| Curriculum generation + versioning | Claude Sonnet 4 |
|
||||
| R42 chat responses | Claude Haiku 4.5 |
|
||||
| Embeddings | text-embedding-3-small |
|
||||
|
||||
---
|
||||
|
||||
## Document ingestion pipeline
|
||||
|
||||
```
|
||||
Admin uploads file (PDF / MD / TXT)
|
||||
↓
|
||||
Format detection → text extraction
|
||||
MD: split on headings → preserve hierarchy
|
||||
PDF: pdfplumber → page + paragraph detection
|
||||
TXT: sliding window chunking with overlap
|
||||
↓
|
||||
Chunk cleaning (strip headers/footers/artefacts)
|
||||
↓
|
||||
Claude Sonnet 4 reads chunks → extracts:
|
||||
- candidate Themes
|
||||
- candidate Topics per Theme
|
||||
- Topic→Topic relationships (related, prerequisite, contrast)
|
||||
- key terms for glossary
|
||||
↓
|
||||
Draft KB written to PocketBase (status: draft)
|
||||
↓
|
||||
Embedding service: embed source chunks → write to Qdrant
|
||||
↓
|
||||
Admin reviews Theme batch → approves / edits / rejects
|
||||
↓
|
||||
On approval: Topics published, micro learning generation queued
|
||||
↓
|
||||
Curriculum regeneration notification queued for admin
|
||||
```
|
||||
|
||||
Note: embeddings are generated from **source chunks**, not only from AI-generated topic summaries. R42 retrieves from grounded source material.
|
||||
|
||||
MD source files are the preferred format for admins — heading structure maps directly to Theme → Topic hierarchy and improves extraction quality.
|
||||
|
||||
---
|
||||
|
||||
## Curriculum lifecycle
|
||||
|
||||
### Generation
|
||||
Input: all published Themes, Topics, relationship graph, complexity weights
|
||||
Process: cluster by Theme → sequence pedagogically (prerequisites first, complexity gradient) → distribute across 26 weeks → ensure full KB coverage
|
||||
Output: versioned 26-week draft schedule
|
||||
|
||||
### Perpetual cycling
|
||||
The curriculum runs continuously. After week 26, the employee begins cycle 2 on the latest curriculum version.
|
||||
|
||||
Second and subsequent cycles are not identical to cycle 1:
|
||||
- Theme sequence is varied
|
||||
- Recommended micro learning types surface types the employee has not yet used
|
||||
- Topics with low engagement in prior cycles receive increased coverage
|
||||
|
||||
### Versioning rules
|
||||
|
||||
| Event | Action |
|
||||
|---|---|
|
||||
| New source doc published to KB | Regenerate curriculum from week N+1 for all active employees |
|
||||
| Topic body edited | Micro learnings regenerated; curriculum unaffected |
|
||||
| Theme batch approved | Regeneration queued; admin confirms before it applies |
|
||||
|
||||
Completed weeks are immutable. Regeneration only affects future unstarted weeks.
|
||||
|
||||
### Admin regeneration flow
|
||||
Admin receives notification: "N new topics added. Regenerate curriculum? This will update unstarted weeks for all active employees."
|
||||
Admin can preview the proposed new schedule before confirming.
|
||||
|
||||
---
|
||||
|
||||
## Weekly session flow (employee)
|
||||
|
||||
```
|
||||
Week N opens
|
||||
↓
|
||||
Employee sees assigned Theme + Topics for the week
|
||||
↓
|
||||
Per Topic: employee selects micro learning type
|
||||
(all published types for that topic are available)
|
||||
↓
|
||||
Employee completes one or more types per topic
|
||||
↓
|
||||
Completion recorded → XP awarded → badges evaluated
|
||||
↓
|
||||
Progress visible on public leaderboard and activity feed
|
||||
```
|
||||
|
||||
Sessions support multiple micro learning types per topic in a single session.
|
||||
|
||||
---
|
||||
|
||||
## Micro learning types
|
||||
|
||||
All 10 types are generated by Claude Sonnet 4 as structured JSON, stored in PocketBase, and rendered by the frontend. One or more types may be published per topic.
|
||||
|
||||
| # | Type | Format |
|
||||
|---|---|---|
|
||||
| 1 | Concept explainer | 2–3 paragraphs + example |
|
||||
| 2 | Scenario quiz | situation + 3–4 MCQ options + explained answers |
|
||||
| 3 | Common misconceptions | 3–5 false beliefs + corrections |
|
||||
| 4 | Step-by-step how-to | numbered procedure |
|
||||
| 5 | Comparison card | side-by-side on 4–6 dimensions |
|
||||
| 6 | Reflection prompt | open question + model answer |
|
||||
| 7 | Spaced repetition flashcards | 5–10 Q&A pairs |
|
||||
| 8 | Case study mini-analysis | 150–200 word scenario + guiding questions |
|
||||
| 9 | Glossary anchor | term + definition + correct use + misuse |
|
||||
| 10 | Myth vs. evidence | false claim + evidence-based rebuttal |
|
||||
|
||||
---
|
||||
|
||||
## R42 — chat service design
|
||||
|
||||
R42 is a functional KB-grounded assistant available on every screen in the employee app.
|
||||
|
||||
Behaviour:
|
||||
- Stateless per session (no memory between conversations)
|
||||
- Retrieves relevant chunks from Qdrant using the employee's query
|
||||
- Knows the employee's current curriculum week → retrieval is context-weighted
|
||||
- Cites source topic in every response ("based on the **Holacratic roles** topic")
|
||||
- Explicitly refuses to answer outside KB scope rather than hallucinating
|
||||
- Scope: internal KB only
|
||||
|
||||
Implementation:
|
||||
- Employee query → embed → Qdrant nearest-neighbour retrieval → top-K chunks
|
||||
- Chunks + employee context injected into Haiku 4.5 prompt
|
||||
- Response streamed to frontend
|
||||
|
||||
UI: floating button bottom-right, unobtrusive on mobile.
|
||||
|
||||
---
|
||||
|
||||
## Gamification system
|
||||
|
||||
Inspired by the visual language of GitHub, Stack Overflow, and Duolingo. Mechanics use developer-native terminology.
|
||||
|
||||
### XP unit: commits
|
||||
Every completed topic earns commits. Quantity varies by micro learning type complexity.
|
||||
|
||||
### Levels
|
||||
`Intern → Junior → Medior → Senior → Staff → Principal`
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
### Streak
|
||||
Counted in consecutive weeks, not days. Resets if a week is skipped entirely.
|
||||
|
||||
### Activity heatmap
|
||||
GitHub-style contribution graph spanning the full 26-week cycle. Cell darkness = number of types completed that week.
|
||||
|
||||
### Badges
|
||||
|
||||
| Tier | Condition |
|
||||
|---|---|
|
||||
| Bronze | Complete any session |
|
||||
| Silver | 5 sessions completed, 5 different types used |
|
||||
| Gold | 13 sessions without skipping a week |
|
||||
| Legendary | All 26 sessions, all 10 types used at least once |
|
||||
|
||||
Named content badges (examples):
|
||||
- `governance nerd` — all holacratic structure topics completed
|
||||
- `process architect` — all internal process topics completed
|
||||
- `deep reader` — case study type used 5+ times
|
||||
|
||||
### Milestone cards (public)
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed:
|
||||
|
||||
```
|
||||
🚀 [Name] shipped the full curriculum
|
||||
26 weeks · [N] commits · [badges]
|
||||
Longest streak: [N] weeks
|
||||
```
|
||||
|
||||
Language: shipping vocabulary, not school vocabulary.
|
||||
|
||||
### Leaderboard
|
||||
Not ranked 1–N by score. Displays multiple dimensions:
|
||||
|
||||
| Employee | Commits | Streak | Types used | Badges |
|
||||
|---|---|---|---|---|
|
||||
|
||||
Multiple paths to visibility. No single metric determines standing.
|
||||
|
||||
---
|
||||
|
||||
## Security and privacy
|
||||
|
||||
- Auth: PocketBase role-based (admin / employee)
|
||||
- Gamification data (commits, badges, streak) is public to all employees
|
||||
- Session completion data (which topic, which type, when) is public
|
||||
- Source documents are admin-only
|
||||
- No PII beyond display name stored in gamification context
|
||||
- R42 is stateless — no chat history persisted
|
||||
397
docs/data-model.md
Normal file
397
docs/data-model.md
Normal file
@@ -0,0 +1,397 @@
|
||||
# Data model: employee learning platform
|
||||
|
||||
## Overview
|
||||
|
||||
Two storage systems:
|
||||
- **PocketBase** — all structured relational data (SQLite under the hood)
|
||||
- **Qdrant** — all vector embeddings for RAG retrieval
|
||||
|
||||
---
|
||||
|
||||
## PocketBase collections
|
||||
|
||||
### `source_documents`
|
||||
Uploaded source files. Parent of all generated KB content.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| filename | string | original filename |
|
||||
| file | file | PocketBase file storage |
|
||||
| format | select | `pdf` `md` `txt` |
|
||||
| status | select | `processing` `processed` `failed` |
|
||||
| ingested_at | datetime | |
|
||||
| chunk_count | number | total chunks extracted |
|
||||
| created_by | relation → `users` | admin who uploaded |
|
||||
|
||||
---
|
||||
|
||||
### `themes`
|
||||
Top-level content groupings. One Theme = one weekly session.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| title | string | |
|
||||
| description | text | AI drafted, admin editable |
|
||||
| status | select | `draft` `published` |
|
||||
| source_documents | relation[] → `source_documents` | which docs contributed |
|
||||
| approved_by | relation → `users` | admin who approved batch |
|
||||
| approved_at | datetime | |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `topics`
|
||||
Atomic knowledge units. Always belong to a Theme.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| theme | relation → `themes` | parent theme |
|
||||
| title | string | |
|
||||
| body | text (rich) | AI drafted, admin editable |
|
||||
| difficulty | select | `introductory` `intermediate` `advanced` |
|
||||
| complexity_weight | number | 1–5, used by curriculum generator |
|
||||
| status | select | `draft` `published` |
|
||||
| related_topics | relation[] → `topics` | lateral relationships |
|
||||
| prerequisite_topics | relation[] → `topics` | must-complete-first |
|
||||
| contrast_topics | relation[] → `topics` | deliberate opposites |
|
||||
| key_terms | json | string[] — feeds glossary |
|
||||
| qdrant_chunk_ids | json | string[] — references to embedded chunks |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale.
|
||||
|
||||
---
|
||||
|
||||
### `micro_learnings`
|
||||
Generated content artifacts. One record per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| topic | relation → `topics` | |
|
||||
| type | select | see type enum below |
|
||||
| content | json | structured output — schema varies per type |
|
||||
| status | select | `queued` `generated` `published` `rejected` |
|
||||
| generation_model | string | model version used |
|
||||
| generated_at | datetime | |
|
||||
| published_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
**Type enum:**
|
||||
`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence`
|
||||
|
||||
**Content JSON schemas per type:**
|
||||
|
||||
```json
|
||||
// concept_explainer
|
||||
{
|
||||
"paragraphs": ["string", "string"],
|
||||
"example": "string"
|
||||
}
|
||||
|
||||
// scenario_quiz
|
||||
{
|
||||
"scenario": "string",
|
||||
"options": [
|
||||
{ "label": "A", "text": "string", "correct": true, "explanation": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// misconceptions
|
||||
{
|
||||
"items": [
|
||||
{ "misconception": "string", "correction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// how_to
|
||||
{
|
||||
"steps": [
|
||||
{ "number": 1, "instruction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// comparison_card
|
||||
{
|
||||
"subject_a": "string",
|
||||
"subject_b": "string",
|
||||
"dimensions": [
|
||||
{ "label": "string", "a": "string", "b": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// reflection_prompt
|
||||
{
|
||||
"prompt": "string",
|
||||
"model_answer": "string"
|
||||
}
|
||||
|
||||
// flashcard_set
|
||||
{
|
||||
"cards": [
|
||||
{ "question": "string", "answer": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// case_study
|
||||
{
|
||||
"scenario": "string",
|
||||
"questions": ["string"]
|
||||
}
|
||||
|
||||
// glossary_anchor
|
||||
{
|
||||
"term": "string",
|
||||
"definition": "string",
|
||||
"correct_use": "string",
|
||||
"misuse": "string"
|
||||
}
|
||||
|
||||
// myth_vs_evidence
|
||||
{
|
||||
"myth": "string",
|
||||
"evidence": "string",
|
||||
"sources": ["string"]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_versions`
|
||||
Versioned 26-week schedule. New version created on each regeneration.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| version | number | increments on each regeneration |
|
||||
| status | select | `draft` `active` `superseded` |
|
||||
| generated_at | datetime | |
|
||||
| approved_by | relation → `users` | admin who confirmed regeneration |
|
||||
| approved_at | datetime | |
|
||||
| generation_notes | text | why this version was created |
|
||||
|
||||
Only one version has status `active` at any time.
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_weeks`
|
||||
Individual week slots. Child of a curriculum version.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| curriculum_version | relation → `curriculum_versions` | |
|
||||
| week_number | number | 1–26 |
|
||||
| theme | relation → `themes` | |
|
||||
| topics | relation[] → `topics` | ordered subset of theme topics |
|
||||
| topic_order | json | number[] — explicit ordering |
|
||||
| estimated_duration_minutes | number | AI estimate |
|
||||
| admin_notes | text | freeform admin annotation |
|
||||
|
||||
---
|
||||
|
||||
### `employee_curriculum_state`
|
||||
Tracks each employee's position in the curriculum. One record per employee.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| current_cycle | number | starts at 1, increments on loop |
|
||||
| current_week | number | 1–26 |
|
||||
| start_date | datetime | rolling start |
|
||||
| active_version | relation → `curriculum_versions` | version employee is on |
|
||||
| updated_at | datetime | |
|
||||
|
||||
When curriculum regenerates: `active_version` updates for all employees whose `current_week` is less than the first regenerated week.
|
||||
|
||||
---
|
||||
|
||||
### `session_completions`
|
||||
Immutable completion records. One record per employee per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| topic | relation → `topics` | |
|
||||
| micro_learning | relation → `micro_learnings` | specific type completed |
|
||||
| week_number | number | curriculum week at time of completion |
|
||||
| cycle | number | which cycle |
|
||||
| completed_at | datetime | |
|
||||
|
||||
Records are never updated or deleted. This is the canonical history.
|
||||
|
||||
---
|
||||
|
||||
### `gamification_profiles`
|
||||
One record per employee. Updated by progress service on each completion.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| total_commits | number | cumulative XP |
|
||||
| current_level | select | `intern` `junior` `medior` `senior` `staff` `principal` |
|
||||
| current_streak_weeks | number | consecutive weeks with ≥1 completion |
|
||||
| longest_streak_weeks | number | all-time high |
|
||||
| types_used | json | string[] — which of 10 types used at least once |
|
||||
| last_active_week | number | used to detect streak breaks |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `badges`
|
||||
Badge definitions. Seeded at startup, not user-generated.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| key | string | unique slug e.g. `governance_nerd` |
|
||||
| tier | select | `bronze` `silver` `gold` `legendary` `content` |
|
||||
| label | string | display name |
|
||||
| description | string | award condition description |
|
||||
| icon | string | emoji or icon key |
|
||||
|
||||
---
|
||||
|
||||
### `employee_badges`
|
||||
Junction: which employees have earned which badges.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| badge | relation → `badges` | |
|
||||
| earned_at | datetime | |
|
||||
| cycle | number | which cycle it was earned in |
|
||||
|
||||
---
|
||||
|
||||
### `milestone_cards`
|
||||
Public milestone events at weeks 13 and 26.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| cycle | number | |
|
||||
| week | number | 13 or 26 |
|
||||
| total_commits | number | snapshot at time of milestone |
|
||||
| streak_weeks | number | snapshot |
|
||||
| badge_keys | json | string[] — badges held at milestone |
|
||||
| created_at | datetime | public feed ordered by this |
|
||||
|
||||
---
|
||||
|
||||
## PocketBase users collection (extended)
|
||||
|
||||
Standard PocketBase `users` collection with additional fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| role | select | `admin` `employee` |
|
||||
| display_name | string | used in gamification feed |
|
||||
| avatar | file | optional |
|
||||
|
||||
---
|
||||
|
||||
## Qdrant collections
|
||||
|
||||
### `source_chunks`
|
||||
Embeddings of raw source document chunks. Primary retrieval target for R42.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions (text-embedding-3-small) |
|
||||
| source_document_id | string | reference to PocketBase |
|
||||
| chunk_index | number | position within document |
|
||||
| text | string | raw chunk text |
|
||||
| theme_id | string | assigned theme (post-extraction) |
|
||||
| topic_id | string | assigned topic (post-extraction, nullable) |
|
||||
| format | string | pdf / md / txt |
|
||||
|
||||
### `topic_summaries`
|
||||
Embeddings of AI-generated topic body text. Secondary retrieval target.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions |
|
||||
| topic_id | string | reference to PocketBase |
|
||||
| theme_id | string | |
|
||||
| title | string | for display in R42 citation |
|
||||
| text | string | full topic body |
|
||||
|
||||
---
|
||||
|
||||
## Retrieval strategy for R42
|
||||
|
||||
R42 queries both Qdrant collections and merges results:
|
||||
|
||||
```
|
||||
Employee query
|
||||
↓
|
||||
Embed query → text-embedding-3-small
|
||||
↓
|
||||
Qdrant search: source_chunks (top 5) + topic_summaries (top 3)
|
||||
↓
|
||||
Filter: boost chunks from employee's current week theme
|
||||
↓
|
||||
Merge + deduplicate by topic_id
|
||||
↓
|
||||
Top-K context injected into Haiku 4.5 prompt
|
||||
↓
|
||||
Response includes: answer + cited topic title(s)
|
||||
```
|
||||
|
||||
Source chunks are weighted higher than topic summaries to keep R42 grounded in original source material rather than AI-generated abstractions.
|
||||
|
||||
---
|
||||
|
||||
## Indexes and query patterns
|
||||
|
||||
Critical query patterns the data model must support efficiently:
|
||||
|
||||
| Query | Collection | Index |
|
||||
|---|---|---|
|
||||
| All published topics for a theme | topics | theme + status |
|
||||
| All micro learnings for a topic | micro_learnings | topic + status |
|
||||
| Employee's current week | employee_curriculum_state | user |
|
||||
| Weeks for a curriculum version | curriculum_weeks | curriculum_version + week_number |
|
||||
| Employee completion history | session_completions | user + cycle |
|
||||
| Public leaderboard | gamification_profiles | total_commits + streak |
|
||||
| Milestone feed | milestone_cards | created_at DESC |
|
||||
| Badges earned by employee | employee_badges | user |
|
||||
|
||||
PocketBase creates indexes on relation fields by default. Composite indexes on `status` fields should be added manually where query frequency warrants it.
|
||||
|
||||
---
|
||||
|
||||
## Data flow summary
|
||||
|
||||
```
|
||||
source_documents
|
||||
└── (ingestion service)
|
||||
└── qdrant: source_chunks
|
||||
└── themes (draft)
|
||||
└── topics (draft)
|
||||
└── (approval)
|
||||
└── topics (published)
|
||||
└── qdrant: topic_summaries
|
||||
└── micro_learnings (queued → published)
|
||||
└── (curriculum service)
|
||||
└── curriculum_versions
|
||||
└── curriculum_weeks
|
||||
└── (employee progress)
|
||||
└── session_completions
|
||||
└── gamification_profiles
|
||||
└── employee_badges
|
||||
└── milestone_cards
|
||||
```
|
||||
231
docs/handover.md
Normal file
231
docs/handover.md
Normal file
@@ -0,0 +1,231 @@
|
||||
# Handover: employee learning platform
|
||||
|
||||
## Purpose of this document
|
||||
|
||||
This document captures every design decision made before implementation started.
|
||||
It is the authoritative source for rationale. When a spec file is ambiguous,
|
||||
resolve it against this document. Do not ask the human — the answers are here.
|
||||
|
||||
---
|
||||
|
||||
## What this application does
|
||||
|
||||
Employees of a tech company use this platform to build and maintain knowledge of
|
||||
the employee handbook, holacratic structures, and internal processes.
|
||||
|
||||
Core mechanics:
|
||||
- Admins upload source documents → AI extracts a structured knowledge base
|
||||
- The KB is organised into Themes (broad) and Topics (specific)
|
||||
- An AI generates 10 types of micro learning content per Topic
|
||||
- Employees follow a 26-week curriculum of weekly sessions
|
||||
- Each session covers one Theme (multiple related Topics)
|
||||
- Employees choose which micro learning type to use per Topic
|
||||
- After 26 weeks the curriculum restarts, varied to reinforce rather than repeat
|
||||
- A chatbot called R42 answers KB-grounded questions on every screen
|
||||
- A gamification system using developer-native language motivates completion
|
||||
|
||||
---
|
||||
|
||||
## All confirmed design decisions
|
||||
|
||||
### Knowledge base
|
||||
|
||||
**Decision: KB is extracted from source documents, not manually authored**
|
||||
Admins upload raw source material. Claude Sonnet 4 extracts Themes, Topics, and
|
||||
relationships. Admins review and approve in batches (one Theme at a time, not
|
||||
one Topic at a time). Topic bodies are AI-drafted and admin-editable after approval.
|
||||
|
||||
**Decision: two-level hierarchy — Theme → Topic**
|
||||
A Theme is a broad subject area. A Topic is one specific concept within a Theme.
|
||||
One weekly session = one Theme. Multiple Topics within that Theme per session.
|
||||
|
||||
**Decision: three relationship types between Topics**
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A should be understood before Topic B
|
||||
- contrast: Topics representing opposing approaches
|
||||
|
||||
These relationships are stored as explicit PocketBase relation fields, not a
|
||||
generic junction table.
|
||||
|
||||
**Decision: source material format priority**
|
||||
Accepted formats: PDF, MD, TXT only. MD is the highest quality input —
|
||||
heading structure maps directly to Theme → Topic hierarchy. Admins should be
|
||||
recommended to provide MD where possible.
|
||||
|
||||
**Decision: embeddings from source chunks, not topic summaries only**
|
||||
R42 retrieves from original source material chunks as primary source, with
|
||||
topic summaries as secondary. This keeps R42 grounded and reduces hallucination.
|
||||
|
||||
---
|
||||
|
||||
### Micro learnings
|
||||
|
||||
**Decision: 10 types, all generated by AI as structured JSON**
|
||||
Types:
|
||||
1. concept_explainer
|
||||
2. scenario_quiz
|
||||
3. misconceptions
|
||||
4. how_to
|
||||
5. comparison_card
|
||||
6. reflection_prompt
|
||||
7. flashcard_set
|
||||
8. case_study
|
||||
9. glossary_anchor
|
||||
10. myth_vs_evidence
|
||||
|
||||
Each type has a defined JSON schema in data-model.md. Generation uses
|
||||
Claude Sonnet 4. Output is validated against Zod schemas before storage.
|
||||
|
||||
**Decision: employees choose type per topic per session**
|
||||
Employees are not locked to one type globally. Each session, per Topic, the
|
||||
employee selects from all published types for that topic. Multiple types can
|
||||
be completed in one session.
|
||||
|
||||
**Decision: pre-generate, don't generate on demand**
|
||||
All 10 types are generated when a Topic is approved, not when an employee
|
||||
requests them. This controls quality and cost. R42 is the only on-demand
|
||||
AI interaction.
|
||||
|
||||
---
|
||||
|
||||
### Curriculum
|
||||
|
||||
**Decision: AI generates curriculum, admin edits**
|
||||
Claude Sonnet 4 reads the full KB graph (Themes, Topics, relationships,
|
||||
complexity weights) and produces a 26-week schedule. Admin reviews, reorders,
|
||||
and finetunes. Admin does not build from scratch.
|
||||
|
||||
**Decision: one Theme per week session**
|
||||
A session covers all Topics under one Theme. Topics within the session are
|
||||
ordered by the curriculum generator based on prerequisites and complexity.
|
||||
|
||||
**Decision: perpetual curriculum with versioning**
|
||||
The curriculum runs indefinitely. After week 26, cycle 2 begins on the latest
|
||||
curriculum version. Cycle 2+ varies sequence, surfaces unused micro learning
|
||||
types, and increases coverage of low-engagement topics.
|
||||
|
||||
**Decision: completed weeks are immutable**
|
||||
Regeneration only affects future unstarted weeks. An employee's completion
|
||||
history is never altered regardless of curriculum version changes.
|
||||
|
||||
**Decision: regeneration requires admin confirmation**
|
||||
When new Topics are approved, the system queues a regeneration but does not
|
||||
apply it until the admin explicitly confirms. Admin sees a preview of the
|
||||
proposed new schedule before confirming.
|
||||
|
||||
**Decision: rolling starts**
|
||||
Each employee has their own start date. There are no cohorts or shared
|
||||
start dates.
|
||||
|
||||
---
|
||||
|
||||
### Gamification
|
||||
|
||||
**Decision: developer-native visual language**
|
||||
Inspired by GitHub (heatmap), Stack Overflow (badges, reputation), and
|
||||
Duolingo (streak, XP, levels). Language uses developer vocabulary throughout.
|
||||
|
||||
**Decision: XP unit is called commits**
|
||||
Every completed Topic earns commits. Quantity varies by micro learning type.
|
||||
|
||||
**Decision: levels use developer rank names**
|
||||
Intern → Junior → Medior → Senior → Staff → Principal
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
**Decision: streak is weekly, not daily**
|
||||
Consecutive weeks with at least one completion. Resets on a skipped week.
|
||||
|
||||
**Decision: activity heatmap covers 26-week cycle**
|
||||
GitHub-style contribution graph. Cell darkness = number of types completed
|
||||
that week.
|
||||
|
||||
**Decision: no social layer**
|
||||
No comments, reactions, or direct messaging. Gamification is visible but
|
||||
not interactive between employees.
|
||||
|
||||
**Decision: public milestone cards, not ranked leaderboard**
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed.
|
||||
Language: "shipped", not "graduated". The leaderboard shows multiple
|
||||
dimensions (commits, streak, types used, badges) — not a single ranking.
|
||||
|
||||
**Decision: named content badges**
|
||||
Examples: governance_nerd, process_architect, deep_reader. These are seeded
|
||||
at startup, not user-generated. See data-model.md for badge schema.
|
||||
|
||||
---
|
||||
|
||||
### R42 chatbot
|
||||
|
||||
**Decision: functional only, no personality**
|
||||
R42 answers questions grounded in the KB. It does not have a defined persona,
|
||||
tone, or name story beyond the label R42.
|
||||
|
||||
**Decision: stateless per session**
|
||||
No chat history is persisted between sessions. This avoids privacy complexity
|
||||
and keeps the implementation simple.
|
||||
|
||||
**Decision: internal KB scope only**
|
||||
R42 cannot search external sources. If a question cannot be answered from the
|
||||
KB, R42 says so explicitly.
|
||||
|
||||
**Decision: context-weighted retrieval**
|
||||
R42 knows the employee's current curriculum week. Retrieval boosts chunks
|
||||
from the current week's Theme. General KB questions are not restricted.
|
||||
|
||||
**Decision: always cites source Topic**
|
||||
Every R42 response includes the Topic title(s) its answer draws from.
|
||||
|
||||
**Decision: Haiku 4.5 for R42, Sonnet 4 for generation**
|
||||
Low latency matters for chat. The retrieval layer compensates for Haiku's
|
||||
smaller knowledge base. Sonnet 4 is reserved for generation tasks where
|
||||
structure and quality matter more than speed.
|
||||
|
||||
---
|
||||
|
||||
### Infrastructure
|
||||
|
||||
**Decision: PocketBase as primary backend**
|
||||
Auth, file storage, structured data, and admin UI in one binary. SQLite is
|
||||
sufficient for ~150 users. No PostgreSQL needed at this scale.
|
||||
|
||||
**Decision: Qdrant for vector storage**
|
||||
Separate Docker container. Keeps vector operations out of SQLite.
|
||||
pgvector was rejected — adding Postgres just for vectors is unnecessary overhead.
|
||||
|
||||
**Decision: Next.js 14 PWA for frontend**
|
||||
Single codebase for admin and employee app. PWA covers mobile without a native
|
||||
app. Learning platforms do not require native device APIs.
|
||||
|
||||
**Decision: five discrete backend services**
|
||||
Ingestion, generation, curriculum, chat, progress. Each is a separate Fastify
|
||||
service with its own port and responsibility. They do not call each other
|
||||
directly — they read/write shared PocketBase collections.
|
||||
|
||||
**Decision: PDF parsing starts with pdf-parse (Node.js)**
|
||||
Switch to pdfplumber Python sidecar only if pdf-parse quality is insufficient
|
||||
for actual source documents. Do not over-engineer the extraction layer upfront.
|
||||
|
||||
---
|
||||
|
||||
## What is not yet specced
|
||||
|
||||
The following spec files still need to be written before their phases begin:
|
||||
- /docs/generation-spec.md — micro learning generation service
|
||||
- /docs/curriculum-spec.md — curriculum generator + versioning
|
||||
- /docs/r42-spec.md — chat service
|
||||
- /docs/gamification-spec.md — progress service + gamification mechanics
|
||||
- /docs/frontend-spec.md — employee app, admin app, PWA config
|
||||
|
||||
Do not begin a phase without its spec file. Flag the gap if you reach it.
|
||||
|
||||
---
|
||||
|
||||
## Source of truth hierarchy
|
||||
|
||||
When sources conflict, resolve in this order:
|
||||
1. This handover document (rationale and decisions)
|
||||
2. The relevant spec file (implementation detail)
|
||||
3. data-model.md (schema is authoritative)
|
||||
4. architecture.md (system structure)
|
||||
|
||||
Do not use legacy/ code as a source of truth for anything.
|
||||
394
docs/implementation-plan.md
Normal file
394
docs/implementation-plan.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# Implementation plan
|
||||
|
||||
## How to use this document
|
||||
|
||||
Work through phases in order. Do not start phase N+1 before phase N passes
|
||||
all acceptance criteria. Each phase lists the spec file to read, the steps
|
||||
to execute, and the criteria that define done.
|
||||
|
||||
At the start of each session: state the phase and step.
|
||||
At the end of each session: state completed steps and next starting point.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Infrastructure + ingestion service
|
||||
|
||||
**Spec to read:** /docs/ingestion-spec.md, /docs/data-model.md
|
||||
|
||||
### Steps
|
||||
|
||||
**1.1 — Repo scaffold**
|
||||
```
|
||||
app/
|
||||
frontend/ (empty, Next.js init comes in phase 4)
|
||||
services/
|
||||
ingestion/
|
||||
generation/ (empty placeholder)
|
||||
curriculum/ (empty placeholder)
|
||||
chat/ (empty placeholder)
|
||||
progress/ (empty placeholder)
|
||||
```
|
||||
|
||||
Create `app/services/ingestion/` with:
|
||||
- package.json (dependencies from ingestion-spec.md)
|
||||
- tsconfig.json (strict mode)
|
||||
- .env.example (all env vars from ingestion-spec.md)
|
||||
- .gitignore
|
||||
|
||||
**1.2 — PocketBase collections**
|
||||
PocketBase runs as a binary. Create a migration script at
|
||||
`app/services/ingestion/migrations/001_initial_schema.ts` that uses the
|
||||
PocketBase JS SDK to create all collections defined in data-model.md:
|
||||
|
||||
Collections to create:
|
||||
- source_documents
|
||||
- themes
|
||||
- topics
|
||||
- micro_learnings (schema only — no data yet)
|
||||
- curriculum_versions (schema only)
|
||||
- curriculum_weeks (schema only)
|
||||
- employee_curriculum_state (schema only)
|
||||
- session_completions (schema only)
|
||||
- gamification_profiles (schema only)
|
||||
- badges (schema only)
|
||||
- employee_badges (schema only)
|
||||
- milestone_cards (schema only)
|
||||
|
||||
Seed the badges collection with all badge definitions from data-model.md.
|
||||
|
||||
**1.3 — Qdrant collections**
|
||||
Create `app/services/ingestion/migrations/002_qdrant_setup.ts` that
|
||||
initialises both Qdrant collections:
|
||||
- source_chunks (1536 dimensions, cosine distance)
|
||||
- topic_summaries (1536 dimensions, cosine distance)
|
||||
|
||||
**1.4 — Ingestion service scaffold**
|
||||
Build the Fastify server with two routes:
|
||||
- POST /ingest
|
||||
- GET /status/:jobId
|
||||
|
||||
Use the file structure from ingestion-spec.md exactly.
|
||||
|
||||
**1.5 — Stage 1: text extraction**
|
||||
Implement extract.ts per ingestion-spec.md:
|
||||
- TXT: direct UTF-8 read
|
||||
- MD: direct UTF-8 read, preserve heading markers
|
||||
- PDF: pdf-parse, page break markers
|
||||
|
||||
**1.6 — Stage 2–3: chunking + cleaning**
|
||||
Implement chunk.ts and clean.ts per ingestion-spec.md:
|
||||
- MD: heading-based splitting
|
||||
- TXT: sliding window (800 chars, 150 overlap)
|
||||
- PDF: page + paragraph splitting
|
||||
- Cleaning: whitespace, artefacts, minimum length filter
|
||||
|
||||
**1.7 — Stage 4: structure extraction**
|
||||
Implement structure.ts per ingestion-spec.md:
|
||||
- Claude Sonnet 4 call with system + user prompt from spec
|
||||
- Zod validation of DraftKB output
|
||||
- Batch handling for documents > 60 chunks
|
||||
- Retry logic on parse failure
|
||||
- Error handling: failed job status + reason
|
||||
|
||||
**1.8 — Stage 5: PocketBase write**
|
||||
Implement the PocketBase write logic:
|
||||
- Create Theme records (status: draft)
|
||||
- Create Topic records under each Theme (status: draft)
|
||||
- Resolve relationships between Topics after all records created
|
||||
|
||||
**1.9 — Stage 6: embeddings + Qdrant write**
|
||||
Implement embed.ts:
|
||||
- OpenAI text-embedding-3-small, batches of 100
|
||||
- Write to Qdrant source_chunks collection
|
||||
- Write to Qdrant topic_summaries collection
|
||||
- Update Topic.qdrant_chunk_ids in PocketBase
|
||||
|
||||
**1.10 — Job status tracking**
|
||||
Wire all stages into the job queue (jobs/queue.ts):
|
||||
- Status transitions: queued → extracting → chunking → structuring →
|
||||
writing → embedding → done / failed
|
||||
- Progress counters (chunksTotal, chunksEmbedded, themesFound, topicsFound)
|
||||
- GET /status/:jobId returns current state
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] POST /ingest with a small MD file completes without error
|
||||
- [ ] GET /status/:jobId returns `done` after processing
|
||||
- [ ] PocketBase contains draft Theme + Topic records with correct hierarchy
|
||||
- [ ] Topic.body contains AI-drafted content (not empty)
|
||||
- [ ] Topic relationships are resolved (related_topics populated where applicable)
|
||||
- [ ] Qdrant source_chunks contains vectors with correct payload fields
|
||||
- [ ] Qdrant topic_summaries contains vectors for each published topic
|
||||
- [ ] Topic.qdrant_chunk_ids is populated
|
||||
- [ ] POST /ingest with a PDF file completes without error
|
||||
- [ ] POST /ingest with a TXT file completes without error
|
||||
- [ ] A document > 60 chunks triggers batch processing without error
|
||||
- [ ] A malformed PDF returns status `failed` with reason, not an uncaught exception
|
||||
- [ ] All Zod validations pass — no `any` types in codebase
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Generation service
|
||||
|
||||
**Spec to read:** /docs/generation-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**2.1 — Generation service scaffold**
|
||||
Fastify service at app/services/generation/
|
||||
Routes: POST /generate, GET /status/:jobId
|
||||
|
||||
**2.2 — Generate all 10 types per topic**
|
||||
One Claude Sonnet 4 call per type per topic.
|
||||
Structured JSON output validated against Zod schemas from data-model.md.
|
||||
Write to micro_learnings collection (status: generated).
|
||||
|
||||
**2.3 — Batch generation on theme approval**
|
||||
When admin approves a Theme batch, queue generation for all Topics in that Theme.
|
||||
All 10 types per Topic.
|
||||
|
||||
**2.4 — Admin publish flow**
|
||||
Route to update micro_learning status from generated → published or rejected.
|
||||
This is called by the admin app (built in phase 4).
|
||||
|
||||
### Acceptance criteria (to be detailed in generation-spec.md)
|
||||
|
||||
- [ ] All 10 micro learning types generated for a test topic
|
||||
- [ ] All 10 JSON outputs validate against their Zod schemas
|
||||
- [ ] Generated content written to PocketBase with status: generated
|
||||
- [ ] Admin can publish or reject individual micro learnings
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Curriculum service
|
||||
|
||||
**Spec to read:** /docs/curriculum-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**3.1 — Curriculum service scaffold**
|
||||
Fastify service at app/services/curriculum/
|
||||
|
||||
**3.2 — Curriculum generator**
|
||||
Claude Sonnet 4 reads full KB graph → produces 26-week schedule.
|
||||
Written to curriculum_versions + curriculum_weeks.
|
||||
|
||||
**3.3 — Versioning logic**
|
||||
- New version created on regeneration
|
||||
- Completed weeks frozen (employee_curriculum_state.current_week used as boundary)
|
||||
- Admin confirmation required before applying new version
|
||||
|
||||
**3.4 — Perpetual cycling**
|
||||
On week 26 completion, cycle increments, new cycle starts on latest version.
|
||||
Second cycle: varied sequence, surfaces unused micro learning types.
|
||||
|
||||
### Acceptance criteria (to be detailed in curriculum-spec.md)
|
||||
|
||||
- [ ] Curriculum generated from a populated KB
|
||||
- [ ] 26 weeks produced, all Themes covered
|
||||
- [ ] Prerequisites respected in ordering
|
||||
- [ ] Regeneration does not alter completed weeks
|
||||
- [ ] Admin confirmation flow works correctly
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Frontend: admin app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**4.1 — Next.js 14 scaffold**
|
||||
Mobile-first, TypeScript strict, Tailwind CSS, PWA config.
|
||||
Role-based routing: /admin/* and /app/* from single Next.js codebase.
|
||||
|
||||
**4.2 — Auth**
|
||||
PocketBase auth integration. Admin role routes to /admin/*.
|
||||
|
||||
**4.3 — Document upload + ingestion status**
|
||||
Upload UI → calls ingestion service → polls job status → shows progress.
|
||||
|
||||
**4.4 — Theme batch review**
|
||||
Display draft Themes with their Topic list.
|
||||
Approve batch / edit individual topics / reject batch.
|
||||
Triggers generation service on approval.
|
||||
|
||||
**4.5 — Curriculum editor**
|
||||
Display AI-generated curriculum (26 weeks).
|
||||
Drag-to-reorder weeks. Edit Theme assignment per week.
|
||||
Confirm regeneration with preview.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Admin can upload a document and see ingestion progress
|
||||
- [ ] Admin can approve a Theme batch
|
||||
- [ ] Admin can edit a Topic before approval
|
||||
- [ ] Admin can view and reorder the curriculum
|
||||
- [ ] Admin can confirm a curriculum regeneration with preview
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Frontend: employee app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (same file, employee section)
|
||||
|
||||
### Steps
|
||||
|
||||
**5.1 — Employee auth + onboarding**
|
||||
PocketBase auth. Employee role routes to /app/*.
|
||||
Set start date on first login → creates employee_curriculum_state record.
|
||||
|
||||
**5.2 — Weekly session flow**
|
||||
Current week's Theme displayed.
|
||||
Topics listed with available micro learning types per topic.
|
||||
Employee selects type → content rendered → mark complete.
|
||||
|
||||
**5.3 — Knowledge library**
|
||||
Browse all published Topics.
|
||||
Filter by Theme, difficulty, key terms.
|
||||
|
||||
**5.4 — R42 chatbot**
|
||||
Floating button, every screen.
|
||||
Calls chat service → streams response.
|
||||
Cites source topic in response.
|
||||
|
||||
**5.5 — Gamification profile**
|
||||
GitHub-style heatmap (26-week view).
|
||||
Badge display.
|
||||
Streak + level + commit count.
|
||||
Public leaderboard (multi-dimension).
|
||||
Milestone cards in activity feed.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Employee sees correct week based on start date
|
||||
- [ ] Employee can complete a topic with a chosen micro learning type
|
||||
- [ ] Completion is recorded and XP awarded
|
||||
- [ ] Knowledge library shows all published topics with filters
|
||||
- [ ] R42 responds with grounded answer and source citation
|
||||
- [ ] R42 is accessible from every screen
|
||||
- [ ] Heatmap renders correctly on mobile (375px)
|
||||
- [ ] Leaderboard shows all employees with multi-dimension data
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Chat service (R42)
|
||||
|
||||
**Spec to read:** /docs/r42-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**6.1 — Chat service scaffold**
|
||||
Fastify service at app/services/chat/
|
||||
|
||||
**6.2 — Query → embed → retrieve**
|
||||
Employee query embedded → Qdrant nearest-neighbour on both collections.
|
||||
Boost chunks from employee's current Theme.
|
||||
|
||||
**6.3 — Response generation**
|
||||
Top-K chunks injected into Haiku 4.5 prompt.
|
||||
Response streamed to frontend.
|
||||
Source Topic titles included in response.
|
||||
|
||||
**6.4 — Out-of-scope handling**
|
||||
If retrieval confidence is below threshold, R42 responds:
|
||||
"I can only answer questions based on the internal knowledge base.
|
||||
This topic doesn't appear to be covered."
|
||||
|
||||
### Acceptance criteria (to be detailed in r42-spec.md)
|
||||
|
||||
- [ ] R42 answers a question about a published topic correctly
|
||||
- [ ] R42 cites the source topic in its response
|
||||
- [ ] R42 refuses to answer out-of-scope questions explicitly
|
||||
- [ ] Response streams to frontend (not batch)
|
||||
- [ ] Response latency < 3 seconds for typical queries
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Progress service
|
||||
|
||||
**Spec to read:** /docs/gamification-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**7.1 — Progress service scaffold**
|
||||
Fastify service at app/services/progress/
|
||||
|
||||
**7.2 — Completion recording**
|
||||
Write session_completions record on topic completion.
|
||||
Calculate XP (commits) per type.
|
||||
|
||||
**7.3 — Gamification updates**
|
||||
Update gamification_profiles: commits, level, streak, types_used.
|
||||
Evaluate badge conditions → write employee_badges on award.
|
||||
|
||||
**7.4 — Milestone cards**
|
||||
Generate milestone_cards record at weeks 13 and 26.
|
||||
|
||||
**7.5 — Leaderboard query**
|
||||
Endpoint returning all gamification_profiles for leaderboard rendering.
|
||||
|
||||
### Acceptance criteria (to be detailed in gamification-spec.md)
|
||||
|
||||
- [ ] Completion writes to session_completions
|
||||
- [ ] Commits calculated and added to gamification_profile
|
||||
- [ ] Level updates correctly at commit thresholds
|
||||
- [ ] Streak increments on weekly completion, resets on skip
|
||||
- [ ] Badge awarded when condition is met
|
||||
- [ ] Milestone card created at weeks 13 and 26
|
||||
- [ ] Leaderboard endpoint returns all employees with correct data
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Integration + hardening
|
||||
|
||||
No new spec required.
|
||||
|
||||
### Steps
|
||||
|
||||
**8.1 — Service wiring**
|
||||
Verify all services communicate through PocketBase correctly.
|
||||
No direct service-to-service calls — all state through PocketBase.
|
||||
|
||||
**8.2 — Error handling audit**
|
||||
Review all services for unhandled promise rejections, missing error states,
|
||||
and uncaught exceptions. Every external call (AI API, PocketBase, Qdrant,
|
||||
OpenAI) wrapped in try/catch with meaningful error logging.
|
||||
|
||||
**8.3 — Mobile QA**
|
||||
Test all employee app flows at 375px width.
|
||||
R42 floating button must not obscure content.
|
||||
Heatmap must render without horizontal scroll.
|
||||
|
||||
**8.4 — Environment variable audit**
|
||||
Verify no hardcoded values. All .env.example files complete.
|
||||
|
||||
**8.5 — Dockerfile update**
|
||||
Update COPY path from legacy app root to /app.
|
||||
This is the one manual change that connects the rebuild to the existing pipeline.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] Full flow works end-to-end: upload doc → approve → curriculum → employee completes session → R42 answers question → gamification updates
|
||||
- [ ] No uncaught exceptions in any service under normal operating conditions
|
||||
- [ ] All screens render correctly on 375px mobile
|
||||
- [ ] Dockerfile builds successfully pointing at /app
|
||||
- [ ] Existing pipeline deploys the rebuilt app without modification
|
||||
|
||||
---
|
||||
|
||||
## Spec files still to be written
|
||||
|
||||
Before starting each phase, write the corresponding spec file.
|
||||
Use ingestion-spec.md as the template for structure and detail level.
|
||||
|
||||
| Phase | Spec file needed |
|
||||
|---|---|
|
||||
| 2 | /docs/generation-spec.md |
|
||||
| 3 | /docs/curriculum-spec.md |
|
||||
| 4–5 | /docs/frontend-spec.md |
|
||||
| 6 | /docs/r42-spec.md |
|
||||
| 7 | /docs/gamification-spec.md |
|
||||
|
||||
When you reach a phase without a spec: stop, draft the spec, then proceed.
|
||||
Do not implement without a spec.
|
||||
483
docs/ingestion-spec.md
Normal file
483
docs/ingestion-spec.md
Normal file
@@ -0,0 +1,483 @@
|
||||
# Ingestion service spec
|
||||
|
||||
## Responsibility
|
||||
|
||||
Accepts uploaded source documents (PDF, MD, TXT), extracts clean text, chunks it,
|
||||
generates embeddings, and produces a structured draft KB (Themes + Topics +
|
||||
relationships) ready for admin review.
|
||||
|
||||
This service runs entirely server-side. The admin app calls it via REST. All AI
|
||||
calls go through the Anthropic API. No ingestion logic lives in the frontend.
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
|
||||
```
|
||||
app/services/ingestion/
|
||||
├── index.ts entry point, Fastify server
|
||||
├── routes/
|
||||
│ └── documents.ts POST /ingest, GET /status/:jobId
|
||||
├── pipeline/
|
||||
│ ├── extract.ts format detection + text extraction
|
||||
│ ├── chunk.ts chunking strategies per format
|
||||
│ ├── clean.ts chunk cleaning
|
||||
│ ├── structure.ts Claude call → Theme/Topic extraction
|
||||
│ └── embed.ts embedding generation + Qdrant write
|
||||
├── jobs/
|
||||
│ └── queue.ts async job queue (in-memory, BullMQ later if needed)
|
||||
├── lib/
|
||||
│ ├── pocketbase.ts PocketBase client
|
||||
│ ├── qdrant.ts Qdrant client
|
||||
│ ├── anthropic.ts Anthropic client
|
||||
│ └── openai.ts OpenAI embeddings client
|
||||
└── types.ts shared TypeScript types
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
|
||||
### POST /ingest
|
||||
|
||||
Triggered by admin app on document upload.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"documentId": "string",
|
||||
"filename": "string",
|
||||
"format": "pdf" | "md" | "txt",
|
||||
"filePath": "string"
|
||||
}
|
||||
```
|
||||
|
||||
Response (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued"
|
||||
}
|
||||
```
|
||||
|
||||
Processing is async. The admin app polls job status.
|
||||
|
||||
---
|
||||
|
||||
### GET /status/:jobId
|
||||
|
||||
Returns current job progress.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued" | "extracting" | "chunking" | "structuring" | "embedding" | "done" | "failed",
|
||||
"progress": {
|
||||
"chunksTotal": 42,
|
||||
"chunksEmbedded": 18,
|
||||
"themesFound": 3,
|
||||
"topicsFound": 14
|
||||
},
|
||||
"error": "string | null"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
### Stage 1 — Text extraction
|
||||
|
||||
Input: file path + format
|
||||
Output: raw text string
|
||||
|
||||
```
|
||||
format === 'txt'
|
||||
→ read file directly as UTF-8
|
||||
|
||||
format === 'md'
|
||||
→ read file directly as UTF-8
|
||||
→ preserve heading markers (# ## ###) — used in chunking
|
||||
|
||||
format === 'pdf'
|
||||
→ pdfplumber: extract text page by page
|
||||
→ concatenate with page break markers: \n\n---PAGE---\n\n
|
||||
→ strip known PDF artefacts: headers/footers repeating on every page,
|
||||
page numbers, watermarks
|
||||
```
|
||||
|
||||
Failure handling:
|
||||
- PDF extraction returns empty string → mark job `failed`, reason: `pdf_extraction_empty`
|
||||
- File not found → mark job `failed`, reason: `file_not_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 2 — Chunking
|
||||
|
||||
Input: raw text + format
|
||||
Output: Chunk[]
|
||||
|
||||
Chunking strategy differs per format.
|
||||
|
||||
**MD chunking — heading-based (preferred)**
|
||||
```
|
||||
Split on heading markers: #, ##, ###
|
||||
Each heading + its following content = one chunk
|
||||
Minimum chunk size: 100 characters
|
||||
→ if heading section is < 100 chars, merge with next sibling
|
||||
Maximum chunk size: 1500 characters
|
||||
→ if section exceeds limit, split on paragraph breaks within section
|
||||
Metadata preserved per chunk:
|
||||
heading_level: 1 | 2 | 3
|
||||
heading_text: string
|
||||
parent_heading: string | null
|
||||
```
|
||||
|
||||
MD chunking produces the highest quality structural signal for Theme/Topic extraction.
|
||||
Admins should be advised to provide source material as MD where possible.
|
||||
|
||||
**TXT chunking — sliding window**
|
||||
```
|
||||
Window size: 800 characters
|
||||
Overlap: 150 characters
|
||||
Split on: paragraph breaks (\n\n) first, then sentence boundaries, then hard cut
|
||||
Metadata per chunk:
|
||||
chunk_index: number
|
||||
approximate_position: 'start' | 'middle' | 'end'
|
||||
```
|
||||
|
||||
**PDF chunking — page + paragraph**
|
||||
```
|
||||
Split on ---PAGE--- markers from extraction stage
|
||||
Within each page: split on paragraph breaks (\n\n)
|
||||
Minimum chunk size: 100 characters
|
||||
→ merge sub-threshold paragraphs with adjacent chunk
|
||||
Maximum chunk size: 1200 characters
|
||||
→ hard split at sentence boundary
|
||||
Metadata per chunk:
|
||||
page_number: number
|
||||
chunk_index_on_page: number
|
||||
```
|
||||
|
||||
**Chunk type:**
|
||||
```typescript
|
||||
type Chunk = {
|
||||
id: string // UUID generated at chunking
|
||||
documentId: string
|
||||
text: string
|
||||
format: 'pdf' | 'md' | 'txt'
|
||||
index: number // global position in document
|
||||
metadata: {
|
||||
// MD-specific
|
||||
headingLevel?: number
|
||||
headingText?: string
|
||||
parentHeading?: string
|
||||
// TXT-specific
|
||||
approximatePosition?: 'start' | 'middle' | 'end'
|
||||
// PDF-specific
|
||||
pageNumber?: number
|
||||
chunkIndexOnPage?: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 3 — Chunk cleaning
|
||||
|
||||
Input: Chunk[]
|
||||
Output: Chunk[] (cleaned)
|
||||
|
||||
Applied to all formats:
|
||||
```
|
||||
- trim leading/trailing whitespace
|
||||
- collapse 3+ consecutive newlines to 2
|
||||
- remove null bytes and non-printable characters
|
||||
- remove chunks where text.length < 80 after cleaning
|
||||
→ these are likely artefacts (page numbers, standalone headers)
|
||||
- normalise unicode: NFC normalisation
|
||||
- do not strip punctuation or alter sentence structure
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 4 — Structure extraction (AI)
|
||||
|
||||
Input: Chunk[]
|
||||
Output: DraftKB
|
||||
|
||||
This is the core AI call. Claude Sonnet 4 reads all chunks and returns a structured
|
||||
KB draft as JSON.
|
||||
|
||||
**Prompt strategy:**
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a knowledge architect. Your task is to analyse a set of text chunks from
|
||||
a source document and extract a structured knowledge base.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
|
||||
no markdown fences.
|
||||
|
||||
Rules:
|
||||
- Group related content into Themes. A Theme is a broad subject area.
|
||||
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
|
||||
- Identify relationships between Topics: related, prerequisite, or contrast.
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A must be understood before Topic B
|
||||
- contrast: Topics that represent opposing approaches or concepts
|
||||
- For each Topic, extract key terms suitable for a glossary.
|
||||
- Assign a complexity weight (1–5) to each Topic.
|
||||
1 = introductory, 5 = advanced
|
||||
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
|
||||
- Draft a description for each Theme (1–2 sentences).
|
||||
- Every Topic must reference the chunk IDs that contributed to it.
|
||||
```
|
||||
|
||||
User prompt:
|
||||
```
|
||||
Source document: {filename}
|
||||
Format: {format}
|
||||
|
||||
Chunks:
|
||||
{chunks mapped as: [CHUNK-{id}]\n{text}\n}
|
||||
|
||||
Extract the knowledge base structure from these chunks.
|
||||
```
|
||||
|
||||
**Output schema:**
|
||||
```typescript
|
||||
type DraftKB = {
|
||||
themes: DraftTheme[]
|
||||
}
|
||||
|
||||
type DraftTheme = {
|
||||
title: string
|
||||
description: string
|
||||
topics: DraftTopic[]
|
||||
}
|
||||
|
||||
type DraftTopic = {
|
||||
title: string
|
||||
body: string
|
||||
difficulty: 'introductory' | 'intermediate' | 'advanced'
|
||||
complexityWeight: number // 1–5
|
||||
keyTerms: string[]
|
||||
sourceChunkIds: string[] // references Chunk.id values
|
||||
relationships: {
|
||||
related: string[] // topic titles (resolved to IDs after write)
|
||||
prerequisites: string[]
|
||||
contrasts: string[]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**AI call configuration:**
|
||||
```typescript
|
||||
{
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
max_tokens: 8000,
|
||||
temperature: 0 // deterministic output for structured extraction
|
||||
}
|
||||
```
|
||||
|
||||
**Chunking strategy for large documents:**
|
||||
If total chunk count exceeds 60 chunks, split into batches of 40 with 5-chunk
|
||||
overlap. Run one Claude call per batch. Merge resulting DraftKB objects:
|
||||
- Themes with identical titles → merge Topics
|
||||
- Duplicate Topic titles within a Theme → keep longer body, merge sourceChunkIds
|
||||
- Relationships are resolved after full merge
|
||||
|
||||
**Error handling:**
|
||||
- JSON parse failure → retry once with stricter prompt ("ensure valid JSON only")
|
||||
- Second failure → mark job `failed`, reason: `structure_extraction_failed`, log raw response
|
||||
- Empty themes array → mark job `failed`, reason: `no_structure_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 5 — Write to PocketBase
|
||||
|
||||
Input: DraftKB
|
||||
Output: written Theme + Topic records with status `draft`
|
||||
|
||||
```
|
||||
For each DraftTheme:
|
||||
create themes record {
|
||||
title, description,
|
||||
status: 'draft',
|
||||
source_documents: [documentId]
|
||||
}
|
||||
|
||||
For each DraftTopic under the theme:
|
||||
create topics record {
|
||||
theme: themeId,
|
||||
title, body, difficulty, complexity_weight, key_terms,
|
||||
status: 'draft',
|
||||
qdrant_chunk_ids: [] // populated in stage 6
|
||||
}
|
||||
|
||||
After all topics created:
|
||||
resolve relationship titles → topic IDs
|
||||
update topics.related_topics, prerequisite_topics, contrast_topics
|
||||
|
||||
If a relationship title cannot be resolved to an existing topic:
|
||||
skip silently (cross-document relationships resolved in a later pass)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 6 — Embedding generation + Qdrant write
|
||||
|
||||
Input: Chunk[], written Topic records
|
||||
Output: vectors in Qdrant, qdrant_chunk_ids updated on Topic records
|
||||
|
||||
**Source chunk embeddings:**
|
||||
```
|
||||
For each Chunk (post-cleaning):
|
||||
embed Chunk.text → text-embedding-3-small (1536 dimensions)
|
||||
write to Qdrant collection: source_chunks {
|
||||
id: Chunk.id,
|
||||
vector: float[],
|
||||
payload: {
|
||||
source_document_id: documentId,
|
||||
chunk_index: Chunk.index,
|
||||
text: Chunk.text,
|
||||
theme_id: resolved themeId | null,
|
||||
topic_id: resolved topicId | null,
|
||||
format: Chunk.format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Topic summary embeddings:**
|
||||
```
|
||||
For each published Topic:
|
||||
embed Topic.body → text-embedding-3-small
|
||||
write to Qdrant collection: topic_summaries {
|
||||
id: UUID,
|
||||
vector: float[],
|
||||
payload: {
|
||||
topic_id: Topic.id,
|
||||
theme_id: Topic.theme,
|
||||
title: Topic.title,
|
||||
text: Topic.body
|
||||
}
|
||||
}
|
||||
|
||||
Update Topic.qdrant_chunk_ids with all Chunk.ids that reference this topic
|
||||
```
|
||||
|
||||
**Batching:**
|
||||
OpenAI embeddings API: batch in groups of 100 texts per request to stay within
|
||||
rate limits and reduce latency.
|
||||
|
||||
---
|
||||
|
||||
## Job lifecycle
|
||||
|
||||
```
|
||||
POST /ingest received
|
||||
↓
|
||||
Job created → status: queued
|
||||
↓
|
||||
Stage 1: extracting
|
||||
↓
|
||||
Stage 2–3: chunking
|
||||
↓
|
||||
Stage 4: structuring
|
||||
↓
|
||||
Stage 5: writing to PocketBase
|
||||
↓
|
||||
Stage 6: embedding
|
||||
↓
|
||||
status: done
|
||||
↓
|
||||
Admin notification: "Document processed. N themes, N topics ready for review."
|
||||
↓
|
||||
Curriculum regeneration queued (status: pending_admin_confirm)
|
||||
```
|
||||
|
||||
On any stage failure:
|
||||
```
|
||||
status: failed
|
||||
error: { stage, reason, detail }
|
||||
Source document status → 'failed' in PocketBase
|
||||
Admin notification: "Ingestion failed: {reason}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables required
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
QDRANT_URL=
|
||||
QDRANT_API_KEY= # empty string if running locally without auth
|
||||
INGESTION_PORT=3001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"openai": "^4",
|
||||
"@qdrant/js-client-rest": "^1.9",
|
||||
"pocketbase": "^0.21",
|
||||
"pdfplumber": "NOT JS — see note below",
|
||||
"pdf-parse": "^1.1",
|
||||
"uuid": "^9",
|
||||
"zod": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**PDF extraction note:**
|
||||
`pdfplumber` is a Python library. Two options:
|
||||
1. Use `pdf-parse` (Node.js) — simpler, covers 90% of cases
|
||||
2. Run `pdfplumber` as a Python sidecar process via child_process — higher quality
|
||||
for complex PDFs with tables and columns
|
||||
|
||||
Default to `pdf-parse` initially. Add pdfplumber sidecar only if extraction
|
||||
quality is insufficient for actual source documents.
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- All Claude response parsing through Zod schema validation
|
||||
- All PocketBase writes typed against collection schemas from `data-model.md`
|
||||
- Qdrant payloads typed explicitly — no untyped objects
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not generate micro learnings → generation service
|
||||
- Does not build or update the curriculum → curriculum service
|
||||
- Does not handle admin approval → admin app + PocketBase directly
|
||||
- Does not serve R42 queries → chat service
|
||||
- Does not handle auth → PocketBase + admin app
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
Before handing to Claude Code for implementation, verify manually:
|
||||
|
||||
1. Upload a short MD file (< 10 headings) → inspect chunk output → confirm heading structure preserved
|
||||
2. Upload a simple PDF (< 5 pages) → inspect chunk output → confirm no artefacts
|
||||
3. Run structure extraction on known chunks → validate JSON parses against Zod schema
|
||||
4. Confirm PocketBase draft records created with correct theme → topic hierarchy
|
||||
5. Confirm Qdrant source_chunks collection populated with correct payload fields
|
||||
6. Confirm topic.qdrant_chunk_ids updated after embedding stage
|
||||
Reference in New Issue
Block a user