diff --git a/AI_AGENT.md b/AI_AGENT.md index 412cead..a4c2060 100644 --- a/AI_AGENT.md +++ b/AI_AGENT.md @@ -2,144 +2,150 @@ Welcome, fellow AI agent! If you are reading this, you are tasked with maintaining or extending the Respellion Learning Platform. This document provides the critical context, architectural patterns, and design standards you need to successfully work on this codebase. -> **Last updated:** 2026-05-18 — Adds 52-week annual curriculum system (§12). Reflects selective content generation (3 types), ISO week alignment, GitHub sync folder change, and AI extraction token limits. +> **Last updated:** 2026-05-26 — Per-user curriculum start (employees enroll on first login; week/cycle derived from each user's `curriculum_started_at`, no longer from the ISO calendar week). Documents the real React/Vite + PocketBase stack, TF-IDF retrieval, 3 learning-content types, 3 micro-learning types, and the tiered Claude model setup. ## 1. Architectural Overview -This is a single-page React application built with **Vite**, backed by **PocketBase** as the database and auth layer. -* **Frontend:** React, React Router, Vanilla CSS (via CSS variables) + Tailwind utility classes mapped to those variables. -* **Backend:** PocketBase (self-hosted). All data is stored in PocketBase collections, not localStorage. -* **Animations:** Framer Motion (used extensively for page transitions, podium effects, and gamification feedback). +This is a single-page React application built with **Vite**, backed by **PocketBase** as the database and auth layer. There is no separate backend server — the browser talks directly to PocketBase, and to the Anthropic API through a reverse proxy. +* **Frontend:** React 19, React Router 7, Vanilla CSS (via CSS variables) + Tailwind v4 utilities mapped to those variables. +* **Backend:** PocketBase (self-hosted, SQLite). All data is stored in PocketBase collections, not localStorage. +* **Animations:** Framer Motion (page transitions, podium effects, gamification feedback). * **Icons:** Lucide React. * **Visualizations:** D3.js (used strictly for the Admin Knowledge Graph). +* **Retrieval:** A dependency-free TF-IDF index over the knowledge graph (`src/lib/retrieval.js`). There is **no Qdrant and no embeddings API** — older specs that mention them describe a design that was never built. + +> The top-level `app/` directory is abandoned Next.js scaffolding from that original design. It is not built or deployed. Ignore it; the real app is `src/`. ## 2. State Management & Storage (Critical) All persistent data lives in **PocketBase**. The data access layer is in `src/lib/db.js`, which wraps the PocketBase SDK client from `src/lib/pb.js`. -**PocketBase Collections:** -* `topics` — Knowledge graph nodes (`id`, `label`, `type`, `description`). -* `relations` — Knowledge graph edges (`source`, `target`, `type`). -* `content` — AI-generated learning modules per topic (`topic_id`, `data`). The `data` field is a **merged JSON object** containing only the content types that have been generated for that topic (e.g. `{ article: {...}, slides: [...] }`). New types are shallow-merged into the existing object by `learningService.js`; nothing is ever overwritten. -* `quiz_banks` — Banks of AI-generated questions per topic (`topic_id`, `questions[]`). -* `quiz_results` — User test scores (`user_id`, `week_number`, `score`, `total`, `percentage`, etc.). -* `quiz_cache` — Cached weekly quiz for a user (`user_id`, `week_number`, `questions[]`, `meta`). -* `learn_progress` — Whether a user completed the weekly learning session (`user_id`, `week_number`, `done`). -* `leaderboard` — Points ledger (`user_id`, `name`, `points`, `tests_completed`). -* `team_members` — Registered users with PIN auth (`name`, `role`, `pin`). -* `sources` — Uploaded source documents and their extraction status (`name`, `status`, `error`). -* `curriculum` — Annual learning schedule (`year`, `week_number`, `topic_id`, `theme`, `quarter`, `is_review_week`, `sort_order`). One entry per week per year. Managed via the admin Curriculum tab. -* `settings` — Key/value store for app-wide settings (`key`, `value`). +**PocketBase Collections (current):** +* `topics` — Knowledge graph nodes (`id`, `label`, `type`, `description`, `learning_relevance`, `relevance_locked`, `theme`, `complexity_weight`, `difficulty`). +* `relations` — Knowledge graph edges (`source`, `target`, `type` — `related_to` / `depends_on` / `part_of` / `executed_by`). +* `content` — AI-generated learning modules per topic (`topic_id`, `data`). The `data` field is a **merged JSON object** containing only the content types generated so far (e.g. `{ article: {...}, slides: [...] }`). New types are shallow-merged in by `learningService.js`; nothing is overwritten. +* `micro_learnings` — Generated micro-learning artifacts (`topic_id`, `type`, `content`, `status`). One record per topic per type. `status='published'` items are visible to employees. +* `micro_learning_completions` — Append-only completion events (`team_member_id`, `micro_learning_id`, `topic_id`, `type`, `session_week`). +* `curriculum_versions` — Versioned 26-week schedules (`version_number`, `status`, `generation_reason`, `confirmed_by`, `confirmed_at`, `schedule` JSON, `coverage_stats` JSON). Exactly one `active` at a time. +* `leaderboard` — Points ledger (`user_id`, `name`, `points`, `tests_completed`, `learnings_completed`). +* `team_members` — Registered users with PIN auth (`name`, `role`, `pin`, `curriculum_started_at`, `enrollment_status`). +* `sources` — Uploaded source documents and extraction status (`name`, `status`, `error`, `progress`). +* `settings` — Key/value store (`key`, `value`). +* `llm_calls` — Per-call telemetry (`task`, `model`, `tier`, `duration_ms`, token counts, `stop_reason`, `ok`, `error_msg`). + +**Dropped collections:** `quiz_banks`, `quiz_results`, `quiz_cache`, `learn_progress`, and the legacy `curriculum` (v1) collection no longer exist. The matching `db.js` helpers are deprecated stubs — do not build on them. **localStorage** is only used for **admin browser settings** (not user data): -* `respellion:admin:anthropic_key` — Anthropic API key. -* `respellion:admin:model` — Model override. -* `respellion:admin:use_simulation` — Simulation mode toggle. -* `kb:suggestions` — Pending/approved/rejected graph deltas proposed by R42. Always mutated via `kbStore` (see §8). -* `quiz:active:{userId}` — Boolean flag set while the user is mid-quiz. R42's FAB is hidden when this is true (quiz-integrity rule). +* `admin:model:fast` / `admin:model:standard` / `admin:model:reasoning` — per-tier model overrides (legacy `admin:model` still honored for `standard`). +* `admin:use_simulation` — when true, `callLLM` returns stub data instead of calling Anthropic. Useful for UI work without spending tokens. +* `kb:suggestions` — Pending/approved/rejected graph deltas proposed by R42. Always mutated via `kbStore` (see §9). +* `quiz:active:{userId}` — Boolean flag set while a user is mid-quiz. R42's launcher is hidden when this is true (quiz-integrity rule). * `chat:thread:{userId}` — Persisted R42 conversation, capped at 50 messages. -**Session:** User login is PIN-based. The logged-in user's ID is stored in `sessionStorage` under `respellion_session` and resolved against `team_members` on app load (see `src/store/AppContext.jsx`). +**Session:** Login is PIN-based. The logged-in user's ID is stored in `sessionStorage` under `respellion_session` and resolved against `team_members` on app load (`src/store/AppContext.jsx`). -**Week Number:** The current ISO-8601 week number is calculated dynamically on app load via `getWeekNumber(new Date())` in `src/store/AppContext.jsx`. It is **not** stored in the database. The `ADVANCE_WEEK` action still exists for admin use, but initial state always reflects the real calendar week. +**Per-user curriculum position (important — changed):** Each employee starts the curriculum when *they* choose. On first login a blocking onboarding screen (`/onboarding`) records `curriculum_started_at` and flips `enrollment_status` to `active`. `AppContext` derives `state.weekNumber` — an absolute counter starting at 1 — from `getPersonalWeekNumber(curriculum_started_at)` (= `floor(days_since_start / 7) + 1`). The 26-week slot and cycle come from `getCurriculumWeek(n)` (`((n-1) % 26) + 1`) and `getCurriculumCycle(n)` (`floor((n-1)/26)+1`) in `curriculumService.js`. The cycle is **detached from the ISO calendar** — week 1 is simply the first 7 days after the user's start. After week 26 the cycle restarts at week 1 with the same content. `state.weekNumber` is `0` until the user enrolls. -**Curriculum Year:** The curriculum year is derived from `new Date().getFullYear()` via `getCurriculumYear()` in `src/lib/curriculumService.js`. It is not stored — always computed. +> Do **not** reintroduce ISO-week-based scheduling or a shared `admin:current_week`. There is no global "current week" anymore — every employee has their own. -**Important:** All `db.js` functions are `async`. Always `await` them — omitting `await` will silently pass a Promise where data is expected. +**Auto-Cancellation:** The PocketBase JS SDK has auto-cancellation enabled by default, which aborts concurrent identical requests (common under React StrictMode and `Promise.all`) with `ClientResponseError 0`. It is **globally disabled** via `pb.autoCancellation(false)` in `src/lib/pb.js`. Never re-enable it. -**Auto-Cancellation:** The PocketBase JS SDK has auto-cancellation enabled by default. This causes concurrent identical requests (like `db.getTopics()` during React StrictMode renders or concurrent Promise.all) to abort with `ClientResponseError 0`. This feature is **globally disabled** in `src/lib/pb.js` via `pb.autoCancellation(false)` to prevent UI crashes during concurrent fetching. +**PocketBase URL:** Resolved from `VITE_PB_URL`, else `window.location.origin` (`src/lib/pb.js`). In production Caddy proxies `/api/*` and `/_/*` to the PocketBase container. + +**Important:** All `db.js` functions are `async`. Always `await` them — omitting `await` silently passes a Promise where data is expected. ## 3. The AI Integration (Anthropic) -The application calls the Anthropic API via a proxy to avoid CORS issues. -* **Location:** `src/lib/llm.js`. -* **Proxy:** In Docker, `/api/anthropic` is proxied via Caddy to the Anthropic API endpoint. In local dev, configure `vite.config.js` to proxy the same path. The API key is injected server-side by Caddy; there is **no client-side API key**. -* **Token limit:** `generateContent` uses `max_tokens: 8192`. Do not lower this — large knowledge-base files need the headroom. -* **JSON Enforcement:** Prompts *must* strictly enforce that Claude returns *only* raw JSON. Do not let the AI wrap the response in markdown blocks — a regex strip via `.match(/\{[\s\S]*\}/)` is already applied in all service files. +All Anthropic calls go through one wrapper. +* **Location:** `src/lib/llm.js`, function `callLLM(...)`. Callers must never reach `/api/anthropic` directly. +* **Proxy:** Requests hit `/api/anthropic/v1/messages`. In Docker, Caddy proxies this to `https://api.anthropic.com` and injects the `x-api-key` header server-side. In local dev, `vite.config.js` does the same using `process.env.ANTHROPIC_API_KEY`. There is **no client-side API key**. +* **Model tiers:** `fast` = `claude-haiku-4-5-20251001`, `standard` = `claude-sonnet-4-6`, `reasoning` = `claude-opus-4-7`. Choose a tier per task; admins can override per tier from Settings. +* **Structured output:** Prefer Anthropic **tool use** with a forced `toolChoice`. Tool inputs are validated against Zod schemas in `src/lib/llmSchemas.js` (auto-looked-up via `toolSchemaRegistry`). For text responses, `parseStructuredText` strips code fences and extracts the outermost balanced JSON. +* **Prompt caching:** Wrap stable system text with `cachedSystem(text)` to attach `cache_control: ephemeral`. +* **Retry/limits:** `src/lib/llmRetry.js` handles exponential backoff with jitter on retryable statuses (408/425/429/5xx/529), honors `Retry-After`, and provides rate limiters (e.g. the extraction limiter caps ~20 req/min, burst 2). Default `maxTokens` is 4096; extraction and long-form content use 8192. +* **Telemetry:** Every call is logged (best-effort, non-blocking) to the `llm_calls` collection. ## 4. Design System & Aesthetics Respellion relies on a premium, modern aesthetic. -* **CSS Variables:** Rely heavily on the variables defined in `src/index.css`. - * Colors: `var(--color-bg)`, `var(--color-paper)`, `var(--color-teal)`, `var(--color-accent)`. - * Radii: `var(--r-sm)`, `var(--r-lg)`, `var(--r-org)` (an organic, pill-like shape used for badges and podiums). -* **Tailwind:** Tailwind is mapped to these CSS variables. Avoid hardcoding random hex codes. Use the existing Tailwind classes like `bg-teal`, `text-fg-muted`, and `border-bg-warm`. -* **Components:** Always reuse the UI primitives in `src/components/ui/` (`Card.jsx`, `Button.jsx`, `Tag.jsx`, `Input.jsx`). +* **CSS Variables:** Rely on the variables in `src/index.css` — `var(--color-bg)`, `var(--color-paper)`, `var(--color-teal)`, `var(--color-accent)`, radii `var(--r-sm)`, `var(--r-lg)`, `var(--r-org)`. +* **Tailwind:** Tailwind v4 utilities are mapped to these variables. Use classes like `bg-teal`, `text-fg-muted`, `border-bg-warm`. Avoid raw hex codes. +* **`stylesheet.css`** (repo root) is the authoritative visual reference and is frozen — do not edit it. +* **Components:** Reuse the UI primitives in `src/components/ui/` (`Card.jsx`, `Button.jsx`, `Tag.jsx`, `Input.jsx`). -## 5. Learning Content Types -`src/lib/learningService.js` supports **three** content types for selective generation: +## 5. Learning Content Types (on-demand, per topic) +`src/lib/learningService.js` generates **three** long-form content types into the `content` collection, on demand: | Type | Schema key | Description | |---|---|---| | `article` | `content.article` | Title, intro, sections, key takeaways | -| `slides` | `content.slides` | Array of slides with bullets and speaker notes | +| `slides` | `content.slides` | Slides with bullets and speaker notes | | `infographic` | `content.infographic` | Headline, tagline, stats, steps, quote | -**There is no podcast type.** It was removed. Do not re-add it. +`generateLearningContent(topic, force, selectedType)` accepts one of the three types, or `'all'` (admin regeneration). Cache-hit logic checks `content[selectedType]` directly; new data is shallow-merged so other types are preserved. **There is no podcast type.** -`generateLearningContent(topic, force, selectedType)` accepts one of the three types above, or `'all'` (admin regeneration). Cache-hit logic checks `content[selectedType]` directly. On generation, the new data is shallow-merged into the existing cached object so other types are preserved. +Article refinement uses targeted patch tools (`set_intro`, `set_section`, `add_section`, `remove_section`, `replace_takeaways`) so the model edits only what changed. -The `LearningContentViewer` tab bar reflects exactly these three modes. The empty-state for an un-generated tab shows a "Generate [Type]" button that calls `onGenerate(activeMode)` passed from the parent. +## 6. Micro-Learnings (the weekly session) +`src/lib/microLearningService.js` generates short, single-topic interactions into the `micro_learnings` collection. **Three** types are currently active (a former `reflection_prompt` type was dropped): +| Type | Tier | Shape | +|---|---|---| +| `concept_explainer` | standard | `{ sections: [{ title, content (HTML) }] }` (≥3 sections) | +| `scenario_quiz` | standard | `{ scenario, options: [{ text, isCorrect, explanation }] }` (3–4 options, 1 correct) | +| `flashcard_set` | fast (Haiku) | `{ cards: [{ front, back }] }` (5–10 cards) | -## 6. Gamification Rules -If you are extending the Gamification system (`src/pages/Leaderboard.jsx`): -* Tests grant **+2 points** per correct answer (handled in `testService.js → saveTestResult`). -* A 100% score grants the **Perfectionist** badge (computed at render time in `Leaderboard.jsx`). -* Completing 1 test grants **First Steps**, completing 5 grants **Veteran**. -* Points are accumulated in the `leaderboard` collection via `db.upsertLeaderboardEntry`. +`getOrGenerateMicroLearning(topicId, type)` returns a cached published record if present, else generates and stores one. Completions are recorded via `useMicroLearningCompletions` into `micro_learning_completions` (append-only). A weekly session is "done" when every required topic has at least one completed micro-learning. -## 7. Docker & Deployment -The app is fully containerized. PocketBase runs as a sidecar service. -* **Build:** `docker build -t respellion-app:latest .` -* **Run:** `docker compose up -d` (see `docker-compose.yml`). -* **Caddy (reverse proxy):** Handles SPA fallback, injects the Anthropic API key via a `Authorization` header on `/api/anthropic/*` requests, and proxies `/pb/*` to the PocketBase service. Config lives in `Caddyfile`. -* **PocketBase URL:** Resolved from `VITE_PB_URL` env var, or falls back to `window.location.origin + '/pb'` (see `src/lib/pb.js`). +## 7. Weekly Test +`src/lib/testService.js` builds a 5-question multiple-choice quiz for the user's current week. +* Primary topic comes from the active curriculum week (else a deterministic hash fallback), plus a few review topics for breadth. +* Generated in **one** `fast`-tier batch call (`emit_quiz_questions`), with quality gates: no duplicate options, no banned fillers ("all of the above"), explanations ≥20 chars, and a check that `correctIndex` isn't dominated by one position. +* Scoring: **+2 points per correct answer** (`saveTestResult` → `score * 2`), written to the `leaderboard` collection. -## 8. Local File Upload -The Admin upload panel (`src/components/admin/UploadZone.jsx`) allows admins to manually upload markdown/text files via a drag-and-drop interface. - -* **Extraction pipeline (`src/lib/extractionPipeline.js`):** Calls `callLLM` with a strict JSON-only system prompt. To prevent truncated responses on large files, the prompt limits extraction to **max 15 topics** and their most important relations. If the AI returns non-JSON, the file is marked `failed` in the `sources` collection. -* **Deduplication:** A file already in `sources` with `status: completed` will throw and not be re-processed. Delete the source record first to force a re-analysis. -* **Do not increase topic cap beyond 15** without also verifying the `max_tokens: 8192` budget is sufficient for the expected file size. +## 8. Gamification +If you are extending gamification (`src/pages/Leaderboard.jsx`): +* Tests grant **+2 points** per correct answer (`testService.js → saveTestResult`). +* Badges are computed at render time: **First Steps** (1 test), **Veteran** (5 tests), **Perfectionist** (a 100% score). +* Points accumulate in `leaderboard` via `db.upsertLeaderboardEntry`. Admins are excluded from the public board. ## 9. R42 Chatbot The platform ships a global chatbot avatar called **R42**, rendered as the Respellion `{ r }` brand mark in three states (idle / typing / error). - -* **Mark component:** `src/components/ui/Mark.jsx`. Pure SVG; renders the brand mark with `state`, `size`, `theme`, `showFrame`, `brace`, `letter` props. Requires the `BallPill` font (loaded via `@font-face` in `src/index.css`, served from `public/fonts/BallPill-light.otf`). Falls back to JetBrains Mono (already loaded). +* **Mark component:** `src/components/ui/Mark.jsx` — pure SVG with `state`, `size`, `theme`, `brace`, `letter` props. Uses the `BallPill` font (`@font-face` in `src/index.css`), falling back to JetBrains Mono. * **Chat module:** `src/components/chat/`. - * `ChatLauncher.jsx` — global FAB; auto-hides when `quiz:active:{userId}` is set. Listens to the `respellion:quiz-state` window event for fast updates. - * `ChatWindow.jsx` — 380×480 chat panel; Esc closes; renders messages from `useChat`. - * `useChat.js` — owns the message list, persists to `chat:thread:{userId}`, calls `anthropicApi.chat()`. `buildKbContext` is async (PocketBase), so `send()` is fully async. - * `prompts.js` — system prompt, greeting, and the `propose_graph_delta` tool spec. - * `rag.js` — fetches `kb:topics` + `kb:relations` from PocketBase; lazy-loads `kb:content:{id}` only when a topic is mentioned. Returns `{ context, topics }` so `validateDelta` can reuse the fetched topics without a second round-trip. -* **Multi-turn API:** `callLLM({ task, system, messages, tools })` in `src/lib/llm.js`. Returns a structured response containing extracted `toolUses` and text. No API key header — Caddy proxy injects it server-side. -* **Quiz-integrity rule:** `src/pages/Testen.jsx` sets `quiz:active:{userId}=true` on start and clears it on every non-quiz phase + unmount, plus dispatches a `respellion:quiz-state` event. Never bypass this — letting users ask R42 mid-quiz would break scoring. -* **Graph refinement:** when R42's `tool_use` block proposes a `propose_graph_delta`, `rag.js` validates (no duplicate ids, no orphan relations, caps 3 topics / 5 relations) and surfaces a confirmation chip inline. - * **Admin user clicks Ja** — `kbStore.applyDelta` writes to PocketBase via `db.upsertTopic` / `db.addRelation` immediately. - * **Non-admin clicks Ja** — `kbStore.appendSuggestion` queues an entry in `kb:suggestions` localStorage (status `pending`). -* **Admin approval UI:** `src/components/admin/SuggestionsQueue.jsx`, mounted at the top of the Knowledge Graph admin panel. Approve calls `kbStore.approveSuggestion(id)` which runs the same `applyDelta` merge (async, PocketBase) and flips status to `approved`. Reject flips to `rejected` for audit. -* **kbStore:** `src/lib/kbStore.js` is the single source of truth for KB mutations from the chatbot path. Topics/relations go to PocketBase; suggestions queue goes to localStorage. Dispatches `respellion:kb-updated` after any write so the D3 graph and the queue panel refresh without a reload. + * `ChatLauncher.jsx` — global FAB; auto-hides when `quiz:active:{userId}` is set; listens to the `respellion:quiz-state` window event. + * `ChatWindow.jsx` — chat panel; renders messages from `useChat`; surfaces graph-delta confirmation chips. + * `useChat.js` — owns the message list, persists to `chat:thread:{userId}` (cap 50; only the last ~12 turns are sent to the API), calls `callLLM`. + * `prompts.js` — the cacheable system prompt blocks, greeting, and the `propose_graph_delta` tool spec (max 3 topics / 5 relations). + * `rag.js` — builds KB context using the TF-IDF index from `src/lib/retrieval.js` (top-K topics + verbatim mentions), filters relations to retrieved topics, and validates proposed deltas (dedupe by id/label, no orphan/self relations, hard caps). +* **Model:** R42 runs on the `fast`/standard Claude tier (Haiku/Sonnet); low latency matters for chat. +* **Quiz-integrity rule:** `src/pages/Testen.jsx` sets `quiz:active:{userId}=true` on start and clears it on every non-quiz phase + unmount, dispatching `respellion:quiz-state`. Never bypass this — letting users ask R42 mid-quiz would break scoring. +* **Graph refinement:** when R42 proposes a `propose_graph_delta`, `rag.js` validates it and a confirmation chip appears inline. **Admin clicks Yes** → `kbStore.applyDelta` writes to PocketBase immediately. **Non-admin clicks Yes** → `kbStore.appendSuggestion` queues a `pending` entry in `kb:suggestions`. +* **Admin approval UI:** `src/components/admin/SuggestionsQueue.jsx` lets admins approve (re-runs `applyDelta`) or reject queued suggestions. +* **kbStore:** `src/lib/kbStore.js` is the single source of truth for chatbot-path KB mutations. It dispatches `respellion:kb-updated` after writes so the D3 graph and queue refresh. -## 10. How to Add New Features -1. **Define Schema:** Add a new PocketBase collection via the Admin UI or the init script (`scripts/setup-pb-collections.mjs`). -2. **Add DB Helpers:** Add async CRUD functions in `src/lib/db.js`. -3. **Build UI:** Create components using `Card`, `Button`, and `framer-motion` for smooth entry (`animate-in fade-in slide-in-from-bottom-4`). -4. **Wire Logic:** Connect to `src/store/AppContext.jsx` if it requires global user context, otherwise manage state locally. -5. **Test:** Run the Docker stack (`docker compose up`) to ensure no build errors exist. +## 10. Admin Panel +`src/pages/Admin/index.jsx` is tabbed: **Sources** (upload + extraction), **Content** (review/refine generated content), **Quizzes**, **Curriculum** (generate/preview/activate a 26-week schedule), **Graph** (D3 knowledge graph + suggestions queue), **Team** (manage members), **Settings** (model overrides, simulation toggle, smoke-test reset). Source upload lives in `src/components/admin/UploadZone.jsx` (`.txt` / `.md`, ≤5 MB). -## 11. Known Gotchas & Decisions -* **No podcast type.** The podcast learning type was deliberately removed. The three supported types are `article`, `slides`, and `infographic`. -* **Week number is computed, not stored.** Do not add a `admin:current_week` setting — the ISO week is always derived from `new Date()`. -* **Caddy, not Nginx.** Older docs/comments may reference Nginx. The reverse proxy is Caddy. Update any references you encounter. -* **AI token budget.** If you see `[Pipeline] AI returned non-JSON response` in the logs, the response was truncated. Increase the topic cap prompt constraint before raising `max_tokens`. -* **PocketBase auto-cancellation is OFF.** `pb.autoCancellation(false)` is set globally in `src/lib/pb.js`. Never re-enable it — it causes abort errors during concurrent fetches in React StrictMode. +## 11. How to Add New Features +1. **Schema:** add a PocketBase collection via the PB Admin UI or a migration in `pb_migrations/` (and mirror it in `scripts/setup-pb-collections.mjs`). +2. **DB helpers:** add async CRUD in `src/lib/db.js`. +3. **UI:** build with `Card`, `Button`, `Tag`, and Framer Motion entry animations. +4. **Logic:** connect to `src/store/AppContext.jsx` for global user/week context; otherwise keep state local. +5. **Verify:** `npm test`, `npm run lint`, `npm run build`. -## 12. 26-Week Versioned Curriculum System -The platform uses a **26-week perpetual curriculum cycle** so every employee covers the knowledge base in focused, thematic weekly blocks. Cycle 1 covers weeks 1-26, Cycle 2 replays the same schedule, and so on. +## 12. Known Gotchas & Decisions +* **Per-user curriculum start.** Week/cycle derive from each user's `curriculum_started_at`, not the calendar. There is no shared current week. New users are gated through `/onboarding` until enrolled. +* **No podcast type.** Three content types only: `article`, `slides`, `infographic`. +* **Three micro-learning types.** `concept_explainer`, `scenario_quiz`, `flashcard_set`. `reflection_prompt` was dropped. +* **No Qdrant / no embeddings.** Retrieval is local TF-IDF (`src/lib/retrieval.js`). +* **Caddy, not Nginx.** The reverse proxy is Caddy (`Caddyfile`). +* **PocketBase auto-cancellation is OFF.** Set in `src/lib/pb.js`; never re-enable. +* **Go through `callLLM`.** Never call the Anthropic proxy directly; you lose retry, schema validation, and telemetry. +* **AI token budget.** Truncation surfaces as `LLMTruncatedError` (`stop_reason: max_tokens`). For extraction, tighten the prompt's topic cap before raising `max_tokens`. -* **Service:** `src/lib/curriculumService.js` — curriculum engine, week resolution, AI generation, version lifecycle. -* **DB Collections:** `curriculum_versions` holds the generated JSON schedules. `topics` includes `theme`, `complexity_weight`, and `difficulty` for generation input. -* **Version Lifecycle:** `draft` -> `active` -> `superseded`. Only one active version exists at a time. -* **Admin UI:** `src/components/admin/CurriculumManager.jsx` — accessed via the "Curriculum" tab. Admins trigger an AI-generated draft, preview the 26-week schedule, and confirm it to activate it globally. -* **Calendar-Driven Weeks:** The current week (1-26) and cycle (1, 2, 3...) are derived via modulo arithmetic from the absolute ISO calendar week in `curriculumService.js`. All employees share the same schedule. -* **Topic Enrichment:** The Curriculum Manager includes a one-off AI enrichment step to assign `theme` and `complexity_weight` to topics missing them before curriculum generation. -* **Progress tracking:** `getYearProgress(userId, isoWeekNumber)` computes completion for the *current 26-week cycle*. -* **Fallback:** If no active curriculum version exists, `getAssignedTopic()` falls back to the legacy hash-based assignment. Do not remove the hash fallback. -* **Deferred Features (V2):** Per-employee staggered start dates, cycle 2+ personalization, and prerequisite DAG sorting are deferred to future iterations to keep the MVP lean. +## 13. 26-Week Per-User Curriculum System +The platform uses a **26-week perpetual curriculum cycle**. Every employee covers the knowledge base in focused, thematic weekly blocks, **starting whenever they enroll**. +* **Service:** `src/lib/curriculumService.js` — week/cycle math, AI schedule generation, version lifecycle, progress. +* **DB:** `curriculum_versions` holds generated JSON schedules; `topics` carry `theme`, `complexity_weight`, `difficulty` as generation input. +* **Version lifecycle:** `draft` → `active` → `superseded`. Only one active version at a time (`CurriculumManager.jsx`). +* **Per-user weeks:** `getPersonalWeekNumber(startedAt)` yields an absolute week counter; `getCurriculumWeek`/`getCurriculumCycle` map it to the 1–26 slot and cycle. Same content each cycle. +* **Topic enrichment:** a one-off AI step assigns `theme`/`complexity_weight`/`difficulty` to topics missing them before generation (`enrichTopicsForCurriculum`, batches of 20). +* **Progress:** `getYearProgress(userId, personalWeekNumber)` computes completion for the current cycle. +* **Fallback:** if no active version exists, `getAssignedTopic()` falls back to deterministic hash-based assignment. Keep the fallback. Good luck! You are building a platform that empowers continuous learning. Keep it fast, keep it beautiful, and keep the user engaged. diff --git a/CLAUDE.md b/CLAUDE.md index 42278fd..fa1ece7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,54 +1,16 @@ # CLAUDE.md -## What you are building +## What this is -A mobile-first progressive web application for employee learning. Employees follow -a perpetual 26-week curriculum built from an internal knowledge base. An AI -assistant called R42 is available on every screen. Admins upload source documents -that are processed into the knowledge base by AI. +The **Respellion Learning Platform** — an internal AI-powered learning app that +keeps employees current with the company's evolving knowledge base. Employees +follow a perpetual **26-week curriculum**, each working through weekly learning +sessions and tests. An AI assistant called **R42** is available on every screen. +Admins upload source documents that Claude extracts into a knowledge graph. -Read the full design before writing any code: -- /docs/handover.md — all decisions made, rationale, constraints -- /docs/architecture.md — system design, data flows, tech stack -- /docs/data-model.md — all PocketBase collections, Qdrant schema, types -- /docs/ingestion-spec.md — ingestion service (build this first) -- /docs/implementation-plan.md — ordered build sequence with acceptance criteria - ---- - -## Absolute constraints - -These rules are non-negotiable and apply to every session: - -1. Never modify any file listed in PROTECTED.md -2. Never modify any file outside /app — the pipeline, Dockerfile, ansible, - and docker-compose files are frozen -3. Never delete files without explicit confirmation -4. Never change package.json scripts that contain 'deploy' or 'build:prod' -5. Ask before acting when scope is unclear — do not infer intent from legacy/ code - ---- - -## Repository structure - -``` -repo/ - ├── CLAUDE.md ← you are here - ├── PROTECTED.md ← files you must never touch - ├── docs/ ← spec files — read, never modify - ├── legacy/ ← old prototype — read-only reference only - └── app/ ← your working directory - ├── frontend/ ← Next.js 14 PWA - └── services/ - ├── ingestion/ ← build first - ├── generation/ ← build second - ├── curriculum/ ← build third - ├── embedding/ ← integrated into ingestion, separate later - ├── chat/ ← R42 - └── progress/ ← gamification -``` - -All work happens inside /app. No exceptions. +This is a **single-page React application** (Vite) backed by **PocketBase**. +There is **no separate backend** — all logic runs in the browser and talks +directly to PocketBase collections and (via a proxy) the Anthropic API. --- @@ -56,98 +18,124 @@ All work happens inside /app. No exceptions. | Layer | Technology | |---|---| -| Frontend | Next.js 14, TypeScript strict, Tailwind CSS, PWA | -| Backend state | PocketBase (binary, not source — do not scaffold from scratch) | -| Vector store | Qdrant via REST client | -| AI generation | Claude Sonnet 4 — model string: claude-sonnet-4-20250514 | -| AI chat (R42) | Claude Haiku 4.5 — model string: claude-haiku-4-5-20251001 | -| Embeddings | OpenAI text-embedding-3-small | -| Services | Node.js, Fastify, TypeScript strict | -| Validation | Zod on all external data (API responses, AI output, PocketBase) | +| Frontend | React 19 + Vite 8, React Router 7 | +| Styling | Vanilla CSS (CSS custom properties) + Tailwind v4 utilities mapped to those variables | +| Animation | Framer Motion | +| Icons | Lucide React | +| Graph viz | D3.js (admin knowledge graph only) | +| Backend / DB / auth | PocketBase (self-hosted, SQLite) | +| AI | Anthropic Claude via a reverse-proxy (Caddy in prod, Vite proxy in dev) | +| Retrieval (RAG) | Local TF-IDF over the knowledge graph — **no Qdrant, no embeddings API** | +| Validation | Zod on all AI tool output | +| Infra | Docker + Caddy; Ansible playbooks under `infra/` | + +**Claude models** (`src/lib/llm.js`, by tier): +- `fast` → `claude-haiku-4-5-20251001` (R42 chat, quiz batches, flashcards) +- `standard` → `claude-sonnet-4-6` (extraction, article/slides/infographic, curriculum) +- `reasoning` → `claude-opus-4-7` + +Admins can override any tier's model string from the Settings tab (persisted in +localStorage as `admin:model:{tier}`). --- -## Stylesheet +## Repository structure -An existing stylesheet lives at /stylesheet.css in the repo root. This is the -authoritative visual style for the application. +``` +repo/ +├── CLAUDE.md ← you are here +├── AI_AGENT.md ← detailed architecture/patterns guide — read this +├── README.md ← quickstart +├── PROTECTED.md ← files you must not modify +├── docs/ ← spec files (describe the system as built) +├── src/ ← THE APPLICATION (React/Vite) +│ ├── pages/ ← route screens (Dashboard, Leren, Testen, Leaderboard, Login, Onboarding, Admin/) +│ ├── components/ ← ui primitives, admin panels, chat (R42), micro_learning +│ ├── lib/ ← services: llm, db, extractionPipeline, learningService, +│ │ microLearningService, testService, curriculumService, retrieval, pb +│ ├── hooks/ ← React hooks +│ └── store/ ← AppContext (global state) +├── pb_migrations/ ← PocketBase schema migrations (JS, applied by the PB binary) +├── scripts/ ← setup-pb-collections.mjs (local collection bootstrap) +├── public/ ← static assets, fonts +├── infra/ ← Ansible deploy playbooks (development / production) +├── Caddyfile, Dockerfile, docker-compose.yml ← deployment (frozen) +└── stylesheet.css ← authoritative visual reference (frozen) +``` -Rules: -- Never modify stylesheet.css -- Import it as a global stylesheet in the Next.js frontend -- Do not override its rules with Tailwind utility classes or inline styles -- When Tailwind and the stylesheet conflict, the stylesheet wins -- If a UI element is not covered by the stylesheet, use Tailwind — but match - the visual language (spacing, colour, type scale) of the existing stylesheet +> **Note on `/app`:** the top-level `app/` directory is **abandoned scaffolding** +> from the original Next.js + Qdrant design that was never shipped. It is not +> wired into `package.json`, Vite, or Docker. **Ignore it.** The real app is `/src`. + +--- + +## Absolute constraints + +1. Never modify any file listed in `PROTECTED.md`. +2. Never modify `stylesheet.css` — it is the authoritative visual reference. Use the + CSS variables in `src/index.css` and the Tailwind classes mapped to them. +3. Treat the deployment files as frozen unless explicitly asked: `Dockerfile`, + `Caddyfile`, `docker-compose.yml`, `infra/`, `.github/workflows/`. +4. Never delete files without explicit confirmation. +5. Never re-enable PocketBase auto-cancellation — `pb.autoCancellation(false)` in + `src/lib/pb.js` is deliberate (see AI_AGENT.md §2). +6. There is **no podcast** learning type. It was removed. Do not re-add it. +7. Ask before acting when scope is unclear. --- ## Code conventions -- TypeScript strict mode everywhere — no `any` types -- All Claude API responses validated through Zod before use -- All PocketBase writes typed against collection schemas in data-model.md -- Qdrant payloads explicitly typed — no untyped objects -- No inline hardcoded API keys — environment variables only -- REST conventions for all service APIs -- Mobile-first CSS — design for 375px width, scale up +- All `src/lib/db.js` functions are `async` — always `await` them. +- All Claude tool output is validated through Zod (`src/lib/llmSchemas.js`) before use. + Never reach `/api/anthropic` directly — go through `callLLM` in `src/lib/llm.js`. +- No client-side API key. The Anthropic key is injected server-side by the proxy. +- Reuse the UI primitives in `src/components/ui/` (`Card`, `Button`, `Tag`, `Input`). +- Mobile-first — the employee app targets 375px width and scales up. +- No hardcoded hex colors; use the design-system CSS variables / Tailwind tokens. --- -## Build order +## Running locally -Follow /docs/implementation-plan.md exactly. -Do not skip phases. Do not build phase N+1 before phase N passes its -acceptance criteria. +```bash +npm install +./pocketbase.exe serve # or the muchobien/pocketbase Docker image +node scripts/setup-pb-collections.mjs # first run only — bootstraps collections +npm run dev # Vite dev server (proxies /api/anthropic) +``` -Current phase: **Phase 1 — Infrastructure + ingestion service** +Set `ANTHROPIC_API_KEY` in the environment so the Vite proxy can inject it +(`vite.config.js`). PocketBase migrations in `pb_migrations/` apply automatically +when the PB binary starts. + +| Command | Purpose | +|---|---| +| `npm run dev` | Vite dev server | +| `npm run build` | Production build to `dist/` | +| `npm test` | Vitest unit tests | +| `npm run lint` | ESLint | --- -## Session discipline +## Documentation map -Each session has a defined scope in implementation-plan.md. -At the start of each session: -1. State which phase and step you are working on -2. Read the relevant spec file(s) -3. Propose a file structure or change plan before writing code -4. Implement after the plan is clear - -At the end of each session: -1. State what was completed -2. State what acceptance criteria have been met -3. State what the next session should start with +- `AI_AGENT.md` — detailed patterns, gotchas, and subsystem guide. **Start here for any real work.** +- `docs/architecture.md` — system design and data flows +- `docs/data-model.md` — every PocketBase collection and field +- `docs/ingestion-spec.md` — upload → knowledge-graph extraction +- `docs/generation-spec.md` — learning content + micro-learning generation +- `docs/curriculum-spec.md` — 26-week per-user curriculum engine +- `docs/r42-spec.md` — the R42 chatbot +- `docs/frontend-spec.md` — screens, routing, onboarding +- `docs/gamification-spec.md` — points, badges, leaderboard +- `micro-learning-spec.md` — micro-learning learner experience --- ## When you are uncertain -Check the spec files in /docs first. -If the spec does not cover it, flag the gap explicitly rather than making -an assumption. Do not look at legacy/ code for implementation guidance — -it is a different stack and will lead you in the wrong direction. - ---- - -## Environment variables - -Each service has its own .env file. Templates are defined in each service spec. -Never commit actual values. Always use .env.example with placeholder values. - -The full set across all services: -``` -ANTHROPIC_API_KEY= -OPENAI_API_KEY= -POCKETBASE_URL= -POCKETBASE_ADMIN_EMAIL= -POCKETBASE_ADMIN_PASSWORD= -QDRANT_URL= -QDRANT_API_KEY= -INGESTION_PORT=3001 -GENERATION_PORT=3002 -CURRICULUM_PORT=3003 -CHAT_PORT=3004 -PROGRESS_PORT=3005 -NEXT_PUBLIC_API_URL= -NEXT_PUBLIC_POCKETBASE_URL= -``` +Check the spec files in `docs/` and `AI_AGENT.md` first. If the spec does not +cover it, read the actual code in `src/` — it is the source of truth. Flag gaps +explicitly rather than assuming. Do not treat the abandoned `/app` scaffolding or +`legacy`-style references in old comments as guidance. diff --git a/PROTECTED.md b/PROTECTED.md index 43c0563..0e3cb82 100644 --- a/PROTECTED.md +++ b/PROTECTED.md @@ -1,34 +1,30 @@ # Protected files — do not modify -These files and directories must never be modified by Claude Code. -They belong to the existing deployment pipeline and are frozen. +These files and directories must never be modified by Claude Code without an +explicit request. They belong to the deployment pipeline and the authoritative +visual reference, and are otherwise frozen. -## Pipeline and CI/CD -.github/ +## Deployment & CI/CD .github/workflows/ -.gitlab-ci.yml (if present) - -## Container and orchestration Dockerfile docker-compose.yml -docker-compose.prod.yml -docker-compose.override.yml (if present) +Caddyfile +Caddyfile.test -## Infrastructure and provisioning -ansible/ -nginx/ +## Infrastructure & provisioning +infra/ (Ansible playbooks: development / production) ## Stylesheet -stylesheet.css (repo root — use as-is, do not modify) +stylesheet.css (repo root — authoritative visual reference, use as-is) -## Environment templates at repo root -.env.example (repo root only — services may have their own) - -## Legacy prototype -legacy/ +## Abandoned scaffolding (do not edit or rely on) +app/ (original Next.js + Qdrant design, never shipped — the real app is /src) --- -The only infrastructure file that will change is Dockerfile, and only in -Phase 8 step 8.5. That change is made manually by the human, not by -Claude Code. +Notes: +- The active application lives entirely in `/src`. Editing the app means editing `/src`. +- PocketBase schema changes go through `pb_migrations/` (and should be mirrored in + `scripts/setup-pb-collections.mjs`) — those are NOT frozen. +- If a task genuinely requires touching a frozen file (e.g. a deliberate deployment + change), confirm with the human first. diff --git a/README.md b/README.md index 0a22f45..e935292 100644 --- a/README.md +++ b/README.md @@ -4,24 +4,28 @@ An internal AI-powered learning platform that keeps Respellion employees up to d ## Features -- **Weekly Learning Station** — Each employee is assigned a topic each week (via deterministic hash of user ID + week number). They choose their preferred format: Article, Slides, or Infographic. Content is generated on-demand by Claude and cached per topic. -- **Weekly Test** — AI-generated quiz based on the knowledge graph. Results are stored and feed the leaderboard. -- **Leaderboard & Gamification** — Points for correct answers, badges for streaks and perfect scores. -- **R42 Chatbot** — An always-available AI assistant (backed by Claude) with access to the full knowledge graph. Can propose graph updates that admins approve or reject. -- **Admin Panel** — Manage the knowledge graph, upload source files, review generated content, refine it with AI, and monitor team progress. +- **Onboarding** — On first login each employee enrolls into the curriculum with one tap. Their 26-week cycle starts then and there. +- **Weekly Learning Station** — Each employee gets a topic for their current curriculum week, drawn from the active 26-week schedule (with a deterministic hash fallback when no curriculum is active). They engage via micro-learnings (concept explainer, scenario quiz, flashcard set) and can also generate long-form formats (Article, Slides, Infographic) on demand. +- **Weekly Test** — A 5-question AI-generated quiz for the week's topic. Results feed the leaderboard. +- **Leaderboard & Gamification** — Points for correct answers, badges for milestones (First Steps, Veteran, Perfectionist). +- **R42 Chatbot** — An always-available AI assistant (Claude) grounded in the knowledge graph via local TF-IDF retrieval. It can propose graph updates that admins approve or reject. +- **Admin Panel** — Manage the knowledge graph, upload source files, review/refine generated content, generate the curriculum, and manage the team. ## Tech Stack | Layer | Technology | |---|---| -| Frontend | React 18 + Vite | -| Styling | Vanilla CSS (custom properties) + Tailwind utility classes | +| Frontend | React 19 + Vite 8, React Router 7 | +| Styling | Vanilla CSS (custom properties) + Tailwind v4 utilities | | Animations | Framer Motion | | Icons | Lucide React | | Graph viz | D3.js (admin knowledge graph only) | -| Backend / DB | PocketBase (self-hosted) | +| Backend / DB / auth | PocketBase (self-hosted, SQLite) | +| Retrieval | Local TF-IDF (no Qdrant, no embeddings API) | | AI | Anthropic Claude (via Caddy reverse proxy) | -| Infra | Docker + Caddy | +| Infra | Docker + Caddy; Ansible playbooks under `infra/` | + +**Claude models (by tier):** `fast` = `claude-haiku-4-5-20251001`, `standard` = `claude-sonnet-4-6`, `reasoning` = `claude-opus-4-7`. ## Getting Started (local dev) @@ -29,14 +33,17 @@ An internal AI-powered learning platform that keeps Respellion employees up to d # 1. Install dependencies npm install -# 2. Start PocketBase (Windows) +# 2. Start PocketBase ./pocketbase.exe serve -# 3. Start the dev server +# 3. Bootstrap collections (first run only) +node scripts/setup-pb-collections.mjs + +# 4. Start the dev server npm run dev ``` -The Vite dev server proxies `/api/anthropic` and `/pb` — see `vite.config.js`. +Set `ANTHROPIC_API_KEY` in your environment — the Vite dev server proxies `/api/anthropic` and injects the key server-side (see `vite.config.js`). PocketBase migrations in `pb_migrations/` apply automatically when the binary starts. ## Deployment (Docker) @@ -46,34 +53,39 @@ docker compose up -d The `Caddyfile` handles: - SPA fallback routing -- `/pb/*` → PocketBase sidecar -- `/api/anthropic/*` → Anthropic API (with server-side API key injection) +- `/api/anthropic/*` → Anthropic API (server-side API key injection) +- `/api/*` and `/_/*` → the PocketBase sidecar ## Key Files | File | Purpose | |---|---| -| `src/lib/learningService.js` | Selective content generation (article / slides / infographic) | -| `src/lib/extractionPipeline.js` | Uploaded file → knowledge graph extraction | -| `src/lib/llm.js` | Core Anthropic LLM wrapper | +| `src/lib/llm.js` | Core Anthropic wrapper (`callLLM`): tiers, retry, schema validation, telemetry | | `src/lib/db.js` | All PocketBase data access | -| `src/store/AppContext.jsx` | Global state; computes ISO week number on load | -| `src/components/admin/UploadZone.jsx` | Drag-and-drop file upload UI | +| `src/lib/extractionPipeline.js` | Uploaded file → knowledge-graph extraction | +| `src/lib/learningService.js` | On-demand content generation (article / slides / infographic) | +| `src/lib/microLearningService.js` | Micro-learning generation (3 types) | +| `src/lib/curriculumService.js` | 26-week per-user curriculum engine | +| `src/lib/retrieval.js` | TF-IDF retrieval for R42 | +| `src/store/AppContext.jsx` | Global state; derives the user's curriculum week from their start date | | `AI_AGENT.md` | Detailed context guide for AI coding agents | ## Content Types -Learning content is generated **on demand per type** and merged into the cached object: +On-demand long-form content (the `content` collection), generated per type and merged into the cached object: | Type | Key in DB | Description | -|---|---|---| +|---|---|---| | Article | `content.article` | Long-form reading | | Slides | `content.slides` | Slide deck with speaker notes | | Infographic | `content.infographic` | Visual summary with stats and steps | > The podcast type was removed. Do not re-add it. +Micro-learnings (the `micro_learnings` collection): `concept_explainer`, `scenario_quiz`, `flashcard_set`. + ## Documentation -- **`AI_AGENT.md`** — Full architectural guide for AI coding agents (patterns, gotchas, decisions). -- **`CHANGELOG.md`** — PocketBase upstream changelog (not application changelog). +- **`CLAUDE.md`** — project overview and constraints for AI coding agents. +- **`AI_AGENT.md`** — full architectural guide (patterns, gotchas, decisions). +- **`docs/`** — subsystem specs (architecture, data model, ingestion, generation, curriculum, R42, frontend, gamification). diff --git a/docs/architecture.md b/docs/architecture.md index 51e0608..43f96e3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,279 +1,179 @@ -# Architecture: employee learning platform +# Architecture: Respellion Learning Platform ## Overview -A mobile-first progressive web application that provides employees with a structured knowledge library, a 26-week perpetual learning curriculum, and an AI-powered assistant (R42). The knowledge base is the single source of truth for all content, micro learnings, curriculum scheduling, and chat retrieval. +A mobile-first single-page web application that gives employees a structured +knowledge library, a 26-week per-user learning curriculum, and an AI assistant +(R42). The **knowledge graph** stored in PocketBase is the single source of truth +for all content, micro-learnings, curriculum scheduling, and chat retrieval. + +Unlike the original design (a Next.js multi-service system with Qdrant), the +shipped platform is a **React/Vite SPA talking directly to PocketBase**, with the +Anthropic API reached through a thin reverse proxy. All application logic — AI +orchestration, retrieval, generation, scoring — runs in the browser. + +``` +Browser (React SPA) + ├── PocketBase SDK ───────────────► PocketBase (SQLite): all structured data + auth + files + └── callLLM ──► /api/anthropic ──► Anthropic API (key injected by the proxy) + (Caddy in prod / Vite proxy in dev) +``` --- -## System domains - -### Admin app -Browser-based interface for content administrators. - -Responsibilities: -- Upload source documents (PDF, MD, TXT) -- Review and approve AI-generated Theme batches -- Edit and finetune AI-generated curriculum -- Confirm curriculum regeneration after KB updates -- Monitor ingestion and generation job status - -### Employee app -Mobile-first PWA accessible on all devices. - -Responsibilities: -- Weekly session delivery (Theme + Topics + micro learning type selection) -- Knowledge library (browse all published Topics) -- Gamification profile (heatmap, badges, streak, leaderboard) -- R42 chatbot (available on every screen) - -### Backend services -Six discrete services, each with a single responsibility. - -| Service | Responsibility | -|---|---| -| Ingestion service | Document upload → chunk → extract KB structure | -| Generation service | Topics → 10 micro learning types (structured JSON) | -| Curriculum service | KB graph → 26-week schedule, versioning, regeneration | -| Embedding service | Chunks + topic summaries → Qdrant | -| Chat service (R42) | Query → vector retrieval → grounded response | -| Progress service | Completions → XP → badges → streak | - ---- - -## Deployment topology +## Runtime topology ``` -repo/ - ├── .github/workflows/ ← pipeline (frozen) - ├── docker-compose.yml ← infrastructure (frozen) - ├── Dockerfile ← updated once to point at /app - ├── ansible/ ← provisioning (frozen) - ├── legacy/ ← original prototype (read-only reference) - └── app/ - ├── frontend/ ← Next.js PWA (admin + employee) - └── services/ - ├── ingestion/ - ├── generation/ - ├── curriculum/ - ├── embedding/ - ├── chat/ - └── progress/ +┌─────────────────────────────┐ ┌──────────────────────────┐ +│ Caddy container │ │ PocketBase container │ +│ - serves built SPA (/srv) │ /api/*│ - SQLite data │ +│ - /api/anthropic/* → Claude │───────►│ - auth (team_members) │ +│ - /api/*, /_/* → PocketBase │ /_/* │ - file storage │ +│ - injects ANTHROPIC_API_KEY │ │ - migrations (pb_migrations) +└─────────────────────────────┘ └──────────────────────────┘ ``` +In local dev, `vite.config.js` replaces Caddy: it proxies `/api/anthropic` to +`https://api.anthropic.com` and injects `ANTHROPIC_API_KEY`. PocketBase runs +directly (`./pocketbase.exe serve`). + --- ## Tech stack | Layer | Technology | Rationale | |---|---|---| -| Frontend | Next.js 14, TypeScript, Tailwind CSS | PWA support, single codebase for admin + employee | -| Backend state | PocketBase | Auth, file storage, admin UI, SQLite — no infra overhead | -| Vector store | Qdrant (Docker) | RAG retrieval, runs as single container | -| AI generation | Claude Sonnet 4 via Anthropic API | Structured JSON output, long-form drafting, graph reasoning | -| AI chat (R42) | Claude Haiku 4.5 via Anthropic API | Low latency, cost-effective, grounded by retrieval layer | -| Embeddings | OpenAI text-embedding-3-small | Cost-effective, high quality at this scale | -| Auth | PocketBase built-in | Role-based: admin / employee | +| Frontend | React 19 + Vite 8, React Router 7 | Fast SPA, single codebase for admin + employee | +| Styling | CSS variables + Tailwind v4 | Premium design system; Tailwind mapped to variables | +| Backend state | PocketBase (SQLite) | Auth, file storage, admin UI — no infra overhead | +| Retrieval | Local TF-IDF (`src/lib/retrieval.js`) | Grounds R42 with zero external vector infra | +| AI | Claude via Anthropic API (proxied) | Structured tool output, long-form drafting, chat | +| Auth | PocketBase `team_members` + PIN | Lightweight internal auth, role = admin / (employee) | +| Infra | Docker + Caddy, Ansible (`infra/`) | Containerized deploy to dev/prod | + +There is **no Qdrant, no OpenAI/embeddings service, and no separate Node backend.** --- ## AI model responsibilities -| Task | Model | -|---|---| -| Document → KB structure extraction | Claude Sonnet 4 | -| Topic body drafting | Claude Sonnet 4 | -| Micro learning generation (all 10 types) | Claude Sonnet 4 | -| Curriculum generation + versioning | Claude Sonnet 4 | -| R42 chat responses | Claude Haiku 4.5 | -| Embeddings | text-embedding-3-small | +`callLLM` (`src/lib/llm.js`) selects a Claude model by **tier**: + +| Tier | Model | Used for | +|---|---|---| +| `fast` | `claude-haiku-4-5-20251001` | R42 chat, weekly quiz batch, flashcard sets | +| `standard` | `claude-sonnet-4-6` | KB extraction, article/slides/infographic, micro-learnings, curriculum generation | +| `reasoning` | `claude-opus-4-7` | reserved for heavier reasoning tasks | + +Admins can override the model string per tier from the Settings tab. --- -## Document ingestion pipeline +## Knowledge ingestion pipeline ``` -Admin uploads file (PDF / MD / TXT) - ↓ -Format detection → text extraction - MD: split on headings → preserve hierarchy - PDF: pdfplumber → page + paragraph detection - TXT: sliding window chunking with overlap - ↓ -Chunk cleaning (strip headers/footers/artefacts) - ↓ -Claude Sonnet 4 reads chunks → extracts: - - candidate Themes - - candidate Topics per Theme - - Topic→Topic relationships (related, prerequisite, contrast) - - key terms for glossary - ↓ -Draft KB written to PocketBase (status: draft) - ↓ -Embedding service: embed source chunks → write to Qdrant - ↓ -Admin reviews Theme batch → approves / edits / rejects - ↓ -On approval: Topics published, micro learning generation queued - ↓ -Curriculum regeneration notification queued for admin +Admin uploads .txt / .md (≤5 MB) in the Sources tab + ↓ +extractionPipeline.js chunks the text (~8000 chars, 800 overlap) + ↓ +Per chunk: callLLM (standard tier) with the emit_knowledge_graph tool + → topics (id, label, type, description, learning_relevance) + → relations (source, target, type) + ↓ +Results merged into the `topics` and `relations` collections + (topic id de-dup; relevance_locked topics keep their relevance) + ↓ +Source status tracked in `sources` (processing → completed / failed / cancelled) ``` -Note: embeddings are generated from **source chunks**, not only from AI-generated topic summaries. R42 retrieves from grounded source material. +There are no embeddings. Retrieval for R42 is computed on the fly with TF-IDF +over `topics` (`label + description`). -MD source files are the preferred format for admins — heading structure maps directly to Theme → Topic hierarchy and improves extraction quality. +See `docs/ingestion-spec.md`. --- -## Curriculum lifecycle +## Content generation + +Two generators, both via `callLLM` with forced tool use and Zod-validated output: + +- **Long-form content** (`learningService.js`) → `content` collection. Three types + generated on demand and shallow-merged: `article`, `slides`, `infographic`. +- **Micro-learnings** (`microLearningService.js`) → `micro_learnings` collection. + Three types: `concept_explainer`, `scenario_quiz`, `flashcard_set`. + +See `docs/generation-spec.md`. + +--- + +## Curriculum lifecycle (per-user) ### Generation -Input: all published Themes, Topics, relationship graph, complexity weights -Process: cluster by Theme → sequence pedagogically (prerequisites first, complexity gradient) → distribute across 26 weeks → ensure full KB coverage -Output: versioned 26-week draft schedule +Input: published topics grouped by `theme`, ordered by `complexity_weight`. +`curriculumService.generateCurriculumDraft()` asks Claude for a 26-week schedule +via `emit_curriculum_schedule`, validates it, and stores a `curriculum_versions` +row (`status='draft'`). Admin previews and confirms → `active`. Only one active +version exists; the prior active becomes `superseded`. -### Perpetual cycling -The curriculum runs continuously. After week 26, the employee begins cycle 2 on the latest curriculum version. +### Per-user cycling +The curriculum is **not** anchored to the calendar. Each employee enrolls when +they choose (first-login onboarding), which sets `team_members.curriculum_started_at`. +Their position is derived: -Second and subsequent cycles are not identical to cycle 1: -- Theme sequence is varied -- Recommended micro learning types surface types the employee has not yet used -- Topics with low engagement in prior cycles receive increased coverage +``` +personalWeek = floor(daysSinceStart / 7) + 1 // absolute counter, ≥1 +curriculumWeek = ((personalWeek - 1) % 26) + 1 // 1..26 slot +cycle = floor((personalWeek - 1) / 26) + 1 // 1, 2, 3, ... +``` -### Versioning rules +After week 26 the cycle restarts at week 1 with the **same** content. -| Event | Action | -|---|---| -| New source doc published to KB | Regenerate curriculum from week N+1 for all active employees | -| Topic body edited | Micro learnings regenerated; curriculum unaffected | -| Theme batch approved | Regeneration queued; admin confirms before it applies | - -Completed weeks are immutable. Regeneration only affects future unstarted weeks. - -### Admin regeneration flow -Admin receives notification: "N new topics added. Regenerate curriculum? This will update unstarted weeks for all active employees." -Admin can preview the proposed new schedule before confirming. +See `docs/curriculum-spec.md`. --- ## Weekly session flow (employee) ``` -Week N opens +Enroll (first login) → curriculum_started_at set ↓ -Employee sees assigned Theme + Topics for the week +Dashboard shows current cycle / week / assigned topic ↓ -Per Topic: employee selects micro learning type - (all published types for that topic are available) +Learning Station: complete ≥1 micro-learning for the week's topic(s) ↓ -Employee completes one or more types per topic +Weekly Test: 5 AI-generated questions → +2 points per correct answer ↓ -Completion recorded → XP awarded → badges evaluated - ↓ -Progress visible on public leaderboard and activity feed +Leaderboard updates; badges evaluated at render time ``` -Sessions support multiple micro learning types per topic in a single session. - ---- - -## Micro learning types - -All 10 types are generated by Claude Sonnet 4 as structured JSON, stored in PocketBase, and rendered by the frontend. One or more types may be published per topic. - -| # | Type | Format | -|---|---|---| -| 1 | Concept explainer | 2–3 paragraphs + example | -| 2 | Scenario quiz | situation + 3–4 MCQ options + explained answers | -| 3 | Common misconceptions | 3–5 false beliefs + corrections | -| 4 | Step-by-step how-to | numbered procedure | -| 5 | Comparison card | side-by-side on 4–6 dimensions | -| 6 | Reflection prompt | open question + model answer | -| 7 | Spaced repetition flashcards | 5–10 Q&A pairs | -| 8 | Case study mini-analysis | 150–200 word scenario + guiding questions | -| 9 | Glossary anchor | term + definition + correct use + misuse | -| 10 | Myth vs. evidence | false claim + evidence-based rebuttal | - --- ## R42 — chat service design -R42 is a functional KB-grounded assistant available on every screen in the employee app. +R42 is a KB-grounded assistant on every screen (`src/components/chat/`). -Behaviour: -- Stateless per session (no memory between conversations) -- Retrieves relevant chunks from Qdrant using the employee's query -- Knows the employee's current curriculum week → retrieval is context-weighted -- Cites source topic in every response ("based on the **Holacratic roles** topic") -- Explicitly refuses to answer outside KB scope rather than hallucinating -- Scope: internal KB only +- Persists the conversation per user in `localStorage` (`chat:thread:{userId}`, cap 50; ~12 turns sent to the API). +- Builds context with the TF-IDF index (top-K topics + verbatim mentions), injects related relations and limited deep content. +- Can propose a `propose_graph_delta` (≤3 topics, ≤5 relations). Admins apply directly; non-admins queue a suggestion for admin approval. +- Hidden during quizzes (the `quiz:active:{userId}` integrity rule). -Implementation: -- Employee query → embed → Qdrant nearest-neighbour retrieval → top-K chunks -- Chunks + employee context injected into Haiku 4.5 prompt -- Response streamed to frontend - -UI: floating button bottom-right, unobtrusive on mobile. +See `docs/r42-spec.md`. --- -## Gamification system +## Gamification -Inspired by the visual language of GitHub, Stack Overflow, and Duolingo. Mechanics use developer-native terminology. +- Points: +2 per correct quiz answer, stored in `leaderboard`. +- Badges (render-time): First Steps (1 test), Veteran (5 tests), Perfectionist (100% score). +- Leaderboard excludes admins. -### XP unit: commits -Every completed topic earns commits. Quantity varies by micro learning type complexity. - -### Levels -`Intern → Junior → Medior → Senior → Staff → Principal` -Based on cumulative commits across all cycles. - -### Streak -Counted in consecutive weeks, not days. Resets if a week is skipped entirely. - -### Activity heatmap -GitHub-style contribution graph spanning the full 26-week cycle. Cell darkness = number of types completed that week. - -### Badges - -| Tier | Condition | -|---|---| -| Bronze | Complete any session | -| Silver | 5 sessions completed, 5 different types used | -| Gold | 13 sessions without skipping a week | -| Legendary | All 26 sessions, all 10 types used at least once | - -Named content badges (examples): -- `governance nerd` — all holacratic structure topics completed -- `process architect` — all internal process topics completed -- `deep reader` — case study type used 5+ times - -### Milestone cards (public) -At weeks 13 and 26, a public card is posted to the shared activity feed: - -``` -🚀 [Name] shipped the full curriculum - 26 weeks · [N] commits · [badges] - Longest streak: [N] weeks -``` - -Language: shipping vocabulary, not school vocabulary. - -### Leaderboard -Not ranked 1–N by score. Displays multiple dimensions: - -| Employee | Commits | Streak | Types used | Badges | -|---|---|---|---|---| - -Multiple paths to visibility. No single metric determines standing. +See `docs/gamification-spec.md`. --- ## Security and privacy -- Auth: PocketBase role-based (admin / employee) -- Gamification data (commits, badges, streak) is public to all employees -- Session completion data (which topic, which type, when) is public -- Source documents are admin-only -- No PII beyond display name stored in gamification context -- R42 is stateless — no chat history persisted +- Auth: PocketBase `team_members` with PIN; role `admin` unlocks the Admin panel. +- The Anthropic API key never reaches the client — it is injected by Caddy / the Vite proxy. +- R42 conversations are stored client-side per user; no server-side chat history. +- Source documents and the knowledge graph are managed by admins. diff --git a/docs/curriculum-spec.md b/docs/curriculum-spec.md index a0bf66d..4a05d1c 100644 --- a/docs/curriculum-spec.md +++ b/docs/curriculum-spec.md @@ -1,407 +1,86 @@ -# Curriculum service spec +# Curriculum spec: 26-week per-user cycle -## Responsibility - -Generates a versioned 26-week learning schedule from the published knowledge -base. Manages perpetual cycling, version transitions, and employee curriculum -state. Handles regeneration when the KB changes. +The curriculum sequences the knowledge base into a 26-week schedule. Every +employee runs their **own** cycle, starting when they enroll. Implemented in +`src/lib/curriculumService.js`; admin UI in +`src/components/admin/CurriculumManager.jsx`. --- -## Service location +## Data -``` -app/services/curriculum/ - ├── src/ - │ ├── index.ts entry point, Fastify server - │ ├── routes/ - │ │ ├── curriculum.ts POST /generate, GET /current, GET /preview - │ │ └── employee.ts GET /state/:userId, POST /advance/:userId - │ ├── generator/ - │ │ ├── build.ts KB graph → 26-week schedule (AI call) - │ │ ├── sequence.ts prerequisite + complexity ordering - │ │ └── cycle.ts cycle 2+ variation logic - │ ├── versioning/ - │ │ ├── apply.ts apply new version to active employees - │ │ └── freeze.ts protect completed weeks - │ └── lib/ - │ ├── pocketbase.ts - │ └── anthropic.ts - ├── package.json - ├── tsconfig.json - └── .env.example -``` +- **`curriculum_versions`** — generated schedules. Lifecycle `draft → active → + superseded`; exactly one `active`. The `schedule` JSON is an array of 26 week + objects: `{ week_number (1–26), theme, topic_ids[], estimated_duration (15–45), + week_rationale }`. `coverage_stats` records theme/topic coverage. +- **`topics`** — supply `theme`, `complexity_weight` (1–5), and `difficulty` as + generation input. --- -## API surface +## Topic enrichment (prerequisite) -### POST /generate - -Triggers curriculum generation from current published KB. -Called by admin app after confirming regeneration. - -Request: -```json -{ - "triggeredBy": "string", - "reason": "new_topics" | "manual" -} -``` - -Response (202 Accepted): -```json -{ - "jobId": "string", - "status": "queued" -} -``` +Generation needs themed, weighted topics. `enrichTopicsForCurriculum()` finds +topics missing a `theme` (excluding `type='fact'` and `learning_relevance='exclude'`) +and, in batches of 20, calls Claude with `emit_topic_enrichment` to assign +`theme`, `complexity_weight`, and `difficulty`. Triggered from the Curriculum tab. --- -### GET /preview +## Generation -Returns proposed new curriculum before admin confirms. -Called by admin app to show preview before regeneration is applied. +`generateCurriculumDraft(reason)`: +1. Group learning topics by `theme` (`buildThemeTopicMap`), sorted by + `complexity_weight` ascending. +2. Build a prompt describing themes and their topic ids. If there are more than 26 + themes, the model is instructed to **merge** closely related themes. +3. `callLLM` (standard tier, `maxTokens: 8192`, temp 0) with forced + `emit_curriculum_schedule`. Up to 2 attempts; on validation failure the errors + are fed back into the retry prompt. +4. Validate (`validateSchedule`): exactly 26 weeks, correct `week_number` sequence, + durations 15–45, every `topic_id` exists, ≥1 topic per week. Unscheduled themes + are warnings, not hard errors. +5. Supersede any existing draft and store the new `draft` version with coverage stats. -Response: -```json -{ - "version": 3, - "weeks": [ - { - "weekNumber": 1, - "theme": { "id": "string", "title": "string" }, - "topics": [ - { "id": "string", "title": "string", "complexityWeight": 2 } - ], - "estimatedDurationMinutes": 25 - } - ], - "coverageStats": { - "themesTotal": 8, - "themesCovered": 8, - "topicsTotal": 42, - "topicsCovered": 42 - } -} -``` +`confirmVersion(versionId, adminUserId)` activates a draft (and supersedes the old +active version); `rejectVersion` discards a draft. --- -### GET /current +## Per-user scheduling (the cycle) -Returns the currently active curriculum version with all week slots. +The cycle is **detached from the calendar**. Enrollment sets +`team_members.curriculum_started_at` (see `docs/frontend-spec.md` for onboarding). +`AppContext` derives the user's position: + +```js +getPersonalWeekNumber(startedAt) // floor(daysSinceStart / 7) + 1, ≥1 (0 if not enrolled) +getCurriculumWeek(personalWeek) // ((n - 1) % 26) + 1 → 1..26 slot +getCurriculumCycle(personalWeek) // floor((n - 1) / 26) + 1 → 1, 2, 3, ... +``` + +- Week 1 begins the day the employee enrolls. +- After week 26 the cycle restarts at week 1 with the **same** content. +- `state.weekNumber` (the absolute counter) is `0` until the user enrolls; pages are + gated behind onboarding so they never render with week 0. --- -### GET /state/:userId +## Content & progress for a week -Returns an employee's current curriculum state. - -Response: -```json -{ - "userId": "string", - "currentCycle": 1, - "currentWeek": 7, - "startDate": "2026-01-15T00:00:00Z", - "activeVersionId": "string", - "nextSessionTheme": { "id": "string", "title": "string" }, - "nextSessionTopics": [] -} -``` +- `getCurrentWeekContent(personalWeek)` reads the active version's schedule, maps the + 26-week slot to its `topic_ids`, and returns `{ cycle, weekNumber, theme, topics, + estimatedDuration, rationale }`. +- `getAssignedTopic(userId, week)` returns the week's primary topic, falling back to + a deterministic hash of `userId:week` when no curriculum is active. **Keep the + fallback.** +- `getYearProgress(userId, personalWeek)` computes completion for the current cycle. --- -### POST /advance/:userId +## Notes -Called by progress service when an employee completes a week. -Increments currentWeek, handles cycle transition at week 26. - -Request: -```json -{ - "completedWeek": 7 -} -``` - ---- - -## Curriculum generation - -### Input - -All published Themes and Topics retrieved from PocketBase: - -```typescript -type KBSnapshot = { - themes: { - id: string - title: string - description: string - topics: { - id: string - title: string - complexityWeight: number // 1–5 - difficulty: string - prerequisiteTopics: string[] // topic IDs - relatedTopics: string[] - contrastTopics: string[] - }[] - }[] -} -``` - ---- - -### Pre-processing: sequence topics within themes - -Before the AI call, the service resolves topic ordering within each Theme -using a topological sort on prerequisite relationships. - -``` -For each Theme: - Build directed graph: prerequisite_topics edges - Topological sort → ordered topic list - If cycle detected (should not occur but handle): log warning, fall back to - complexity_weight ascending order -``` - -This pre-processing means the AI does not need to reason about prerequisites — -it receives already-ordered topic lists and focuses on Theme sequencing. - ---- - -### AI call: Theme sequencing across 26 weeks - -System prompt: -``` -You are a curriculum designer. Your task is to distribute a set of learning -Themes across 26 weekly sessions to create an effective learning journey. - -Output ONLY valid JSON matching the schema provided. No preamble, no -explanation, no markdown fences. - -Rules: -- Every Theme must appear at least once across 26 weeks -- Themes with more Topics (higher topic count) may span multiple weeks or - appear in multiple cycles within the 26 weeks -- Sequence Themes so foundational concepts precede dependent ones -- Distribute complexity progressively: introductory Themes early, advanced - Themes in the second half -- If total Topics across all Themes exceeds what 26 weeks can cover in depth, - prioritise breadth in cycle 1 — every Theme covered, key Topics per Theme -- Assign an estimated duration in minutes per week (15–45 minutes per session) -- Return exactly 26 week slots -``` - -User prompt: -``` -Knowledge base snapshot: -{KBSnapshot as JSON} - -Generate a 26-week curriculum schedule. -``` - -Output schema: -```typescript -type CurriculumDraft = { - weeks: { - weekNumber: number // 1–26 - themeId: string - topicIds: string[] // ordered subset of theme's topics - estimatedDurationMinutes: number - rationale: string // one sentence — shown to admin in preview - }[] -} -``` - -AI call configuration: -```typescript -{ - model: 'claude-sonnet-4-20250514', - max_tokens: 4000, - temperature: 0 -} -``` - -Validation: Zod schema on output. Check all themeIds and topicIds exist in -the KB snapshot before writing. Reject and retry once on validation failure. - ---- - -### Write to PocketBase - -``` -Create curriculum_versions record { - version: latest + 1, - status: 'draft', - generated_at: now, - generation_notes: reason -} - -For each week in CurriculumDraft: - Create curriculum_weeks record { - curriculum_version: versionId, - week_number: weekNumber, - theme: themeId, - topics: topicIds, - topic_order: [0, 1, 2, ...], - estimated_duration_minutes: value, - admin_notes: '' - } - -Set curriculum_versions.status → 'draft' -Notify admin: preview available at GET /preview -``` - -Draft version is not applied until admin confirms via POST /generate confirm. - ---- - -## Versioning and regeneration - -### Applying a new version - -When admin confirms, `apply.ts` runs: - -``` -Get all employees from employee_curriculum_state - -For each employee: - frozenWeek = employee.current_week - - Update employee_curriculum_state: - active_version = new version ID - - Note: completed weeks are protected by current_week value - The frontend only renders weeks >= current_week from active_version - Weeks < current_week are rendered from session_completions history - (immutable records — not from curriculum_weeks) - -Set old curriculum_versions.status → 'superseded' -Set new curriculum_versions.status → 'active' -``` - -Completed weeks are never stored against a curriculum version — they live -in session_completions. The version only determines future week content. - ---- - -## Perpetual cycling - -### Week 26 completion → cycle transition - -When progress service calls POST /advance/:userId with completedWeek: 26: - -``` -employee.currentCycle += 1 -employee.currentWeek = 1 -employee.startDate = now -employee.activeVersion = current active version - -Generate cycle variant (see below) -``` - -### Cycle variant generation - -Cycle 2+ is not identical to cycle 1. The AI call receives additional context: - -Additional fields in user prompt for cycle 2+: -```json -{ - "cycleNumber": 2, - "employeeHistory": { - "typesUsed": ["concept_explainer", "scenario_quiz", "how_to"], - "typesNotUsed": ["case_study", "myth_vs_evidence", "comparison_card"], - "lowEngagementTopics": ["topic-id-1", "topic-id-2"] - } -} -``` - -Additional rules added to system prompt for cycle 2+: -``` -- Vary the Theme sequence from the previous cycle -- Topics identified as low engagement should appear earlier in this cycle -- The rationale field should note what is different from cycle 1 -``` - -Low engagement is determined by: topics where the employee completed only -one micro learning type (minimum engagement). Retrieved from session_completions -by progress service and passed to curriculum service on cycle transition. - ---- - -## Admin curriculum editor - -The curriculum editor in the admin app (built in frontend phase) calls: -- GET /preview to display the proposed schedule -- PATCH /weeks/:weekId to update theme or topic assignment -- POST /confirm to apply the version - -The PATCH route allows admin to: -- Reassign a Theme to a different week (swap two weeks) -- Add or remove Topics from a week's topic list -- Edit admin_notes per week - -Changes made via PATCH update the draft curriculum_weeks records before -the version is confirmed and applied. - ---- - -## Environment variables - -``` -ANTHROPIC_API_KEY= -POCKETBASE_URL= -POCKETBASE_ADMIN_EMAIL= -POCKETBASE_ADMIN_PASSWORD= -CURRICULUM_PORT=3003 -``` - ---- - -## Dependencies - -```json -{ - "dependencies": { - "fastify": "^4", - "@anthropic-ai/sdk": "^0.24", - "pocketbase": "^0.21", - "zod": "^3", - "uuid": "^9" - } -} -``` - ---- - -## TypeScript strict mode requirements - -- No `any` types -- KBSnapshot typed explicitly — validated against PocketBase response -- CurriculumDraft validated through Zod before any PocketBase writes -- Topological sort implemented with explicit typed graph structure - ---- - -## What this service does NOT do - -- Does not generate micro learnings → generation service -- Does not record completions → progress service -- Does not serve KB content → frontend reads PocketBase directly -- Does not handle auth → PocketBase + frontend - ---- - -## Testing checkpoints - -1. Generate curriculum from a KB with 5+ themes → confirm 26 weeks produced -2. Confirm all themes appear at least once -3. Confirm topic order within a week respects prerequisites -4. Add a new theme to KB → trigger regeneration → confirm employee at week 5 - sees weeks 1–5 unchanged, weeks 6–26 updated -5. Advance employee through week 26 → confirm cycle 2 starts with varied sequence -6. Admin edits week 3 theme → confirm patch updates draft before confirmation +- There is no shared "current week" and no `admin:current_week` setting. +- Regeneration produces a new version; activating it changes future weeks for all + users. Completion history (`micro_learning_completions`) is append-only and never + rewritten. diff --git a/docs/data-model.md b/docs/data-model.md index 2b34d3b..ae0812f 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -1,397 +1,252 @@ -# Data model: employee learning platform +# Data model: Respellion Learning Platform ## Overview -Two storage systems: -- **PocketBase** — all structured relational data (SQLite under the hood) -- **Qdrant** — all vector embeddings for RAG retrieval +All structured data lives in **PocketBase** (SQLite). There is **no vector store** +— retrieval is computed at runtime with a local TF-IDF index over `topics` +(`src/lib/retrieval.js`). + +Schema is defined by JS migrations in `pb_migrations/` (applied automatically by +the PocketBase binary) and mirrored for local bootstrap in +`scripts/setup-pb-collections.mjs`. The data-access layer is `src/lib/db.js`. + +All collections use PocketBase's auto `id`, plus `created` / `updated` autodate +fields unless noted otherwise. --- ## PocketBase collections -### `source_documents` -Uploaded source files. Parent of all generated KB content. - -| Field | Type | Notes | -|---|---|---| -| id | string | PocketBase auto | -| filename | string | original filename | -| file | file | PocketBase file storage | -| format | select | `pdf` `md` `txt` | -| status | select | `processing` `processed` `failed` | -| ingested_at | datetime | | -| chunk_count | number | total chunks extracted | -| created_by | relation → `users` | admin who uploaded | - ---- - -### `themes` -Top-level content groupings. One Theme = one weekly session. - -| Field | Type | Notes | -|---|---|---| -| id | string | PocketBase auto | -| title | string | | -| description | text | AI drafted, admin editable | -| status | select | `draft` `published` | -| source_documents | relation[] → `source_documents` | which docs contributed | -| approved_by | relation → `users` | admin who approved batch | -| approved_at | datetime | | -| created_at | datetime | | -| updated_at | datetime | | - ---- - ### `topics` -Atomic knowledge units. Always belong to a Theme. +Knowledge graph nodes. Created during ingestion, enriched for curriculum. | Field | Type | Notes | |---|---|---| -| id | string | PocketBase auto | -| theme | relation → `themes` | parent theme | -| title | string | | -| body | text (rich) | AI drafted, admin editable | -| difficulty | select | `introductory` `intermediate` `advanced` | -| complexity_weight | number | 1–5, used by curriculum generator | -| status | select | `draft` `published` | -| related_topics | relation[] → `topics` | lateral relationships | -| prerequisite_topics | relation[] → `topics` | must-complete-first | -| contrast_topics | relation[] → `topics` | deliberate opposites | -| key_terms | json | string[] — feeds glossary | -| qdrant_chunk_ids | json | string[] — references to embedded chunks | -| created_at | datetime | | -| updated_at | datetime | | +| id | text | kebab-case slug (e.g. `holacratic-roles`) | +| label | text | display name | +| type | text | `concept` · `role` · `process` (ingestion); `fact` is excluded from learning | +| description | text | 1–2 sentence summary | +| learning_relevance | text | `core` · `standard` · `peripheral` · `exclude` | +| relevance_locked | bool | if true, re-ingestion will not overwrite `learning_relevance` | +| theme | text | subject grouping (used by curriculum generation) | +| complexity_weight | number | 1–5 (curriculum ordering) | +| difficulty | text | `introductory` · `intermediate` · `advanced` | -Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale. +Topics with `type='fact'` or `learning_relevance='exclude'` are filtered out of +learning, micro-learning, curriculum, and test selection. + +--- + +### `relations` +Knowledge graph edges between topics. + +| Field | Type | Notes | +|---|---|---| +| source | text | topic id | +| target | text | topic id | +| type | text | `related_to` · `depends_on` · `part_of` · `executed_by` | + +Edges are de-duplicated on the `(source, target, type)` tuple. + +--- + +### `content` +On-demand long-form learning content, one record per topic. + +| Field | Type | Notes | +|---|---|---| +| topic_id | text | topic this content belongs to | +| data | json | merged object — only generated types are present | + +`data` shape (each key generated independently and shallow-merged): + +```json +{ + "article": { "title", "intro", "sections": [{ "heading", "body" }], "keyTakeaways": [] }, + "slides": [ { "title", "bullets": [], "speakerNote" } ], + "infographic": { "headline", "tagline", "stats": [{ "value", "label", "icon" }], + "steps": [{ "number", "title", "description", "icon" }], "quote", "colorTheme" } +} +``` + +> There is no `podcast` key. The podcast type was removed. --- ### `micro_learnings` -Generated content artifacts. One record per topic per type. +Generated micro-learning artifacts. One record per topic per type. | Field | Type | Notes | |---|---|---| -| id | string | PocketBase auto | -| topic | relation → `topics` | | -| type | select | see type enum below | -| content | json | structured output — schema varies per type | -| status | select | `queued` `generated` `published` `rejected` | -| generation_model | string | model version used | -| generated_at | datetime | | -| published_at | datetime | | -| updated_at | datetime | | +| topic_id | relation → `topics` | cascade delete | +| type | select | `concept_explainer` · `scenario_quiz` · `flashcard_set` | +| content | json | structured output, schema varies per type | +| status | select | `draft` · `published` (only `published` is visible to employees) | -**Type enum:** -`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence` - -**Content JSON schemas per type:** +**Content JSON per type:** ```json // concept_explainer -{ - "paragraphs": ["string", "string"], - "example": "string" -} +{ "sections": [ { "title": "string", "content": "string (HTML:

,