learning-platform/docs/architecture.md

# Architecture: employee learning platform

## Overview

A mobile-first progressive web application that provides employees with a structured knowledge library, a 26-week perpetual learning curriculum, and an AI-powered assistant (R42). The knowledge base is the single source of truth for all content, micro learnings, curriculum scheduling, and chat retrieval.

---

## System domains

### Admin app
Browser-based interface for content administrators.

Responsibilities:
- Upload source documents (PDF, MD, TXT)
- Review and approve AI-generated Theme batches
- Edit and finetune AI-generated curriculum
- Confirm curriculum regeneration after KB updates
- Monitor ingestion and generation job status

### Employee app
Mobile-first PWA accessible on all devices.

Responsibilities:
- Weekly session delivery (Theme + Topics + micro learning type selection)
- Knowledge library (browse all published Topics)
- Gamification profile (heatmap, badges, streak, leaderboard)
- R42 chatbot (available on every screen)

### Backend services
Six discrete services, each with a single responsibility.

| Service | Responsibility |
|---|---|
| Ingestion service | Document upload → chunk → extract KB structure |
| Generation service | Topics → 10 micro learning types (structured JSON) |
| Curriculum service | KB graph → 26-week schedule, versioning, regeneration |
| Embedding service | Chunks + topic summaries → Qdrant |
| Chat service (R42) | Query → vector retrieval → grounded response |
| Progress service | Completions → XP → badges → streak |

---

## Deployment topology

```
repo/
  ├── .github/workflows/    ← pipeline (frozen)
  ├── docker-compose.yml    ← infrastructure (frozen)
  ├── Dockerfile            ← updated once to point at /app
  ├── ansible/              ← provisioning (frozen)
  ├── legacy/               ← original prototype (read-only reference)
  └── app/
        ├── frontend/       ← Next.js PWA (admin + employee)
        └── services/
              ├── ingestion/
              ├── generation/
              ├── curriculum/
              ├── embedding/
              ├── chat/
              └── progress/
```

---

## Tech stack

| Layer | Technology | Rationale |
|---|---|---|
| Frontend | Next.js 14, TypeScript, Tailwind CSS | PWA support, single codebase for admin + employee |
| Backend state | PocketBase | Auth, file storage, admin UI, SQLite — no infra overhead |
| Vector store | Qdrant (Docker) | RAG retrieval, runs as single container |
| AI generation | Claude Sonnet 4 via Anthropic API | Structured JSON output, long-form drafting, graph reasoning |
| AI chat (R42) | Claude Haiku 4.5 via Anthropic API | Low latency, cost-effective, grounded by retrieval layer |
| Embeddings | OpenAI text-embedding-3-small | Cost-effective, high quality at this scale |
| Auth | PocketBase built-in | Role-based: admin / employee |

---

## AI model responsibilities

| Task | Model |
|---|---|
| Document → KB structure extraction | Claude Sonnet 4 |
| Topic body drafting | Claude Sonnet 4 |
| Micro learning generation (all 10 types) | Claude Sonnet 4 |
| Curriculum generation + versioning | Claude Sonnet 4 |
| R42 chat responses | Claude Haiku 4.5 |
| Embeddings | text-embedding-3-small |

---

## Document ingestion pipeline

```
Admin uploads file (PDF / MD / TXT)
         ↓
Format detection → text extraction
  MD:  split on headings → preserve hierarchy
  PDF: pdfplumber → page + paragraph detection
  TXT: sliding window chunking with overlap
         ↓
Chunk cleaning (strip headers/footers/artefacts)
         ↓
Claude Sonnet 4 reads chunks → extracts:
  - candidate Themes
  - candidate Topics per Theme
  - Topic→Topic relationships (related, prerequisite, contrast)
  - key terms for glossary
         ↓
Draft KB written to PocketBase (status: draft)
         ↓
Embedding service: embed source chunks → write to Qdrant
         ↓
Admin reviews Theme batch → approves / edits / rejects
         ↓
On approval: Topics published, micro learning generation queued
         ↓
Curriculum regeneration notification queued for admin
```

Note: embeddings are generated from **source chunks**, not only from AI-generated topic summaries. R42 retrieves from grounded source material.

MD source files are the preferred format for admins — heading structure maps directly to Theme → Topic hierarchy and improves extraction quality.

---

## Curriculum lifecycle

### Generation
Input: all published Themes, Topics, relationship graph, complexity weights
Process: cluster by Theme → sequence pedagogically (prerequisites first, complexity gradient) → distribute across 26 weeks → ensure full KB coverage
Output: versioned 26-week draft schedule

### Perpetual cycling
The curriculum runs continuously. After week 26, the employee begins cycle 2 on the latest curriculum version.

Second and subsequent cycles are not identical to cycle 1:
- Theme sequence is varied
- Recommended micro learning types surface types the employee has not yet used
- Topics with low engagement in prior cycles receive increased coverage

### Versioning rules

| Event | Action |
|---|---|
| New source doc published to KB | Regenerate curriculum from week N+1 for all active employees |
| Topic body edited | Micro learnings regenerated; curriculum unaffected |
| Theme batch approved | Regeneration queued; admin confirms before it applies |

Completed weeks are immutable. Regeneration only affects future unstarted weeks.

### Admin regeneration flow
Admin receives notification: "N new topics added. Regenerate curriculum? This will update unstarted weeks for all active employees."
Admin can preview the proposed new schedule before confirming.

---

## Weekly session flow (employee)

```
Week N opens
  ↓
Employee sees assigned Theme + Topics for the week
  ↓
Per Topic: employee selects micro learning type
  (all published types for that topic are available)
  ↓
Employee completes one or more types per topic
  ↓
Completion recorded → XP awarded → badges evaluated
  ↓
Progress visible on public leaderboard and activity feed
```

Sessions support multiple micro learning types per topic in a single session.

---

## Micro learning types

All 10 types are generated by Claude Sonnet 4 as structured JSON, stored in PocketBase, and rendered by the frontend. One or more types may be published per topic.

| # | Type | Format |
|---|---|---|
| 1 | Concept explainer | 2–3 paragraphs + example |
| 2 | Scenario quiz | situation + 3–4 MCQ options + explained answers |
| 3 | Common misconceptions | 3–5 false beliefs + corrections |
| 4 | Step-by-step how-to | numbered procedure |
| 5 | Comparison card | side-by-side on 4–6 dimensions |
| 6 | Reflection prompt | open question + model answer |
| 7 | Spaced repetition flashcards | 5–10 Q&A pairs |
| 8 | Case study mini-analysis | 150–200 word scenario + guiding questions |
| 9 | Glossary anchor | term + definition + correct use + misuse |
| 10 | Myth vs. evidence | false claim + evidence-based rebuttal |

---

## R42 — chat service design

R42 is a functional KB-grounded assistant available on every screen in the employee app.

Behaviour:
- Stateless per session (no memory between conversations)
- Retrieves relevant chunks from Qdrant using the employee's query
- Knows the employee's current curriculum week → retrieval is context-weighted
- Cites source topic in every response ("based on the **Holacratic roles** topic")
- Explicitly refuses to answer outside KB scope rather than hallucinating
- Scope: internal KB only

Implementation:
- Employee query → embed → Qdrant nearest-neighbour retrieval → top-K chunks
- Chunks + employee context injected into Haiku 4.5 prompt
- Response streamed to frontend

UI: floating button bottom-right, unobtrusive on mobile.

---

## Gamification system

Inspired by the visual language of GitHub, Stack Overflow, and Duolingo. Mechanics use developer-native terminology.

### XP unit: commits
Every completed topic earns commits. Quantity varies by micro learning type complexity.

### Levels
`Intern → Junior → Medior → Senior → Staff → Principal`
Based on cumulative commits across all cycles.

### Streak
Counted in consecutive weeks, not days. Resets if a week is skipped entirely.

### Activity heatmap
GitHub-style contribution graph spanning the full 26-week cycle. Cell darkness = number of types completed that week.

### Badges

| Tier | Condition |
|---|---|
| Bronze | Complete any session |
| Silver | 5 sessions completed, 5 different types used |
| Gold | 13 sessions without skipping a week |
| Legendary | All 26 sessions, all 10 types used at least once |

Named content badges (examples):
- `governance nerd` — all holacratic structure topics completed
- `process architect` — all internal process topics completed
- `deep reader` — case study type used 5+ times

### Milestone cards (public)
At weeks 13 and 26, a public card is posted to the shared activity feed:

```
🚀  [Name] shipped the full curriculum
    26 weeks · [N] commits · [badges]
    Longest streak: [N] weeks
```

Language: shipping vocabulary, not school vocabulary.

### Leaderboard
Not ranked 1–N by score. Displays multiple dimensions:

| Employee | Commits | Streak | Types used | Badges |
|---|---|---|---|---|

Multiple paths to visibility. No single metric determines standing.

---

## Security and privacy

- Auth: PocketBase role-based (admin / employee)
- Gamification data (commits, badges, streak) is public to all employees
- Session completion data (which topic, which type, when) is public
- Source documents are admin-only
- No PII beyond display name stored in gamification context
- R42 is stateless — no chat history persisted