Add comprehensive documentation for employee learning platform
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
This commit is contained in:
RaymondVerhoef
279
docs/architecture.md
Normal file
279
docs/architecture.md
Normal file
@@ -0,0 +1,279 @@
|
||||
# Architecture: employee learning platform
|
||||
|
||||
## Overview
|
||||
|
||||
A mobile-first progressive web application that provides employees with a structured knowledge library, a 26-week perpetual learning curriculum, and an AI-powered assistant (R42). The knowledge base is the single source of truth for all content, micro learnings, curriculum scheduling, and chat retrieval.
|
||||
|
||||
---
|
||||
|
||||
## System domains
|
||||
|
||||
### Admin app
|
||||
Browser-based interface for content administrators.
|
||||
|
||||
Responsibilities:
|
||||
- Upload source documents (PDF, MD, TXT)
|
||||
- Review and approve AI-generated Theme batches
|
||||
- Edit and finetune AI-generated curriculum
|
||||
- Confirm curriculum regeneration after KB updates
|
||||
- Monitor ingestion and generation job status
|
||||
|
||||
### Employee app
|
||||
Mobile-first PWA accessible on all devices.
|
||||
|
||||
Responsibilities:
|
||||
- Weekly session delivery (Theme + Topics + micro learning type selection)
|
||||
- Knowledge library (browse all published Topics)
|
||||
- Gamification profile (heatmap, badges, streak, leaderboard)
|
||||
- R42 chatbot (available on every screen)
|
||||
|
||||
### Backend services
|
||||
Six discrete services, each with a single responsibility.
|
||||
|
||||
| Service | Responsibility |
|
||||
|---|---|
|
||||
| Ingestion service | Document upload → chunk → extract KB structure |
|
||||
| Generation service | Topics → 10 micro learning types (structured JSON) |
|
||||
| Curriculum service | KB graph → 26-week schedule, versioning, regeneration |
|
||||
| Embedding service | Chunks + topic summaries → Qdrant |
|
||||
| Chat service (R42) | Query → vector retrieval → grounded response |
|
||||
| Progress service | Completions → XP → badges → streak |
|
||||
|
||||
---
|
||||
|
||||
## Deployment topology
|
||||
|
||||
```
|
||||
repo/
|
||||
├── .github/workflows/ ← pipeline (frozen)
|
||||
├── docker-compose.yml ← infrastructure (frozen)
|
||||
├── Dockerfile ← updated once to point at /app
|
||||
├── ansible/ ← provisioning (frozen)
|
||||
├── legacy/ ← original prototype (read-only reference)
|
||||
└── app/
|
||||
├── frontend/ ← Next.js PWA (admin + employee)
|
||||
└── services/
|
||||
├── ingestion/
|
||||
├── generation/
|
||||
├── curriculum/
|
||||
├── embedding/
|
||||
├── chat/
|
||||
└── progress/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Layer | Technology | Rationale |
|
||||
|---|---|---|
|
||||
| Frontend | Next.js 14, TypeScript, Tailwind CSS | PWA support, single codebase for admin + employee |
|
||||
| Backend state | PocketBase | Auth, file storage, admin UI, SQLite — no infra overhead |
|
||||
| Vector store | Qdrant (Docker) | RAG retrieval, runs as single container |
|
||||
| AI generation | Claude Sonnet 4 via Anthropic API | Structured JSON output, long-form drafting, graph reasoning |
|
||||
| AI chat (R42) | Claude Haiku 4.5 via Anthropic API | Low latency, cost-effective, grounded by retrieval layer |
|
||||
| Embeddings | OpenAI text-embedding-3-small | Cost-effective, high quality at this scale |
|
||||
| Auth | PocketBase built-in | Role-based: admin / employee |
|
||||
|
||||
---
|
||||
|
||||
## AI model responsibilities
|
||||
|
||||
| Task | Model |
|
||||
|---|---|
|
||||
| Document → KB structure extraction | Claude Sonnet 4 |
|
||||
| Topic body drafting | Claude Sonnet 4 |
|
||||
| Micro learning generation (all 10 types) | Claude Sonnet 4 |
|
||||
| Curriculum generation + versioning | Claude Sonnet 4 |
|
||||
| R42 chat responses | Claude Haiku 4.5 |
|
||||
| Embeddings | text-embedding-3-small |
|
||||
|
||||
---
|
||||
|
||||
## Document ingestion pipeline
|
||||
|
||||
```
|
||||
Admin uploads file (PDF / MD / TXT)
|
||||
↓
|
||||
Format detection → text extraction
|
||||
MD: split on headings → preserve hierarchy
|
||||
PDF: pdfplumber → page + paragraph detection
|
||||
TXT: sliding window chunking with overlap
|
||||
↓
|
||||
Chunk cleaning (strip headers/footers/artefacts)
|
||||
↓
|
||||
Claude Sonnet 4 reads chunks → extracts:
|
||||
- candidate Themes
|
||||
- candidate Topics per Theme
|
||||
- Topic→Topic relationships (related, prerequisite, contrast)
|
||||
- key terms for glossary
|
||||
↓
|
||||
Draft KB written to PocketBase (status: draft)
|
||||
↓
|
||||
Embedding service: embed source chunks → write to Qdrant
|
||||
↓
|
||||
Admin reviews Theme batch → approves / edits / rejects
|
||||
↓
|
||||
On approval: Topics published, micro learning generation queued
|
||||
↓
|
||||
Curriculum regeneration notification queued for admin
|
||||
```
|
||||
|
||||
Note: embeddings are generated from **source chunks**, not only from AI-generated topic summaries. R42 retrieves from grounded source material.
|
||||
|
||||
MD source files are the preferred format for admins — heading structure maps directly to Theme → Topic hierarchy and improves extraction quality.
|
||||
|
||||
---
|
||||
|
||||
## Curriculum lifecycle
|
||||
|
||||
### Generation
|
||||
Input: all published Themes, Topics, relationship graph, complexity weights
|
||||
Process: cluster by Theme → sequence pedagogically (prerequisites first, complexity gradient) → distribute across 26 weeks → ensure full KB coverage
|
||||
Output: versioned 26-week draft schedule
|
||||
|
||||
### Perpetual cycling
|
||||
The curriculum runs continuously. After week 26, the employee begins cycle 2 on the latest curriculum version.
|
||||
|
||||
Second and subsequent cycles are not identical to cycle 1:
|
||||
- Theme sequence is varied
|
||||
- Recommended micro learning types surface types the employee has not yet used
|
||||
- Topics with low engagement in prior cycles receive increased coverage
|
||||
|
||||
### Versioning rules
|
||||
|
||||
| Event | Action |
|
||||
|---|---|
|
||||
| New source doc published to KB | Regenerate curriculum from week N+1 for all active employees |
|
||||
| Topic body edited | Micro learnings regenerated; curriculum unaffected |
|
||||
| Theme batch approved | Regeneration queued; admin confirms before it applies |
|
||||
|
||||
Completed weeks are immutable. Regeneration only affects future unstarted weeks.
|
||||
|
||||
### Admin regeneration flow
|
||||
Admin receives notification: "N new topics added. Regenerate curriculum? This will update unstarted weeks for all active employees."
|
||||
Admin can preview the proposed new schedule before confirming.
|
||||
|
||||
---
|
||||
|
||||
## Weekly session flow (employee)
|
||||
|
||||
```
|
||||
Week N opens
|
||||
↓
|
||||
Employee sees assigned Theme + Topics for the week
|
||||
↓
|
||||
Per Topic: employee selects micro learning type
|
||||
(all published types for that topic are available)
|
||||
↓
|
||||
Employee completes one or more types per topic
|
||||
↓
|
||||
Completion recorded → XP awarded → badges evaluated
|
||||
↓
|
||||
Progress visible on public leaderboard and activity feed
|
||||
```
|
||||
|
||||
Sessions support multiple micro learning types per topic in a single session.
|
||||
|
||||
---
|
||||
|
||||
## Micro learning types
|
||||
|
||||
All 10 types are generated by Claude Sonnet 4 as structured JSON, stored in PocketBase, and rendered by the frontend. One or more types may be published per topic.
|
||||
|
||||
| # | Type | Format |
|
||||
|---|---|---|
|
||||
| 1 | Concept explainer | 2–3 paragraphs + example |
|
||||
| 2 | Scenario quiz | situation + 3–4 MCQ options + explained answers |
|
||||
| 3 | Common misconceptions | 3–5 false beliefs + corrections |
|
||||
| 4 | Step-by-step how-to | numbered procedure |
|
||||
| 5 | Comparison card | side-by-side on 4–6 dimensions |
|
||||
| 6 | Reflection prompt | open question + model answer |
|
||||
| 7 | Spaced repetition flashcards | 5–10 Q&A pairs |
|
||||
| 8 | Case study mini-analysis | 150–200 word scenario + guiding questions |
|
||||
| 9 | Glossary anchor | term + definition + correct use + misuse |
|
||||
| 10 | Myth vs. evidence | false claim + evidence-based rebuttal |
|
||||
|
||||
---
|
||||
|
||||
## R42 — chat service design
|
||||
|
||||
R42 is a functional KB-grounded assistant available on every screen in the employee app.
|
||||
|
||||
Behaviour:
|
||||
- Stateless per session (no memory between conversations)
|
||||
- Retrieves relevant chunks from Qdrant using the employee's query
|
||||
- Knows the employee's current curriculum week → retrieval is context-weighted
|
||||
- Cites source topic in every response ("based on the **Holacratic roles** topic")
|
||||
- Explicitly refuses to answer outside KB scope rather than hallucinating
|
||||
- Scope: internal KB only
|
||||
|
||||
Implementation:
|
||||
- Employee query → embed → Qdrant nearest-neighbour retrieval → top-K chunks
|
||||
- Chunks + employee context injected into Haiku 4.5 prompt
|
||||
- Response streamed to frontend
|
||||
|
||||
UI: floating button bottom-right, unobtrusive on mobile.
|
||||
|
||||
---
|
||||
|
||||
## Gamification system
|
||||
|
||||
Inspired by the visual language of GitHub, Stack Overflow, and Duolingo. Mechanics use developer-native terminology.
|
||||
|
||||
### XP unit: commits
|
||||
Every completed topic earns commits. Quantity varies by micro learning type complexity.
|
||||
|
||||
### Levels
|
||||
`Intern → Junior → Medior → Senior → Staff → Principal`
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
### Streak
|
||||
Counted in consecutive weeks, not days. Resets if a week is skipped entirely.
|
||||
|
||||
### Activity heatmap
|
||||
GitHub-style contribution graph spanning the full 26-week cycle. Cell darkness = number of types completed that week.
|
||||
|
||||
### Badges
|
||||
|
||||
| Tier | Condition |
|
||||
|---|---|
|
||||
| Bronze | Complete any session |
|
||||
| Silver | 5 sessions completed, 5 different types used |
|
||||
| Gold | 13 sessions without skipping a week |
|
||||
| Legendary | All 26 sessions, all 10 types used at least once |
|
||||
|
||||
Named content badges (examples):
|
||||
- `governance nerd` — all holacratic structure topics completed
|
||||
- `process architect` — all internal process topics completed
|
||||
- `deep reader` — case study type used 5+ times
|
||||
|
||||
### Milestone cards (public)
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed:
|
||||
|
||||
```
|
||||
🚀 [Name] shipped the full curriculum
|
||||
26 weeks · [N] commits · [badges]
|
||||
Longest streak: [N] weeks
|
||||
```
|
||||
|
||||
Language: shipping vocabulary, not school vocabulary.
|
||||
|
||||
### Leaderboard
|
||||
Not ranked 1–N by score. Displays multiple dimensions:
|
||||
|
||||
| Employee | Commits | Streak | Types used | Badges |
|
||||
|---|---|---|---|---|
|
||||
|
||||
Multiple paths to visibility. No single metric determines standing.
|
||||
|
||||
---
|
||||
|
||||
## Security and privacy
|
||||
|
||||
- Auth: PocketBase role-based (admin / employee)
|
||||
- Gamification data (commits, badges, streak) is public to all employees
|
||||
- Session completion data (which topic, which type, when) is public
|
||||
- Source documents are admin-only
|
||||
- No PII beyond display name stored in gamification context
|
||||
- R42 is stateless — no chat history persisted
|
||||
Reference in New Issue
Block a user