Add comprehensive documentation for employee learning platform
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
This commit is contained in:
RaymondVerhoef
394
docs/implementation-plan.md
Normal file
394
docs/implementation-plan.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# Implementation plan
|
||||
|
||||
## How to use this document
|
||||
|
||||
Work through phases in order. Do not start phase N+1 before phase N passes
|
||||
all acceptance criteria. Each phase lists the spec file to read, the steps
|
||||
to execute, and the criteria that define done.
|
||||
|
||||
At the start of each session: state the phase and step.
|
||||
At the end of each session: state completed steps and next starting point.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Infrastructure + ingestion service
|
||||
|
||||
**Spec to read:** /docs/ingestion-spec.md, /docs/data-model.md
|
||||
|
||||
### Steps
|
||||
|
||||
**1.1 — Repo scaffold**
|
||||
```
|
||||
app/
|
||||
frontend/ (empty, Next.js init comes in phase 4)
|
||||
services/
|
||||
ingestion/
|
||||
generation/ (empty placeholder)
|
||||
curriculum/ (empty placeholder)
|
||||
chat/ (empty placeholder)
|
||||
progress/ (empty placeholder)
|
||||
```
|
||||
|
||||
Create `app/services/ingestion/` with:
|
||||
- package.json (dependencies from ingestion-spec.md)
|
||||
- tsconfig.json (strict mode)
|
||||
- .env.example (all env vars from ingestion-spec.md)
|
||||
- .gitignore
|
||||
|
||||
**1.2 — PocketBase collections**
|
||||
PocketBase runs as a binary. Create a migration script at
|
||||
`app/services/ingestion/migrations/001_initial_schema.ts` that uses the
|
||||
PocketBase JS SDK to create all collections defined in data-model.md:
|
||||
|
||||
Collections to create:
|
||||
- source_documents
|
||||
- themes
|
||||
- topics
|
||||
- micro_learnings (schema only — no data yet)
|
||||
- curriculum_versions (schema only)
|
||||
- curriculum_weeks (schema only)
|
||||
- employee_curriculum_state (schema only)
|
||||
- session_completions (schema only)
|
||||
- gamification_profiles (schema only)
|
||||
- badges (schema only)
|
||||
- employee_badges (schema only)
|
||||
- milestone_cards (schema only)
|
||||
|
||||
Seed the badges collection with all badge definitions from data-model.md.
|
||||
|
||||
**1.3 — Qdrant collections**
|
||||
Create `app/services/ingestion/migrations/002_qdrant_setup.ts` that
|
||||
initialises both Qdrant collections:
|
||||
- source_chunks (1536 dimensions, cosine distance)
|
||||
- topic_summaries (1536 dimensions, cosine distance)
|
||||
|
||||
**1.4 — Ingestion service scaffold**
|
||||
Build the Fastify server with two routes:
|
||||
- POST /ingest
|
||||
- GET /status/:jobId
|
||||
|
||||
Use the file structure from ingestion-spec.md exactly.
|
||||
|
||||
**1.5 — Stage 1: text extraction**
|
||||
Implement extract.ts per ingestion-spec.md:
|
||||
- TXT: direct UTF-8 read
|
||||
- MD: direct UTF-8 read, preserve heading markers
|
||||
- PDF: pdf-parse, page break markers
|
||||
|
||||
**1.6 — Stage 2–3: chunking + cleaning**
|
||||
Implement chunk.ts and clean.ts per ingestion-spec.md:
|
||||
- MD: heading-based splitting
|
||||
- TXT: sliding window (800 chars, 150 overlap)
|
||||
- PDF: page + paragraph splitting
|
||||
- Cleaning: whitespace, artefacts, minimum length filter
|
||||
|
||||
**1.7 — Stage 4: structure extraction**
|
||||
Implement structure.ts per ingestion-spec.md:
|
||||
- Claude Sonnet 4 call with system + user prompt from spec
|
||||
- Zod validation of DraftKB output
|
||||
- Batch handling for documents > 60 chunks
|
||||
- Retry logic on parse failure
|
||||
- Error handling: failed job status + reason
|
||||
|
||||
**1.8 — Stage 5: PocketBase write**
|
||||
Implement the PocketBase write logic:
|
||||
- Create Theme records (status: draft)
|
||||
- Create Topic records under each Theme (status: draft)
|
||||
- Resolve relationships between Topics after all records created
|
||||
|
||||
**1.9 — Stage 6: embeddings + Qdrant write**
|
||||
Implement embed.ts:
|
||||
- OpenAI text-embedding-3-small, batches of 100
|
||||
- Write to Qdrant source_chunks collection
|
||||
- Write to Qdrant topic_summaries collection
|
||||
- Update Topic.qdrant_chunk_ids in PocketBase
|
||||
|
||||
**1.10 — Job status tracking**
|
||||
Wire all stages into the job queue (jobs/queue.ts):
|
||||
- Status transitions: queued → extracting → chunking → structuring →
|
||||
writing → embedding → done / failed
|
||||
- Progress counters (chunksTotal, chunksEmbedded, themesFound, topicsFound)
|
||||
- GET /status/:jobId returns current state
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] POST /ingest with a small MD file completes without error
|
||||
- [ ] GET /status/:jobId returns `done` after processing
|
||||
- [ ] PocketBase contains draft Theme + Topic records with correct hierarchy
|
||||
- [ ] Topic.body contains AI-drafted content (not empty)
|
||||
- [ ] Topic relationships are resolved (related_topics populated where applicable)
|
||||
- [ ] Qdrant source_chunks contains vectors with correct payload fields
|
||||
- [ ] Qdrant topic_summaries contains vectors for each published topic
|
||||
- [ ] Topic.qdrant_chunk_ids is populated
|
||||
- [ ] POST /ingest with a PDF file completes without error
|
||||
- [ ] POST /ingest with a TXT file completes without error
|
||||
- [ ] A document > 60 chunks triggers batch processing without error
|
||||
- [ ] A malformed PDF returns status `failed` with reason, not an uncaught exception
|
||||
- [ ] All Zod validations pass — no `any` types in codebase
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Generation service
|
||||
|
||||
**Spec to read:** /docs/generation-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**2.1 — Generation service scaffold**
|
||||
Fastify service at app/services/generation/
|
||||
Routes: POST /generate, GET /status/:jobId
|
||||
|
||||
**2.2 — Generate all 10 types per topic**
|
||||
One Claude Sonnet 4 call per type per topic.
|
||||
Structured JSON output validated against Zod schemas from data-model.md.
|
||||
Write to micro_learnings collection (status: generated).
|
||||
|
||||
**2.3 — Batch generation on theme approval**
|
||||
When admin approves a Theme batch, queue generation for all Topics in that Theme.
|
||||
All 10 types per Topic.
|
||||
|
||||
**2.4 — Admin publish flow**
|
||||
Route to update micro_learning status from generated → published or rejected.
|
||||
This is called by the admin app (built in phase 4).
|
||||
|
||||
### Acceptance criteria (to be detailed in generation-spec.md)
|
||||
|
||||
- [ ] All 10 micro learning types generated for a test topic
|
||||
- [ ] All 10 JSON outputs validate against their Zod schemas
|
||||
- [ ] Generated content written to PocketBase with status: generated
|
||||
- [ ] Admin can publish or reject individual micro learnings
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Curriculum service
|
||||
|
||||
**Spec to read:** /docs/curriculum-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**3.1 — Curriculum service scaffold**
|
||||
Fastify service at app/services/curriculum/
|
||||
|
||||
**3.2 — Curriculum generator**
|
||||
Claude Sonnet 4 reads full KB graph → produces 26-week schedule.
|
||||
Written to curriculum_versions + curriculum_weeks.
|
||||
|
||||
**3.3 — Versioning logic**
|
||||
- New version created on regeneration
|
||||
- Completed weeks frozen (employee_curriculum_state.current_week used as boundary)
|
||||
- Admin confirmation required before applying new version
|
||||
|
||||
**3.4 — Perpetual cycling**
|
||||
On week 26 completion, cycle increments, new cycle starts on latest version.
|
||||
Second cycle: varied sequence, surfaces unused micro learning types.
|
||||
|
||||
### Acceptance criteria (to be detailed in curriculum-spec.md)
|
||||
|
||||
- [ ] Curriculum generated from a populated KB
|
||||
- [ ] 26 weeks produced, all Themes covered
|
||||
- [ ] Prerequisites respected in ordering
|
||||
- [ ] Regeneration does not alter completed weeks
|
||||
- [ ] Admin confirmation flow works correctly
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Frontend: admin app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**4.1 — Next.js 14 scaffold**
|
||||
Mobile-first, TypeScript strict, Tailwind CSS, PWA config.
|
||||
Role-based routing: /admin/* and /app/* from single Next.js codebase.
|
||||
|
||||
**4.2 — Auth**
|
||||
PocketBase auth integration. Admin role routes to /admin/*.
|
||||
|
||||
**4.3 — Document upload + ingestion status**
|
||||
Upload UI → calls ingestion service → polls job status → shows progress.
|
||||
|
||||
**4.4 — Theme batch review**
|
||||
Display draft Themes with their Topic list.
|
||||
Approve batch / edit individual topics / reject batch.
|
||||
Triggers generation service on approval.
|
||||
|
||||
**4.5 — Curriculum editor**
|
||||
Display AI-generated curriculum (26 weeks).
|
||||
Drag-to-reorder weeks. Edit Theme assignment per week.
|
||||
Confirm regeneration with preview.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Admin can upload a document and see ingestion progress
|
||||
- [ ] Admin can approve a Theme batch
|
||||
- [ ] Admin can edit a Topic before approval
|
||||
- [ ] Admin can view and reorder the curriculum
|
||||
- [ ] Admin can confirm a curriculum regeneration with preview
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Frontend: employee app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (same file, employee section)
|
||||
|
||||
### Steps
|
||||
|
||||
**5.1 — Employee auth + onboarding**
|
||||
PocketBase auth. Employee role routes to /app/*.
|
||||
Set start date on first login → creates employee_curriculum_state record.
|
||||
|
||||
**5.2 — Weekly session flow**
|
||||
Current week's Theme displayed.
|
||||
Topics listed with available micro learning types per topic.
|
||||
Employee selects type → content rendered → mark complete.
|
||||
|
||||
**5.3 — Knowledge library**
|
||||
Browse all published Topics.
|
||||
Filter by Theme, difficulty, key terms.
|
||||
|
||||
**5.4 — R42 chatbot**
|
||||
Floating button, every screen.
|
||||
Calls chat service → streams response.
|
||||
Cites source topic in response.
|
||||
|
||||
**5.5 — Gamification profile**
|
||||
GitHub-style heatmap (26-week view).
|
||||
Badge display.
|
||||
Streak + level + commit count.
|
||||
Public leaderboard (multi-dimension).
|
||||
Milestone cards in activity feed.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Employee sees correct week based on start date
|
||||
- [ ] Employee can complete a topic with a chosen micro learning type
|
||||
- [ ] Completion is recorded and XP awarded
|
||||
- [ ] Knowledge library shows all published topics with filters
|
||||
- [ ] R42 responds with grounded answer and source citation
|
||||
- [ ] R42 is accessible from every screen
|
||||
- [ ] Heatmap renders correctly on mobile (375px)
|
||||
- [ ] Leaderboard shows all employees with multi-dimension data
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Chat service (R42)
|
||||
|
||||
**Spec to read:** /docs/r42-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**6.1 — Chat service scaffold**
|
||||
Fastify service at app/services/chat/
|
||||
|
||||
**6.2 — Query → embed → retrieve**
|
||||
Employee query embedded → Qdrant nearest-neighbour on both collections.
|
||||
Boost chunks from employee's current Theme.
|
||||
|
||||
**6.3 — Response generation**
|
||||
Top-K chunks injected into Haiku 4.5 prompt.
|
||||
Response streamed to frontend.
|
||||
Source Topic titles included in response.
|
||||
|
||||
**6.4 — Out-of-scope handling**
|
||||
If retrieval confidence is below threshold, R42 responds:
|
||||
"I can only answer questions based on the internal knowledge base.
|
||||
This topic doesn't appear to be covered."
|
||||
|
||||
### Acceptance criteria (to be detailed in r42-spec.md)
|
||||
|
||||
- [ ] R42 answers a question about a published topic correctly
|
||||
- [ ] R42 cites the source topic in its response
|
||||
- [ ] R42 refuses to answer out-of-scope questions explicitly
|
||||
- [ ] Response streams to frontend (not batch)
|
||||
- [ ] Response latency < 3 seconds for typical queries
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Progress service
|
||||
|
||||
**Spec to read:** /docs/gamification-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**7.1 — Progress service scaffold**
|
||||
Fastify service at app/services/progress/
|
||||
|
||||
**7.2 — Completion recording**
|
||||
Write session_completions record on topic completion.
|
||||
Calculate XP (commits) per type.
|
||||
|
||||
**7.3 — Gamification updates**
|
||||
Update gamification_profiles: commits, level, streak, types_used.
|
||||
Evaluate badge conditions → write employee_badges on award.
|
||||
|
||||
**7.4 — Milestone cards**
|
||||
Generate milestone_cards record at weeks 13 and 26.
|
||||
|
||||
**7.5 — Leaderboard query**
|
||||
Endpoint returning all gamification_profiles for leaderboard rendering.
|
||||
|
||||
### Acceptance criteria (to be detailed in gamification-spec.md)
|
||||
|
||||
- [ ] Completion writes to session_completions
|
||||
- [ ] Commits calculated and added to gamification_profile
|
||||
- [ ] Level updates correctly at commit thresholds
|
||||
- [ ] Streak increments on weekly completion, resets on skip
|
||||
- [ ] Badge awarded when condition is met
|
||||
- [ ] Milestone card created at weeks 13 and 26
|
||||
- [ ] Leaderboard endpoint returns all employees with correct data
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Integration + hardening
|
||||
|
||||
No new spec required.
|
||||
|
||||
### Steps
|
||||
|
||||
**8.1 — Service wiring**
|
||||
Verify all services communicate through PocketBase correctly.
|
||||
No direct service-to-service calls — all state through PocketBase.
|
||||
|
||||
**8.2 — Error handling audit**
|
||||
Review all services for unhandled promise rejections, missing error states,
|
||||
and uncaught exceptions. Every external call (AI API, PocketBase, Qdrant,
|
||||
OpenAI) wrapped in try/catch with meaningful error logging.
|
||||
|
||||
**8.3 — Mobile QA**
|
||||
Test all employee app flows at 375px width.
|
||||
R42 floating button must not obscure content.
|
||||
Heatmap must render without horizontal scroll.
|
||||
|
||||
**8.4 — Environment variable audit**
|
||||
Verify no hardcoded values. All .env.example files complete.
|
||||
|
||||
**8.5 — Dockerfile update**
|
||||
Update COPY path from legacy app root to /app.
|
||||
This is the one manual change that connects the rebuild to the existing pipeline.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] Full flow works end-to-end: upload doc → approve → curriculum → employee completes session → R42 answers question → gamification updates
|
||||
- [ ] No uncaught exceptions in any service under normal operating conditions
|
||||
- [ ] All screens render correctly on 375px mobile
|
||||
- [ ] Dockerfile builds successfully pointing at /app
|
||||
- [ ] Existing pipeline deploys the rebuilt app without modification
|
||||
|
||||
---
|
||||
|
||||
## Spec files still to be written
|
||||
|
||||
Before starting each phase, write the corresponding spec file.
|
||||
Use ingestion-spec.md as the template for structure and detail level.
|
||||
|
||||
| Phase | Spec file needed |
|
||||
|---|---|
|
||||
| 2 | /docs/generation-spec.md |
|
||||
| 3 | /docs/curriculum-spec.md |
|
||||
| 4–5 | /docs/frontend-spec.md |
|
||||
| 6 | /docs/r42-spec.md |
|
||||
| 7 | /docs/gamification-spec.md |
|
||||
|
||||
When you reach a phase without a spec: stop, draft the spec, then proceed.
|
||||
Do not implement without a spec.
|
||||
Reference in New Issue
Block a user