learning-platform/docs/implementation-plan.md

# Implementation plan

## How to use this document

Work through phases in order. Do not start phase N+1 before phase N passes
all acceptance criteria. Each phase lists the spec file to read, the steps
to execute, and the criteria that define done.

At the start of each session: state the phase and step.
At the end of each session: state completed steps and next starting point.

---

## Phase 1 — Infrastructure + ingestion service

**Spec to read:** /docs/ingestion-spec.md, /docs/data-model.md

### Steps

**1.1 — Repo scaffold**
```
app/
  frontend/          (empty, Next.js init comes in phase 4)
  services/
    ingestion/
    generation/      (empty placeholder)
    curriculum/      (empty placeholder)
    chat/            (empty placeholder)
    progress/        (empty placeholder)
```

Create `app/services/ingestion/` with:
- package.json (dependencies from ingestion-spec.md)
- tsconfig.json (strict mode)
- .env.example (all env vars from ingestion-spec.md)
- .gitignore

**1.2 — PocketBase collections**
PocketBase runs as a binary. Create a migration script at
`app/services/ingestion/migrations/001_initial_schema.ts` that uses the
PocketBase JS SDK to create all collections defined in data-model.md:

Collections to create:
- source_documents
- themes
- topics
- micro_learnings (schema only — no data yet)
- curriculum_versions (schema only)
- curriculum_weeks (schema only)
- employee_curriculum_state (schema only)
- session_completions (schema only)
- gamification_profiles (schema only)
- badges (schema only)
- employee_badges (schema only)
- milestone_cards (schema only)

Seed the badges collection with all badge definitions from data-model.md.

**1.3 — Qdrant collections**
Create `app/services/ingestion/migrations/002_qdrant_setup.ts` that
initialises both Qdrant collections:
- source_chunks (1536 dimensions, cosine distance)
- topic_summaries (1536 dimensions, cosine distance)

**1.4 — Ingestion service scaffold**
Build the Fastify server with two routes:
- POST /ingest
- GET /status/:jobId

Use the file structure from ingestion-spec.md exactly.

**1.5 — Stage 1: text extraction**
Implement extract.ts per ingestion-spec.md:
- TXT: direct UTF-8 read
- MD: direct UTF-8 read, preserve heading markers
- PDF: pdf-parse, page break markers

**1.6 — Stage 2–3: chunking + cleaning**
Implement chunk.ts and clean.ts per ingestion-spec.md:
- MD: heading-based splitting
- TXT: sliding window (800 chars, 150 overlap)
- PDF: page + paragraph splitting
- Cleaning: whitespace, artefacts, minimum length filter

**1.7 — Stage 4: structure extraction**
Implement structure.ts per ingestion-spec.md:
- Claude Sonnet 4 call with system + user prompt from spec
- Zod validation of DraftKB output
- Batch handling for documents > 60 chunks
- Retry logic on parse failure
- Error handling: failed job status + reason

**1.8 — Stage 5: PocketBase write**
Implement the PocketBase write logic:
- Create Theme records (status: draft)
- Create Topic records under each Theme (status: draft)
- Resolve relationships between Topics after all records created

**1.9 — Stage 6: embeddings + Qdrant write**
Implement embed.ts:
- OpenAI text-embedding-3-small, batches of 100
- Write to Qdrant source_chunks collection
- Write to Qdrant topic_summaries collection
- Update Topic.qdrant_chunk_ids in PocketBase

**1.10 — Job status tracking**
Wire all stages into the job queue (jobs/queue.ts):
- Status transitions: queued → extracting → chunking → structuring →
  writing → embedding → done / failed
- Progress counters (chunksTotal, chunksEmbedded, themesFound, topicsFound)
- GET /status/:jobId returns current state

### Acceptance criteria

- [ ] POST /ingest with a small MD file completes without error
- [ ] GET /status/:jobId returns `done` after processing
- [ ] PocketBase contains draft Theme + Topic records with correct hierarchy
- [ ] Topic.body contains AI-drafted content (not empty)
- [ ] Topic relationships are resolved (related_topics populated where applicable)
- [ ] Qdrant source_chunks contains vectors with correct payload fields
- [ ] Qdrant topic_summaries contains vectors for each published topic
- [ ] Topic.qdrant_chunk_ids is populated
- [ ] POST /ingest with a PDF file completes without error
- [ ] POST /ingest with a TXT file completes without error
- [ ] A document > 60 chunks triggers batch processing without error
- [ ] A malformed PDF returns status `failed` with reason, not an uncaught exception
- [ ] All Zod validations pass — no `any` types in codebase

---

## Phase 2 — Generation service

**Spec to read:** /docs/generation-spec.md (write this spec before starting)

### Steps

**2.1 — Generation service scaffold**
Fastify service at app/services/generation/
Routes: POST /generate, GET /status/:jobId

**2.2 — Generate all 10 types per topic**
One Claude Sonnet 4 call per type per topic.
Structured JSON output validated against Zod schemas from data-model.md.
Write to micro_learnings collection (status: generated).

**2.3 — Batch generation on theme approval**
When admin approves a Theme batch, queue generation for all Topics in that Theme.
All 10 types per Topic.

**2.4 — Admin publish flow**
Route to update micro_learning status from generated → published or rejected.
This is called by the admin app (built in phase 4).

### Acceptance criteria (to be detailed in generation-spec.md)

- [ ] All 10 micro learning types generated for a test topic
- [ ] All 10 JSON outputs validate against their Zod schemas
- [ ] Generated content written to PocketBase with status: generated
- [ ] Admin can publish or reject individual micro learnings

---

## Phase 3 — Curriculum service

**Spec to read:** /docs/curriculum-spec.md (write this spec before starting)

### Steps

**3.1 — Curriculum service scaffold**
Fastify service at app/services/curriculum/

**3.2 — Curriculum generator**
Claude Sonnet 4 reads full KB graph → produces 26-week schedule.
Written to curriculum_versions + curriculum_weeks.

**3.3 — Versioning logic**
- New version created on regeneration
- Completed weeks frozen (employee_curriculum_state.current_week used as boundary)
- Admin confirmation required before applying new version

**3.4 — Perpetual cycling**
On week 26 completion, cycle increments, new cycle starts on latest version.
Second cycle: varied sequence, surfaces unused micro learning types.

### Acceptance criteria (to be detailed in curriculum-spec.md)

- [ ] Curriculum generated from a populated KB
- [ ] 26 weeks produced, all Themes covered
- [ ] Prerequisites respected in ordering
- [ ] Regeneration does not alter completed weeks
- [ ] Admin confirmation flow works correctly

---

## Phase 4 — Frontend: admin app

**Spec to read:** /docs/frontend-spec.md (write this spec before starting)

### Steps

**4.1 — Next.js 14 scaffold**
Mobile-first, TypeScript strict, Tailwind CSS, PWA config.
Role-based routing: /admin/* and /app/* from single Next.js codebase.

**4.2 — Auth**
PocketBase auth integration. Admin role routes to /admin/*.

**4.3 — Document upload + ingestion status**
Upload UI → calls ingestion service → polls job status → shows progress.

**4.4 — Theme batch review**
Display draft Themes with their Topic list.
Approve batch / edit individual topics / reject batch.
Triggers generation service on approval.

**4.5 — Curriculum editor**
Display AI-generated curriculum (26 weeks).
Drag-to-reorder weeks. Edit Theme assignment per week.
Confirm regeneration with preview.

### Acceptance criteria (to be detailed in frontend-spec.md)

- [ ] Admin can upload a document and see ingestion progress
- [ ] Admin can approve a Theme batch
- [ ] Admin can edit a Topic before approval
- [ ] Admin can view and reorder the curriculum
- [ ] Admin can confirm a curriculum regeneration with preview

---

## Phase 5 — Frontend: employee app

**Spec to read:** /docs/frontend-spec.md (same file, employee section)

### Steps

**5.1 — Employee auth + onboarding**
PocketBase auth. Employee role routes to /app/*.
Set start date on first login → creates employee_curriculum_state record.

**5.2 — Weekly session flow**
Current week's Theme displayed.
Topics listed with available micro learning types per topic.
Employee selects type → content rendered → mark complete.

**5.3 — Knowledge library**
Browse all published Topics.
Filter by Theme, difficulty, key terms.

**5.4 — R42 chatbot**
Floating button, every screen.
Calls chat service → streams response.
Cites source topic in response.

**5.5 — Gamification profile**
GitHub-style heatmap (26-week view).
Badge display.
Streak + level + commit count.
Public leaderboard (multi-dimension).
Milestone cards in activity feed.

### Acceptance criteria (to be detailed in frontend-spec.md)

- [ ] Employee sees correct week based on start date
- [ ] Employee can complete a topic with a chosen micro learning type
- [ ] Completion is recorded and XP awarded
- [ ] Knowledge library shows all published topics with filters
- [ ] R42 responds with grounded answer and source citation
- [ ] R42 is accessible from every screen
- [ ] Heatmap renders correctly on mobile (375px)
- [ ] Leaderboard shows all employees with multi-dimension data

---

## Phase 6 — Chat service (R42)

**Spec to read:** /docs/r42-spec.md (write this spec before starting)

### Steps

**6.1 — Chat service scaffold**
Fastify service at app/services/chat/

**6.2 — Query → embed → retrieve**
Employee query embedded → Qdrant nearest-neighbour on both collections.
Boost chunks from employee's current Theme.

**6.3 — Response generation**
Top-K chunks injected into Haiku 4.5 prompt.
Response streamed to frontend.
Source Topic titles included in response.

**6.4 — Out-of-scope handling**
If retrieval confidence is below threshold, R42 responds:
"I can only answer questions based on the internal knowledge base.
This topic doesn't appear to be covered."

### Acceptance criteria (to be detailed in r42-spec.md)

- [ ] R42 answers a question about a published topic correctly
- [ ] R42 cites the source topic in its response
- [ ] R42 refuses to answer out-of-scope questions explicitly
- [ ] Response streams to frontend (not batch)
- [ ] Response latency < 3 seconds for typical queries

---

## Phase 7 — Progress service

**Spec to read:** /docs/gamification-spec.md (write this spec before starting)

### Steps

**7.1 — Progress service scaffold**
Fastify service at app/services/progress/

**7.2 — Completion recording**
Write session_completions record on topic completion.
Calculate XP (commits) per type.

**7.3 — Gamification updates**
Update gamification_profiles: commits, level, streak, types_used.
Evaluate badge conditions → write employee_badges on award.

**7.4 — Milestone cards**
Generate milestone_cards record at weeks 13 and 26.

**7.5 — Leaderboard query**
Endpoint returning all gamification_profiles for leaderboard rendering.

### Acceptance criteria (to be detailed in gamification-spec.md)

- [ ] Completion writes to session_completions
- [ ] Commits calculated and added to gamification_profile
- [ ] Level updates correctly at commit thresholds
- [ ] Streak increments on weekly completion, resets on skip
- [ ] Badge awarded when condition is met
- [ ] Milestone card created at weeks 13 and 26
- [ ] Leaderboard endpoint returns all employees with correct data

---

## Phase 8 — Integration + hardening

No new spec required.

### Steps

**8.1 — Service wiring**
Verify all services communicate through PocketBase correctly.
No direct service-to-service calls — all state through PocketBase.

**8.2 — Error handling audit**
Review all services for unhandled promise rejections, missing error states,
and uncaught exceptions. Every external call (AI API, PocketBase, Qdrant,
OpenAI) wrapped in try/catch with meaningful error logging.

**8.3 — Mobile QA**
Test all employee app flows at 375px width.
R42 floating button must not obscure content.
Heatmap must render without horizontal scroll.

**8.4 — Environment variable audit**
Verify no hardcoded values. All .env.example files complete.

**8.5 — Dockerfile update**
Update COPY path from legacy app root to /app.
This is the one manual change that connects the rebuild to the existing pipeline.

### Acceptance criteria

- [ ] Full flow works end-to-end: upload doc → approve → curriculum → employee completes session → R42 answers question → gamification updates
- [ ] No uncaught exceptions in any service under normal operating conditions
- [ ] All screens render correctly on 375px mobile
- [ ] Dockerfile builds successfully pointing at /app
- [ ] Existing pipeline deploys the rebuilt app without modification

---

## Spec files still to be written

Before starting each phase, write the corresponding spec file.
Use ingestion-spec.md as the template for structure and detail level.

| Phase | Spec file needed |
|---|---|
| 2 | /docs/generation-spec.md |
| 3 | /docs/curriculum-spec.md |
| 4–5 | /docs/frontend-spec.md |
| 6 | /docs/r42-spec.md |
| 7 | /docs/gamification-spec.md |

When you reach a phase without a spec: stop, draft the spec, then proceed.
Do not implement without a spec.