learning-platform/docs/implementation-plan.md

Implementation plan

How to use this document

Work through phases in order. Do not start phase N+1 before phase N passes all acceptance criteria. Each phase lists the spec file to read, the steps to execute, and the criteria that define done.

At the start of each session: state the phase and step. At the end of each session: state completed steps and next starting point.

---

Phase 1 — Infrastructure + ingestion service

Spec to read: /docs/ingestion-spec.md, /docs/data-model.md

Steps

1.1 — Repo scaffold

app/
  frontend/          (empty, Next.js init comes in phase 4)
  services/
    ingestion/
    generation/      (empty placeholder)
    curriculum/      (empty placeholder)
    chat/            (empty placeholder)
    progress/        (empty placeholder)

Create app/services/ingestion/ with:

• package.json (dependencies from ingestion-spec.md)
• tsconfig.json (strict mode)
• .env.example (all env vars from ingestion-spec.md)
• .gitignore

1.2 — PocketBase collections PocketBase runs as a binary. Create a migration script at app/services/ingestion/migrations/001_initial_schema.ts that uses the PocketBase JS SDK to create all collections defined in data-model.md:

Collections to create:

• source_documents
• themes
• topics
• micro_learnings (schema only — no data yet)
• curriculum_versions (schema only)
• curriculum_weeks (schema only)
• employee_curriculum_state (schema only)
• session_completions (schema only)
• gamification_profiles (schema only)
• badges (schema only)
• employee_badges (schema only)
• milestone_cards (schema only)

Seed the badges collection with all badge definitions from data-model.md.

1.3 — Qdrant collections Create app/services/ingestion/migrations/002_qdrant_setup.ts that initialises both Qdrant collections:

• source_chunks (1536 dimensions, cosine distance)
• topic_summaries (1536 dimensions, cosine distance)

1.4 — Ingestion service scaffold Build the Fastify server with two routes:

• POST /ingest
• GET /status/:jobId

Use the file structure from ingestion-spec.md exactly.

1.5 — Stage 1: text extraction Implement extract.ts per ingestion-spec.md:

• TXT: direct UTF-8 read
• MD: direct UTF-8 read, preserve heading markers
• PDF: pdf-parse, page break markers

1.6 — Stage 2–3: chunking + cleaning Implement chunk.ts and clean.ts per ingestion-spec.md:

• MD: heading-based splitting
• TXT: sliding window (800 chars, 150 overlap)
• PDF: page + paragraph splitting
• Cleaning: whitespace, artefacts, minimum length filter

1.7 — Stage 4: structure extraction Implement structure.ts per ingestion-spec.md:

• Claude Sonnet 4 call with system + user prompt from spec
• Zod validation of DraftKB output
• Batch handling for documents > 60 chunks
• Retry logic on parse failure
• Error handling: failed job status + reason

1.8 — Stage 5: PocketBase write Implement the PocketBase write logic:

• Create Theme records (status: draft)
• Create Topic records under each Theme (status: draft)
• Resolve relationships between Topics after all records created

1.9 — Stage 6: embeddings + Qdrant write Implement embed.ts:

• OpenAI text-embedding-3-small, batches of 100
• Write to Qdrant source_chunks collection
• Write to Qdrant topic_summaries collection
• Update Topic.qdrant_chunk_ids in PocketBase

1.10 — Job status tracking Wire all stages into the job queue (jobs/queue.ts):

• Status transitions: queued → extracting → chunking → structuring → writing → embedding → done / failed
• Progress counters (chunksTotal, chunksEmbedded, themesFound, topicsFound)
• GET /status/:jobId returns current state

Acceptance criteria

• POST /ingest with a small MD file completes without error
• GET /status/:jobId returns done after processing
• PocketBase contains draft Theme + Topic records with correct hierarchy
• Topic.body contains AI-drafted content (not empty)
• Topic relationships are resolved (related_topics populated where applicable)
• Qdrant source_chunks contains vectors with correct payload fields
• Qdrant topic_summaries contains vectors for each published topic
• Topic.qdrant_chunk_ids is populated
• POST /ingest with a PDF file completes without error
• POST /ingest with a TXT file completes without error
• A document > 60 chunks triggers batch processing without error
• A malformed PDF returns status failed with reason, not an uncaught exception
• All Zod validations pass — no any types in codebase

---

Phase 2 — Generation service

Spec to read: /docs/generation-spec.md (write this spec before starting)

Steps

2.1 — Generation service scaffold Fastify service at app/services/generation/ Routes: POST /generate, GET /status/:jobId

2.2 — Generate all 10 types per topic One Claude Sonnet 4 call per type per topic. Structured JSON output validated against Zod schemas from data-model.md. Write to micro_learnings collection (status: generated).

2.3 — Batch generation on theme approval When admin approves a Theme batch, queue generation for all Topics in that Theme. All 10 types per Topic.

2.4 — Admin publish flow Route to update micro_learning status from generated → published or rejected. This is called by the admin app (built in phase 4).

Acceptance criteria (to be detailed in generation-spec.md)

• All 10 micro learning types generated for a test topic
• All 10 JSON outputs validate against their Zod schemas
• Generated content written to PocketBase with status: generated
• Admin can publish or reject individual micro learnings

---

Phase 3 — Curriculum service

Spec to read: /docs/curriculum-spec.md (write this spec before starting)

Steps

3.1 — Curriculum service scaffold Fastify service at app/services/curriculum/

3.2 — Curriculum generator Claude Sonnet 4 reads full KB graph → produces 26-week schedule. Written to curriculum_versions + curriculum_weeks.

3.3 — Versioning logic

• New version created on regeneration
• Completed weeks frozen (employee_curriculum_state.current_week used as boundary)
• Admin confirmation required before applying new version

3.4 — Perpetual cycling On week 26 completion, cycle increments, new cycle starts on latest version. Second cycle: varied sequence, surfaces unused micro learning types.

Acceptance criteria (to be detailed in curriculum-spec.md)

• Curriculum generated from a populated KB
• 26 weeks produced, all Themes covered
• Prerequisites respected in ordering
• Regeneration does not alter completed weeks
• Admin confirmation flow works correctly

---

Phase 4 — Frontend: admin app

Spec to read: /docs/frontend-spec.md (write this spec before starting)

Steps

4.1 — Next.js 14 scaffold Mobile-first, TypeScript strict, Tailwind CSS, PWA config. Role-based routing: /admin/* and /app/* from single Next.js codebase.

4.2 — Auth PocketBase auth integration. Admin role routes to /admin/*.

4.3 — Document upload + ingestion status Upload UI → calls ingestion service → polls job status → shows progress.

4.4 — Theme batch review Display draft Themes with their Topic list. Approve batch / edit individual topics / reject batch. Triggers generation service on approval.

4.5 — Curriculum editor Display AI-generated curriculum (26 weeks). Drag-to-reorder weeks. Edit Theme assignment per week. Confirm regeneration with preview.

Acceptance criteria (to be detailed in frontend-spec.md)

• Admin can upload a document and see ingestion progress
• Admin can approve a Theme batch
• Admin can edit a Topic before approval
• Admin can view and reorder the curriculum
• Admin can confirm a curriculum regeneration with preview

---

Phase 5 — Frontend: employee app

Spec to read: /docs/frontend-spec.md (same file, employee section)

Steps

5.1 — Employee auth + onboarding PocketBase auth. Employee role routes to /app/*. Set start date on first login → creates employee_curriculum_state record.

5.2 — Weekly session flow Current week's Theme displayed. Topics listed with available micro learning types per topic. Employee selects type → content rendered → mark complete.

5.3 — Knowledge library Browse all published Topics. Filter by Theme, difficulty, key terms.

5.4 — R42 chatbot Floating button, every screen. Calls chat service → streams response. Cites source topic in response.

5.5 — Gamification profile GitHub-style heatmap (26-week view). Badge display. Streak + level + commit count. Public leaderboard (multi-dimension). Milestone cards in activity feed.

Acceptance criteria (to be detailed in frontend-spec.md)

• Employee sees correct week based on start date
• Employee can complete a topic with a chosen micro learning type
• Completion is recorded and XP awarded
• Knowledge library shows all published topics with filters
• R42 responds with grounded answer and source citation
• R42 is accessible from every screen
• Heatmap renders correctly on mobile (375px)
• Leaderboard shows all employees with multi-dimension data

---

Phase 6 — Chat service (R42)

Spec to read: /docs/r42-spec.md (write this spec before starting)

Steps

6.1 — Chat service scaffold Fastify service at app/services/chat/

6.2 — Query → embed → retrieve Employee query embedded → Qdrant nearest-neighbour on both collections. Boost chunks from employee's current Theme.

6.3 — Response generation Top-K chunks injected into Haiku 4.5 prompt. Response streamed to frontend. Source Topic titles included in response.

6.4 — Out-of-scope handling If retrieval confidence is below threshold, R42 responds: "I can only answer questions based on the internal knowledge base. This topic doesn't appear to be covered."

Acceptance criteria (to be detailed in r42-spec.md)

• R42 answers a question about a published topic correctly
• R42 cites the source topic in its response
• R42 refuses to answer out-of-scope questions explicitly
• Response streams to frontend (not batch)
• Response latency < 3 seconds for typical queries

---

Phase 7 — Progress service

Spec to read: /docs/gamification-spec.md (write this spec before starting)

Steps

7.1 — Progress service scaffold Fastify service at app/services/progress/

7.2 — Completion recording Write session_completions record on topic completion. Calculate XP (commits) per type.

7.3 — Gamification updates Update gamification_profiles: commits, level, streak, types_used. Evaluate badge conditions → write employee_badges on award.

7.4 — Milestone cards Generate milestone_cards record at weeks 13 and 26.

7.5 — Leaderboard query Endpoint returning all gamification_profiles for leaderboard rendering.

Acceptance criteria (to be detailed in gamification-spec.md)

• Completion writes to session_completions
• Commits calculated and added to gamification_profile
• Level updates correctly at commit thresholds
• Streak increments on weekly completion, resets on skip
• Badge awarded when condition is met
• Milestone card created at weeks 13 and 26
• Leaderboard endpoint returns all employees with correct data

---

Phase 8 — Integration + hardening

No new spec required.

Steps

8.1 — Service wiring Verify all services communicate through PocketBase correctly. No direct service-to-service calls — all state through PocketBase.

8.2 — Error handling audit Review all services for unhandled promise rejections, missing error states, and uncaught exceptions. Every external call (AI API, PocketBase, Qdrant, OpenAI) wrapped in try/catch with meaningful error logging.

8.3 — Mobile QA Test all employee app flows at 375px width. R42 floating button must not obscure content. Heatmap must render without horizontal scroll.

8.4 — Environment variable audit Verify no hardcoded values. All .env.example files complete.

8.5 — Dockerfile update Update COPY path from legacy app root to /app. This is the one manual change that connects the rebuild to the existing pipeline.

Acceptance criteria

• Full flow works end-to-end: upload doc → approve → curriculum → employee completes session → R42 answers question → gamification updates
• No uncaught exceptions in any service under normal operating conditions
• All screens render correctly on 375px mobile
• Dockerfile builds successfully pointing at /app
• Existing pipeline deploys the rebuilt app without modification

---

Spec files still to be written

Before starting each phase, write the corresponding spec file. Use ingestion-spec.md as the template for structure and detail level.

Phase	Spec file needed
2	/docs/generation-spec.md
3	/docs/curriculum-spec.md
4–5	/docs/frontend-spec.md
6	/docs/r42-spec.md
7	/docs/gamification-spec.md

When you reach a phase without a spec: stop, draft the spec, then proceed. Do not implement without a spec.