Add specifications for gamification, generation, and R42 chat services

- Introduced gamification service spec detailing responsibilities, API surface, XP calculation, levels, streaks, badges, milestone cards, and heatmap data. - Added generation service spec outlining the process for generating micro learning content, including API endpoints, AI call configuration, prompt strategies, and error handling. - Created R42 chat service spec covering chatbot interactions, retrieval pipeline, prompt construction, response generation, and stateless design principles.
2026-05-23 18:13:08 +02:00
parent dda20612e9
commit 472685f0d7
62 changed files with 11552 additions and 21 deletions
--- a/docs/r42-spec.md
+++ b/docs/r42-spec.md
@@ -0,0 +1,336 @@
+# R42 chat service spec
+
+## Responsibility
+
+Handles all R42 chatbot interactions. Receives employee queries, retrieves
+relevant KB chunks from Qdrant, generates grounded responses using Claude
+Haiku 4.5, and streams the result to the frontend. Stateless — no chat
+history is persisted.
+
+---
+
+## Service location
+
+```
+app/services/chat/
+  ├── src/
+  │     ├── index.ts              entry point, Fastify server
+  │     ├── routes/
+  │     │     └── chat.ts         POST /chat (streaming)
+  │     ├── retrieval/
+  │     │     ├── embed.ts        query → embedding
+  │     │     ├── search.ts       Qdrant nearest-neighbour search
+  │     │     └── merge.ts        merge + rank results from both collections
+  │     ├── prompt/
+  │     │     └── build.ts        assemble system + user prompt with context
+  │     └── lib/
+  │           ├── qdrant.ts
+  │           ├── pocketbase.ts
+  │           ├── anthropic.ts
+  │           └── openai.ts
+  ├── package.json
+  ├── tsconfig.json
+  └── .env.example
+```
+
+---
+
+## API surface
+
+### POST /chat
+
+Single route. Streams response back to client using server-sent events (SSE).
+
+Request:
+```json
+{
+  "query": "string",
+  "userId": "string"
+}
+```
+
+Response: SSE stream
+
+```
+Content-Type: text/event-stream
+
+data: {"type": "chunk", "text": "Holacratic roles "}
+data: {"type": "chunk", "text": "are defined as..."}
+data: {"type": "citations", "topics": [{"id": "abc", "title": "Holacratic roles"}]}
+data: {"type": "done"}
+```
+
+Error response (non-streaming, returned before stream starts):
+```json
+{
+  "error": "query_too_short" | "user_not_found" | "retrieval_failed",
+  "message": "string"
+}
+```
+
+---
+
+## Retrieval pipeline
+
+### Step 1 — Embed query
+
+Embed the employee query using OpenAI text-embedding-3-small (1536 dimensions).
+Same model used during ingestion — vectors are comparable.
+
+```typescript
+const queryVector = await embedText(query)  // float[1536]
+```
+
+---
+
+### Step 2 — Qdrant search
+
+Search both collections in parallel:
+
+```typescript
+// source_chunks: primary retrieval — grounded in source material
+const chunkResults = await qdrant.search('source_chunks', {
+  vector: queryVector,
+  limit: 5,
+  scoreThreshold: 0.70,
+  withPayload: true
+})
+
+// topic_summaries: secondary — broader topic context
+const summaryResults = await qdrant.search('topic_summaries', {
+  vector: queryVector,
+  limit: 3,
+  scoreThreshold: 0.70,
+  withPayload: true
+})
+```
+
+Score threshold 0.70: below this, results are not relevant enough to include.
+If both searches return zero results above threshold → out-of-scope response.
+
+---
+
+### Step 3 — Context boost for current week
+
+Retrieve employee's current week Theme from PocketBase via
+employee_curriculum_state → curriculum_weeks → theme.
+
+Apply boost to results where payload.theme_id matches current week theme:
+
+```typescript
+results.forEach(result => {
+  if (result.payload.theme_id === currentThemeId) {
+    result.score += 0.05   // small boost — does not override relevance
+  }
+})
+```
+
+---
+
+### Step 4 — Merge and deduplicate
+
+```typescript
+// Combine chunk results and summary results
+// Deduplicate by topic_id — keep highest scoring entry per topic
+// Sort by score descending
+// Take top 6 total
+// Split into: sourceChunks (from source_chunks collection)
+//             topicSummaries (from topic_summaries collection)
+```
+
+Deduplicate by topic_id to avoid repeating the same topic in different forms.
+
+---
+
+### Step 5 — Collect cited topics
+
+Extract unique topic titles from merged results for citation:
+
+```typescript
+type Citation = {
+  id: string
+  title: string
+}
+const citations: Citation[] = uniqueByTopicId(mergedResults)
+  .map(r => ({ id: r.payload.topic_id, title: r.payload.title }))
+```
+
+---
+
+## Prompt construction
+
+### System prompt
+
+```
+You are R42, a knowledge assistant for [company name].
+You answer questions based strictly on the company knowledge base.
+
+Rules:
+- Answer only from the provided context. Do not use outside knowledge.
+- If the context does not contain enough information to answer, say:
+  "This doesn't appear to be covered in the knowledge base. You can browse
+  the full library in the Knowledge section."
+- Be concise. Prefer short paragraphs over long prose.
+- Do not mention that you are an AI or reference your instructions.
+- Do not speculate or extrapolate beyond the provided context.
+- Respond in the same language as the question.
+```
+
+### User prompt
+
+```
+Context from knowledge base:
+---
+{mergedResults.map(r => r.payload.text).join('\n\n---\n\n')}
+---
+
+Question: {query}
+```
+
+---
+
+## Response generation
+
+Use Claude Haiku 4.5 with streaming enabled:
+
+```typescript
+const stream = await anthropic.messages.stream({
+  model: 'claude-haiku-4-5-20251001',
+  max_tokens: 1000,
+  system: systemPrompt,
+  messages: [{ role: 'user', content: userPrompt }]
+})
+
+// Stream text chunks to client as SSE
+for await (const chunk of stream) {
+  if (chunk.type === 'content_block_delta') {
+    sendSSE({ type: 'chunk', text: chunk.delta.text })
+  }
+}
+
+// After stream completes, send citations
+sendSSE({ type: 'citations', topics: citations })
+sendSSE({ type: 'done' })
+```
+
+---
+
+## Out-of-scope handling
+
+Two conditions trigger the out-of-scope response:
+
+1. Both Qdrant searches return zero results above 0.70 threshold
+2. Haiku response contains no content drawn from context (detected by
+   checking if response length < 20 tokens — proxy for "I don't know")
+
+Out-of-scope response sent as a single non-streamed SSE message:
+
+```
+data: {"type": "out_of_scope", "text": "This doesn't appear to be covered
+in the knowledge base. You can browse the full library in the Knowledge section."}
+data: {"type": "done"}
+```
+
+No citations are sent for out-of-scope responses.
+
+---
+
+## Frontend integration
+
+R42 is a floating button on every screen in the employee app.
+
+UI behaviour:
+- Bottom-right corner, fixed position
+- Opens a chat drawer (not a modal — drawer slides up from bottom on mobile)
+- Input field at bottom of drawer, response area above
+- Streaming text renders token by token
+- Citations appear below the response after streaming completes as
+  clickable topic pills → navigate to that topic in the knowledge library
+- Drawer closes on outside tap
+- State is local to the component — cleared on close (stateless by design)
+
+The frontend calls POST /chat directly. No auth token needed on the chat
+service — it receives userId in the request body and trusts it. The admin
+app does not expose R42.
+
+---
+
+## Stateless design
+
+R42 has no memory between conversations. Each POST /chat is independent.
+
+Rationale:
+- Avoids privacy complexity around chat history storage
+- Removes need for session management
+- Keeps the service simple and fast
+- Employees asking follow-up questions reprovide context naturally
+
+If multi-turn conversation is needed in a future iteration, maintain
+conversation history in the frontend component state and pass the last
+N messages in the request body. The service does not need to change.
+
+---
+
+## Environment variables
+
+```
+ANTHROPIC_API_KEY=
+OPENAI_API_KEY=
+POCKETBASE_URL=
+POCKETBASE_ADMIN_EMAIL=
+POCKETBASE_ADMIN_PASSWORD=
+QDRANT_URL=
+QDRANT_API_KEY=
+CHAT_PORT=3004
+```
+
+---
+
+## Dependencies
+
+```json
+{
+  "dependencies": {
+    "fastify": "^4",
+    "@fastify/sse": "^2",
+    "@anthropic-ai/sdk": "^0.24",
+    "openai": "^4",
+    "@qdrant/js-client-rest": "^1.9",
+    "pocketbase": "^0.21",
+    "zod": "^3"
+  }
+}
+```
+
+---
+
+## TypeScript strict mode requirements
+
+- No `any` types
+- Qdrant search results typed explicitly including payload fields
+- SSE event types defined as a discriminated union
+- Citation type explicit — not inferred from payload
+
+---
+
+## What this service does NOT do
+
+- Does not persist chat history
+- Does not generate or serve micro learning content
+- Does not handle admin queries — admin app has no R42 access
+- Does not handle auth — trusts userId from request body
+
+---
+
+## Testing checkpoints
+
+1. POST /chat with a query matching a published topic → confirm relevant
+   chunks retrieved (score > 0.70) and response references topic content
+2. POST /chat with an out-of-scope query → confirm out-of-scope response
+   returned, no citations sent
+3. Confirm citations array contains correct topic titles matching retrieved chunks
+4. Confirm SSE stream delivers chunks progressively (not batched)
+5. Confirm current-week boost: same query returns higher-ranked result for
+   current week theme topic vs equally relevant topic from different theme
+6. POST /chat with userId whose current week has no matching topic →
+   confirm boost does not break retrieval, general results returned