Add comprehensive documentation for employee learning platform
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
This commit is contained in:
RaymondVerhoef
397
docs/data-model.md
Normal file
397
docs/data-model.md
Normal file
@@ -0,0 +1,397 @@
|
||||
# Data model: employee learning platform
|
||||
|
||||
## Overview
|
||||
|
||||
Two storage systems:
|
||||
- **PocketBase** — all structured relational data (SQLite under the hood)
|
||||
- **Qdrant** — all vector embeddings for RAG retrieval
|
||||
|
||||
---
|
||||
|
||||
## PocketBase collections
|
||||
|
||||
### `source_documents`
|
||||
Uploaded source files. Parent of all generated KB content.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| filename | string | original filename |
|
||||
| file | file | PocketBase file storage |
|
||||
| format | select | `pdf` `md` `txt` |
|
||||
| status | select | `processing` `processed` `failed` |
|
||||
| ingested_at | datetime | |
|
||||
| chunk_count | number | total chunks extracted |
|
||||
| created_by | relation → `users` | admin who uploaded |
|
||||
|
||||
---
|
||||
|
||||
### `themes`
|
||||
Top-level content groupings. One Theme = one weekly session.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| title | string | |
|
||||
| description | text | AI drafted, admin editable |
|
||||
| status | select | `draft` `published` |
|
||||
| source_documents | relation[] → `source_documents` | which docs contributed |
|
||||
| approved_by | relation → `users` | admin who approved batch |
|
||||
| approved_at | datetime | |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `topics`
|
||||
Atomic knowledge units. Always belong to a Theme.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| theme | relation → `themes` | parent theme |
|
||||
| title | string | |
|
||||
| body | text (rich) | AI drafted, admin editable |
|
||||
| difficulty | select | `introductory` `intermediate` `advanced` |
|
||||
| complexity_weight | number | 1–5, used by curriculum generator |
|
||||
| status | select | `draft` `published` |
|
||||
| related_topics | relation[] → `topics` | lateral relationships |
|
||||
| prerequisite_topics | relation[] → `topics` | must-complete-first |
|
||||
| contrast_topics | relation[] → `topics` | deliberate opposites |
|
||||
| key_terms | json | string[] — feeds glossary |
|
||||
| qdrant_chunk_ids | json | string[] — references to embedded chunks |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale.
|
||||
|
||||
---
|
||||
|
||||
### `micro_learnings`
|
||||
Generated content artifacts. One record per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| topic | relation → `topics` | |
|
||||
| type | select | see type enum below |
|
||||
| content | json | structured output — schema varies per type |
|
||||
| status | select | `queued` `generated` `published` `rejected` |
|
||||
| generation_model | string | model version used |
|
||||
| generated_at | datetime | |
|
||||
| published_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
**Type enum:**
|
||||
`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence`
|
||||
|
||||
**Content JSON schemas per type:**
|
||||
|
||||
```json
|
||||
// concept_explainer
|
||||
{
|
||||
"paragraphs": ["string", "string"],
|
||||
"example": "string"
|
||||
}
|
||||
|
||||
// scenario_quiz
|
||||
{
|
||||
"scenario": "string",
|
||||
"options": [
|
||||
{ "label": "A", "text": "string", "correct": true, "explanation": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// misconceptions
|
||||
{
|
||||
"items": [
|
||||
{ "misconception": "string", "correction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// how_to
|
||||
{
|
||||
"steps": [
|
||||
{ "number": 1, "instruction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// comparison_card
|
||||
{
|
||||
"subject_a": "string",
|
||||
"subject_b": "string",
|
||||
"dimensions": [
|
||||
{ "label": "string", "a": "string", "b": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// reflection_prompt
|
||||
{
|
||||
"prompt": "string",
|
||||
"model_answer": "string"
|
||||
}
|
||||
|
||||
// flashcard_set
|
||||
{
|
||||
"cards": [
|
||||
{ "question": "string", "answer": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// case_study
|
||||
{
|
||||
"scenario": "string",
|
||||
"questions": ["string"]
|
||||
}
|
||||
|
||||
// glossary_anchor
|
||||
{
|
||||
"term": "string",
|
||||
"definition": "string",
|
||||
"correct_use": "string",
|
||||
"misuse": "string"
|
||||
}
|
||||
|
||||
// myth_vs_evidence
|
||||
{
|
||||
"myth": "string",
|
||||
"evidence": "string",
|
||||
"sources": ["string"]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_versions`
|
||||
Versioned 26-week schedule. New version created on each regeneration.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| version | number | increments on each regeneration |
|
||||
| status | select | `draft` `active` `superseded` |
|
||||
| generated_at | datetime | |
|
||||
| approved_by | relation → `users` | admin who confirmed regeneration |
|
||||
| approved_at | datetime | |
|
||||
| generation_notes | text | why this version was created |
|
||||
|
||||
Only one version has status `active` at any time.
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_weeks`
|
||||
Individual week slots. Child of a curriculum version.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| curriculum_version | relation → `curriculum_versions` | |
|
||||
| week_number | number | 1–26 |
|
||||
| theme | relation → `themes` | |
|
||||
| topics | relation[] → `topics` | ordered subset of theme topics |
|
||||
| topic_order | json | number[] — explicit ordering |
|
||||
| estimated_duration_minutes | number | AI estimate |
|
||||
| admin_notes | text | freeform admin annotation |
|
||||
|
||||
---
|
||||
|
||||
### `employee_curriculum_state`
|
||||
Tracks each employee's position in the curriculum. One record per employee.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| current_cycle | number | starts at 1, increments on loop |
|
||||
| current_week | number | 1–26 |
|
||||
| start_date | datetime | rolling start |
|
||||
| active_version | relation → `curriculum_versions` | version employee is on |
|
||||
| updated_at | datetime | |
|
||||
|
||||
When curriculum regenerates: `active_version` updates for all employees whose `current_week` is less than the first regenerated week.
|
||||
|
||||
---
|
||||
|
||||
### `session_completions`
|
||||
Immutable completion records. One record per employee per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| topic | relation → `topics` | |
|
||||
| micro_learning | relation → `micro_learnings` | specific type completed |
|
||||
| week_number | number | curriculum week at time of completion |
|
||||
| cycle | number | which cycle |
|
||||
| completed_at | datetime | |
|
||||
|
||||
Records are never updated or deleted. This is the canonical history.
|
||||
|
||||
---
|
||||
|
||||
### `gamification_profiles`
|
||||
One record per employee. Updated by progress service on each completion.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| total_commits | number | cumulative XP |
|
||||
| current_level | select | `intern` `junior` `medior` `senior` `staff` `principal` |
|
||||
| current_streak_weeks | number | consecutive weeks with ≥1 completion |
|
||||
| longest_streak_weeks | number | all-time high |
|
||||
| types_used | json | string[] — which of 10 types used at least once |
|
||||
| last_active_week | number | used to detect streak breaks |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `badges`
|
||||
Badge definitions. Seeded at startup, not user-generated.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| key | string | unique slug e.g. `governance_nerd` |
|
||||
| tier | select | `bronze` `silver` `gold` `legendary` `content` |
|
||||
| label | string | display name |
|
||||
| description | string | award condition description |
|
||||
| icon | string | emoji or icon key |
|
||||
|
||||
---
|
||||
|
||||
### `employee_badges`
|
||||
Junction: which employees have earned which badges.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| badge | relation → `badges` | |
|
||||
| earned_at | datetime | |
|
||||
| cycle | number | which cycle it was earned in |
|
||||
|
||||
---
|
||||
|
||||
### `milestone_cards`
|
||||
Public milestone events at weeks 13 and 26.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| cycle | number | |
|
||||
| week | number | 13 or 26 |
|
||||
| total_commits | number | snapshot at time of milestone |
|
||||
| streak_weeks | number | snapshot |
|
||||
| badge_keys | json | string[] — badges held at milestone |
|
||||
| created_at | datetime | public feed ordered by this |
|
||||
|
||||
---
|
||||
|
||||
## PocketBase users collection (extended)
|
||||
|
||||
Standard PocketBase `users` collection with additional fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| role | select | `admin` `employee` |
|
||||
| display_name | string | used in gamification feed |
|
||||
| avatar | file | optional |
|
||||
|
||||
---
|
||||
|
||||
## Qdrant collections
|
||||
|
||||
### `source_chunks`
|
||||
Embeddings of raw source document chunks. Primary retrieval target for R42.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions (text-embedding-3-small) |
|
||||
| source_document_id | string | reference to PocketBase |
|
||||
| chunk_index | number | position within document |
|
||||
| text | string | raw chunk text |
|
||||
| theme_id | string | assigned theme (post-extraction) |
|
||||
| topic_id | string | assigned topic (post-extraction, nullable) |
|
||||
| format | string | pdf / md / txt |
|
||||
|
||||
### `topic_summaries`
|
||||
Embeddings of AI-generated topic body text. Secondary retrieval target.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions |
|
||||
| topic_id | string | reference to PocketBase |
|
||||
| theme_id | string | |
|
||||
| title | string | for display in R42 citation |
|
||||
| text | string | full topic body |
|
||||
|
||||
---
|
||||
|
||||
## Retrieval strategy for R42
|
||||
|
||||
R42 queries both Qdrant collections and merges results:
|
||||
|
||||
```
|
||||
Employee query
|
||||
↓
|
||||
Embed query → text-embedding-3-small
|
||||
↓
|
||||
Qdrant search: source_chunks (top 5) + topic_summaries (top 3)
|
||||
↓
|
||||
Filter: boost chunks from employee's current week theme
|
||||
↓
|
||||
Merge + deduplicate by topic_id
|
||||
↓
|
||||
Top-K context injected into Haiku 4.5 prompt
|
||||
↓
|
||||
Response includes: answer + cited topic title(s)
|
||||
```
|
||||
|
||||
Source chunks are weighted higher than topic summaries to keep R42 grounded in original source material rather than AI-generated abstractions.
|
||||
|
||||
---
|
||||
|
||||
## Indexes and query patterns
|
||||
|
||||
Critical query patterns the data model must support efficiently:
|
||||
|
||||
| Query | Collection | Index |
|
||||
|---|---|---|
|
||||
| All published topics for a theme | topics | theme + status |
|
||||
| All micro learnings for a topic | micro_learnings | topic + status |
|
||||
| Employee's current week | employee_curriculum_state | user |
|
||||
| Weeks for a curriculum version | curriculum_weeks | curriculum_version + week_number |
|
||||
| Employee completion history | session_completions | user + cycle |
|
||||
| Public leaderboard | gamification_profiles | total_commits + streak |
|
||||
| Milestone feed | milestone_cards | created_at DESC |
|
||||
| Badges earned by employee | employee_badges | user |
|
||||
|
||||
PocketBase creates indexes on relation fields by default. Composite indexes on `status` fields should be added manually where query frequency warrants it.
|
||||
|
||||
---
|
||||
|
||||
## Data flow summary
|
||||
|
||||
```
|
||||
source_documents
|
||||
└── (ingestion service)
|
||||
└── qdrant: source_chunks
|
||||
└── themes (draft)
|
||||
└── topics (draft)
|
||||
└── (approval)
|
||||
└── topics (published)
|
||||
└── qdrant: topic_summaries
|
||||
└── micro_learnings (queued → published)
|
||||
└── (curriculum service)
|
||||
└── curriculum_versions
|
||||
└── curriculum_weeks
|
||||
└── (employee progress)
|
||||
└── session_completions
|
||||
└── gamification_profiles
|
||||
└── employee_badges
|
||||
└── milestone_cards
|
||||
```
|
||||
Reference in New Issue
Block a user