Add comprehensive documentation for employee learning platform
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
This commit is contained in:
RaymondVerhoef
483
docs/ingestion-spec.md
Normal file
483
docs/ingestion-spec.md
Normal file
@@ -0,0 +1,483 @@
|
||||
# Ingestion service spec
|
||||
|
||||
## Responsibility
|
||||
|
||||
Accepts uploaded source documents (PDF, MD, TXT), extracts clean text, chunks it,
|
||||
generates embeddings, and produces a structured draft KB (Themes + Topics +
|
||||
relationships) ready for admin review.
|
||||
|
||||
This service runs entirely server-side. The admin app calls it via REST. All AI
|
||||
calls go through the Anthropic API. No ingestion logic lives in the frontend.
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
|
||||
```
|
||||
app/services/ingestion/
|
||||
├── index.ts entry point, Fastify server
|
||||
├── routes/
|
||||
│ └── documents.ts POST /ingest, GET /status/:jobId
|
||||
├── pipeline/
|
||||
│ ├── extract.ts format detection + text extraction
|
||||
│ ├── chunk.ts chunking strategies per format
|
||||
│ ├── clean.ts chunk cleaning
|
||||
│ ├── structure.ts Claude call → Theme/Topic extraction
|
||||
│ └── embed.ts embedding generation + Qdrant write
|
||||
├── jobs/
|
||||
│ └── queue.ts async job queue (in-memory, BullMQ later if needed)
|
||||
├── lib/
|
||||
│ ├── pocketbase.ts PocketBase client
|
||||
│ ├── qdrant.ts Qdrant client
|
||||
│ ├── anthropic.ts Anthropic client
|
||||
│ └── openai.ts OpenAI embeddings client
|
||||
└── types.ts shared TypeScript types
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
|
||||
### POST /ingest
|
||||
|
||||
Triggered by admin app on document upload.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"documentId": "string",
|
||||
"filename": "string",
|
||||
"format": "pdf" | "md" | "txt",
|
||||
"filePath": "string"
|
||||
}
|
||||
```
|
||||
|
||||
Response (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued"
|
||||
}
|
||||
```
|
||||
|
||||
Processing is async. The admin app polls job status.
|
||||
|
||||
---
|
||||
|
||||
### GET /status/:jobId
|
||||
|
||||
Returns current job progress.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued" | "extracting" | "chunking" | "structuring" | "embedding" | "done" | "failed",
|
||||
"progress": {
|
||||
"chunksTotal": 42,
|
||||
"chunksEmbedded": 18,
|
||||
"themesFound": 3,
|
||||
"topicsFound": 14
|
||||
},
|
||||
"error": "string | null"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
### Stage 1 — Text extraction
|
||||
|
||||
Input: file path + format
|
||||
Output: raw text string
|
||||
|
||||
```
|
||||
format === 'txt'
|
||||
→ read file directly as UTF-8
|
||||
|
||||
format === 'md'
|
||||
→ read file directly as UTF-8
|
||||
→ preserve heading markers (# ## ###) — used in chunking
|
||||
|
||||
format === 'pdf'
|
||||
→ pdfplumber: extract text page by page
|
||||
→ concatenate with page break markers: \n\n---PAGE---\n\n
|
||||
→ strip known PDF artefacts: headers/footers repeating on every page,
|
||||
page numbers, watermarks
|
||||
```
|
||||
|
||||
Failure handling:
|
||||
- PDF extraction returns empty string → mark job `failed`, reason: `pdf_extraction_empty`
|
||||
- File not found → mark job `failed`, reason: `file_not_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 2 — Chunking
|
||||
|
||||
Input: raw text + format
|
||||
Output: Chunk[]
|
||||
|
||||
Chunking strategy differs per format.
|
||||
|
||||
**MD chunking — heading-based (preferred)**
|
||||
```
|
||||
Split on heading markers: #, ##, ###
|
||||
Each heading + its following content = one chunk
|
||||
Minimum chunk size: 100 characters
|
||||
→ if heading section is < 100 chars, merge with next sibling
|
||||
Maximum chunk size: 1500 characters
|
||||
→ if section exceeds limit, split on paragraph breaks within section
|
||||
Metadata preserved per chunk:
|
||||
heading_level: 1 | 2 | 3
|
||||
heading_text: string
|
||||
parent_heading: string | null
|
||||
```
|
||||
|
||||
MD chunking produces the highest quality structural signal for Theme/Topic extraction.
|
||||
Admins should be advised to provide source material as MD where possible.
|
||||
|
||||
**TXT chunking — sliding window**
|
||||
```
|
||||
Window size: 800 characters
|
||||
Overlap: 150 characters
|
||||
Split on: paragraph breaks (\n\n) first, then sentence boundaries, then hard cut
|
||||
Metadata per chunk:
|
||||
chunk_index: number
|
||||
approximate_position: 'start' | 'middle' | 'end'
|
||||
```
|
||||
|
||||
**PDF chunking — page + paragraph**
|
||||
```
|
||||
Split on ---PAGE--- markers from extraction stage
|
||||
Within each page: split on paragraph breaks (\n\n)
|
||||
Minimum chunk size: 100 characters
|
||||
→ merge sub-threshold paragraphs with adjacent chunk
|
||||
Maximum chunk size: 1200 characters
|
||||
→ hard split at sentence boundary
|
||||
Metadata per chunk:
|
||||
page_number: number
|
||||
chunk_index_on_page: number
|
||||
```
|
||||
|
||||
**Chunk type:**
|
||||
```typescript
|
||||
type Chunk = {
|
||||
id: string // UUID generated at chunking
|
||||
documentId: string
|
||||
text: string
|
||||
format: 'pdf' | 'md' | 'txt'
|
||||
index: number // global position in document
|
||||
metadata: {
|
||||
// MD-specific
|
||||
headingLevel?: number
|
||||
headingText?: string
|
||||
parentHeading?: string
|
||||
// TXT-specific
|
||||
approximatePosition?: 'start' | 'middle' | 'end'
|
||||
// PDF-specific
|
||||
pageNumber?: number
|
||||
chunkIndexOnPage?: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 3 — Chunk cleaning
|
||||
|
||||
Input: Chunk[]
|
||||
Output: Chunk[] (cleaned)
|
||||
|
||||
Applied to all formats:
|
||||
```
|
||||
- trim leading/trailing whitespace
|
||||
- collapse 3+ consecutive newlines to 2
|
||||
- remove null bytes and non-printable characters
|
||||
- remove chunks where text.length < 80 after cleaning
|
||||
→ these are likely artefacts (page numbers, standalone headers)
|
||||
- normalise unicode: NFC normalisation
|
||||
- do not strip punctuation or alter sentence structure
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 4 — Structure extraction (AI)
|
||||
|
||||
Input: Chunk[]
|
||||
Output: DraftKB
|
||||
|
||||
This is the core AI call. Claude Sonnet 4 reads all chunks and returns a structured
|
||||
KB draft as JSON.
|
||||
|
||||
**Prompt strategy:**
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a knowledge architect. Your task is to analyse a set of text chunks from
|
||||
a source document and extract a structured knowledge base.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
|
||||
no markdown fences.
|
||||
|
||||
Rules:
|
||||
- Group related content into Themes. A Theme is a broad subject area.
|
||||
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
|
||||
- Identify relationships between Topics: related, prerequisite, or contrast.
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A must be understood before Topic B
|
||||
- contrast: Topics that represent opposing approaches or concepts
|
||||
- For each Topic, extract key terms suitable for a glossary.
|
||||
- Assign a complexity weight (1–5) to each Topic.
|
||||
1 = introductory, 5 = advanced
|
||||
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
|
||||
- Draft a description for each Theme (1–2 sentences).
|
||||
- Every Topic must reference the chunk IDs that contributed to it.
|
||||
```
|
||||
|
||||
User prompt:
|
||||
```
|
||||
Source document: {filename}
|
||||
Format: {format}
|
||||
|
||||
Chunks:
|
||||
{chunks mapped as: [CHUNK-{id}]\n{text}\n}
|
||||
|
||||
Extract the knowledge base structure from these chunks.
|
||||
```
|
||||
|
||||
**Output schema:**
|
||||
```typescript
|
||||
type DraftKB = {
|
||||
themes: DraftTheme[]
|
||||
}
|
||||
|
||||
type DraftTheme = {
|
||||
title: string
|
||||
description: string
|
||||
topics: DraftTopic[]
|
||||
}
|
||||
|
||||
type DraftTopic = {
|
||||
title: string
|
||||
body: string
|
||||
difficulty: 'introductory' | 'intermediate' | 'advanced'
|
||||
complexityWeight: number // 1–5
|
||||
keyTerms: string[]
|
||||
sourceChunkIds: string[] // references Chunk.id values
|
||||
relationships: {
|
||||
related: string[] // topic titles (resolved to IDs after write)
|
||||
prerequisites: string[]
|
||||
contrasts: string[]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**AI call configuration:**
|
||||
```typescript
|
||||
{
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
max_tokens: 8000,
|
||||
temperature: 0 // deterministic output for structured extraction
|
||||
}
|
||||
```
|
||||
|
||||
**Chunking strategy for large documents:**
|
||||
If total chunk count exceeds 60 chunks, split into batches of 40 with 5-chunk
|
||||
overlap. Run one Claude call per batch. Merge resulting DraftKB objects:
|
||||
- Themes with identical titles → merge Topics
|
||||
- Duplicate Topic titles within a Theme → keep longer body, merge sourceChunkIds
|
||||
- Relationships are resolved after full merge
|
||||
|
||||
**Error handling:**
|
||||
- JSON parse failure → retry once with stricter prompt ("ensure valid JSON only")
|
||||
- Second failure → mark job `failed`, reason: `structure_extraction_failed`, log raw response
|
||||
- Empty themes array → mark job `failed`, reason: `no_structure_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 5 — Write to PocketBase
|
||||
|
||||
Input: DraftKB
|
||||
Output: written Theme + Topic records with status `draft`
|
||||
|
||||
```
|
||||
For each DraftTheme:
|
||||
create themes record {
|
||||
title, description,
|
||||
status: 'draft',
|
||||
source_documents: [documentId]
|
||||
}
|
||||
|
||||
For each DraftTopic under the theme:
|
||||
create topics record {
|
||||
theme: themeId,
|
||||
title, body, difficulty, complexity_weight, key_terms,
|
||||
status: 'draft',
|
||||
qdrant_chunk_ids: [] // populated in stage 6
|
||||
}
|
||||
|
||||
After all topics created:
|
||||
resolve relationship titles → topic IDs
|
||||
update topics.related_topics, prerequisite_topics, contrast_topics
|
||||
|
||||
If a relationship title cannot be resolved to an existing topic:
|
||||
skip silently (cross-document relationships resolved in a later pass)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 6 — Embedding generation + Qdrant write
|
||||
|
||||
Input: Chunk[], written Topic records
|
||||
Output: vectors in Qdrant, qdrant_chunk_ids updated on Topic records
|
||||
|
||||
**Source chunk embeddings:**
|
||||
```
|
||||
For each Chunk (post-cleaning):
|
||||
embed Chunk.text → text-embedding-3-small (1536 dimensions)
|
||||
write to Qdrant collection: source_chunks {
|
||||
id: Chunk.id,
|
||||
vector: float[],
|
||||
payload: {
|
||||
source_document_id: documentId,
|
||||
chunk_index: Chunk.index,
|
||||
text: Chunk.text,
|
||||
theme_id: resolved themeId | null,
|
||||
topic_id: resolved topicId | null,
|
||||
format: Chunk.format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Topic summary embeddings:**
|
||||
```
|
||||
For each published Topic:
|
||||
embed Topic.body → text-embedding-3-small
|
||||
write to Qdrant collection: topic_summaries {
|
||||
id: UUID,
|
||||
vector: float[],
|
||||
payload: {
|
||||
topic_id: Topic.id,
|
||||
theme_id: Topic.theme,
|
||||
title: Topic.title,
|
||||
text: Topic.body
|
||||
}
|
||||
}
|
||||
|
||||
Update Topic.qdrant_chunk_ids with all Chunk.ids that reference this topic
|
||||
```
|
||||
|
||||
**Batching:**
|
||||
OpenAI embeddings API: batch in groups of 100 texts per request to stay within
|
||||
rate limits and reduce latency.
|
||||
|
||||
---
|
||||
|
||||
## Job lifecycle
|
||||
|
||||
```
|
||||
POST /ingest received
|
||||
↓
|
||||
Job created → status: queued
|
||||
↓
|
||||
Stage 1: extracting
|
||||
↓
|
||||
Stage 2–3: chunking
|
||||
↓
|
||||
Stage 4: structuring
|
||||
↓
|
||||
Stage 5: writing to PocketBase
|
||||
↓
|
||||
Stage 6: embedding
|
||||
↓
|
||||
status: done
|
||||
↓
|
||||
Admin notification: "Document processed. N themes, N topics ready for review."
|
||||
↓
|
||||
Curriculum regeneration queued (status: pending_admin_confirm)
|
||||
```
|
||||
|
||||
On any stage failure:
|
||||
```
|
||||
status: failed
|
||||
error: { stage, reason, detail }
|
||||
Source document status → 'failed' in PocketBase
|
||||
Admin notification: "Ingestion failed: {reason}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables required
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
QDRANT_URL=
|
||||
QDRANT_API_KEY= # empty string if running locally without auth
|
||||
INGESTION_PORT=3001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"openai": "^4",
|
||||
"@qdrant/js-client-rest": "^1.9",
|
||||
"pocketbase": "^0.21",
|
||||
"pdfplumber": "NOT JS — see note below",
|
||||
"pdf-parse": "^1.1",
|
||||
"uuid": "^9",
|
||||
"zod": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**PDF extraction note:**
|
||||
`pdfplumber` is a Python library. Two options:
|
||||
1. Use `pdf-parse` (Node.js) — simpler, covers 90% of cases
|
||||
2. Run `pdfplumber` as a Python sidecar process via child_process — higher quality
|
||||
for complex PDFs with tables and columns
|
||||
|
||||
Default to `pdf-parse` initially. Add pdfplumber sidecar only if extraction
|
||||
quality is insufficient for actual source documents.
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- All Claude response parsing through Zod schema validation
|
||||
- All PocketBase writes typed against collection schemas from `data-model.md`
|
||||
- Qdrant payloads typed explicitly — no untyped objects
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not generate micro learnings → generation service
|
||||
- Does not build or update the curriculum → curriculum service
|
||||
- Does not handle admin approval → admin app + PocketBase directly
|
||||
- Does not serve R42 queries → chat service
|
||||
- Does not handle auth → PocketBase + admin app
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
Before handing to Claude Code for implementation, verify manually:
|
||||
|
||||
1. Upload a short MD file (< 10 headings) → inspect chunk output → confirm heading structure preserved
|
||||
2. Upload a simple PDF (< 5 pages) → inspect chunk output → confirm no artefacts
|
||||
3. Run structure extraction on known chunks → validate JSON parses against Zod schema
|
||||
4. Confirm PocketBase draft records created with correct theme → topic hierarchy
|
||||
5. Confirm Qdrant source_chunks collection populated with correct payload fields
|
||||
6. Confirm topic.qdrant_chunk_ids updated after embedding stage
|
||||
Reference in New Issue
Block a user