learning-platform/docs/ingestion-spec.md

# Ingestion service spec

## Responsibility

Accepts uploaded source documents (PDF, MD, TXT), extracts clean text, chunks it,
generates embeddings, and produces a structured draft KB (Themes + Topics +
relationships) ready for admin review.

This service runs entirely server-side. The admin app calls it via REST. All AI
calls go through the Anthropic API. No ingestion logic lives in the frontend.

---

## Service location

```
app/services/ingestion/
  ├── index.ts              entry point, Fastify server
  ├── routes/
  │     └── documents.ts    POST /ingest, GET /status/:jobId
  ├── pipeline/
  │     ├── extract.ts      format detection + text extraction
  │     ├── chunk.ts        chunking strategies per format
  │     ├── clean.ts        chunk cleaning
  │     ├── structure.ts    Claude call → Theme/Topic extraction
  │     └── embed.ts        embedding generation + Qdrant write
  ├── jobs/
  │     └── queue.ts        async job queue (in-memory, BullMQ later if needed)
  ├── lib/
  │     ├── pocketbase.ts   PocketBase client
  │     ├── qdrant.ts       Qdrant client
  │     ├── anthropic.ts    Anthropic client
  │     └── openai.ts       OpenAI embeddings client
  └── types.ts              shared TypeScript types
```

---

## API surface

### POST /ingest

Triggered by admin app on document upload.

Request:
```json
{
  "documentId": "string",
  "filename": "string",
  "format": "pdf" | "md" | "txt",
  "filePath": "string"
}
```

Response (202 Accepted):
```json
{
  "jobId": "string",
  "status": "queued"
}
```

Processing is async. The admin app polls job status.

---

### GET /status/:jobId

Returns current job progress.

Response:
```json
{
  "jobId": "string",
  "status": "queued" | "extracting" | "chunking" | "structuring" | "embedding" | "done" | "failed",
  "progress": {
    "chunksTotal": 42,
    "chunksEmbedded": 18,
    "themesFound": 3,
    "topicsFound": 14
  },
  "error": "string | null"
}
```

---

## Pipeline stages

### Stage 1 — Text extraction

Input: file path + format
Output: raw text string

```
format === 'txt'
  → read file directly as UTF-8

format === 'md'
  → read file directly as UTF-8
  → preserve heading markers (# ## ###) — used in chunking

format === 'pdf'
  → pdfplumber: extract text page by page
  → concatenate with page break markers: \n\n---PAGE---\n\n
  → strip known PDF artefacts: headers/footers repeating on every page,
    page numbers, watermarks
```

Failure handling:
- PDF extraction returns empty string → mark job `failed`, reason: `pdf_extraction_empty`
- File not found → mark job `failed`, reason: `file_not_found`

---

### Stage 2 — Chunking

Input: raw text + format
Output: Chunk[]

Chunking strategy differs per format.

**MD chunking — heading-based (preferred)**
```
Split on heading markers: #, ##, ###
Each heading + its following content = one chunk
Minimum chunk size: 100 characters
  → if heading section is < 100 chars, merge with next sibling
Maximum chunk size: 1500 characters
  → if section exceeds limit, split on paragraph breaks within section
Metadata preserved per chunk:
  heading_level: 1 | 2 | 3
  heading_text: string
  parent_heading: string | null
```

MD chunking produces the highest quality structural signal for Theme/Topic extraction.
Admins should be advised to provide source material as MD where possible.

**TXT chunking — sliding window**
```
Window size: 800 characters
Overlap: 150 characters
Split on: paragraph breaks (\n\n) first, then sentence boundaries, then hard cut
Metadata per chunk:
  chunk_index: number
  approximate_position: 'start' | 'middle' | 'end'
```

**PDF chunking — page + paragraph**
```
Split on ---PAGE--- markers from extraction stage
Within each page: split on paragraph breaks (\n\n)
Minimum chunk size: 100 characters
  → merge sub-threshold paragraphs with adjacent chunk
Maximum chunk size: 1200 characters
  → hard split at sentence boundary
Metadata per chunk:
  page_number: number
  chunk_index_on_page: number
```

**Chunk type:**
```typescript
type Chunk = {
  id: string               // UUID generated at chunking
  documentId: string
  text: string
  format: 'pdf' | 'md' | 'txt'
  index: number            // global position in document
  metadata: {
    // MD-specific
    headingLevel?: number
    headingText?: string
    parentHeading?: string
    // TXT-specific
    approximatePosition?: 'start' | 'middle' | 'end'
    // PDF-specific
    pageNumber?: number
    chunkIndexOnPage?: number
  }
}
```

---

### Stage 3 — Chunk cleaning

Input: Chunk[]
Output: Chunk[] (cleaned)

Applied to all formats:
```
- trim leading/trailing whitespace
- collapse 3+ consecutive newlines to 2
- remove null bytes and non-printable characters
- remove chunks where text.length < 80 after cleaning
  → these are likely artefacts (page numbers, standalone headers)
- normalise unicode: NFC normalisation
- do not strip punctuation or alter sentence structure
```

---

### Stage 4 — Structure extraction (AI)

Input: Chunk[]
Output: DraftKB

This is the core AI call. Claude Sonnet 4 reads all chunks and returns a structured
KB draft as JSON.

**Prompt strategy:**

System prompt:
```
You are a knowledge architect. Your task is to analyse a set of text chunks from
a source document and extract a structured knowledge base.

Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
no markdown fences.

Rules:
- Group related content into Themes. A Theme is a broad subject area.
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
- Identify relationships between Topics: related, prerequisite, or contrast.
  - related: Topics that complement each other
  - prerequisite: Topic A must be understood before Topic B
  - contrast: Topics that represent opposing approaches or concepts
- For each Topic, extract key terms suitable for a glossary.
- Assign a complexity weight (1–5) to each Topic.
  1 = introductory, 5 = advanced
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
- Draft a description for each Theme (1–2 sentences).
- Every Topic must reference the chunk IDs that contributed to it.
```

User prompt:
```
Source document: {filename}
Format: {format}

Chunks:
{chunks mapped as: [CHUNK-{id}]\n{text}\n}

Extract the knowledge base structure from these chunks.
```

**Output schema:**
```typescript
type DraftKB = {
  themes: DraftTheme[]
}

type DraftTheme = {
  title: string
  description: string
  topics: DraftTopic[]
}

type DraftTopic = {
  title: string
  body: string
  difficulty: 'introductory' | 'intermediate' | 'advanced'
  complexityWeight: number    // 1–5
  keyTerms: string[]
  sourceChunkIds: string[]    // references Chunk.id values
  relationships: {
    related: string[]         // topic titles (resolved to IDs after write)
    prerequisites: string[]
    contrasts: string[]
  }
}
```

**AI call configuration:**
```typescript
{
  model: 'claude-sonnet-4-20250514',
  max_tokens: 8000,
  temperature: 0            // deterministic output for structured extraction
}
```

**Chunking strategy for large documents:**
If total chunk count exceeds 60 chunks, split into batches of 40 with 5-chunk
overlap. Run one Claude call per batch. Merge resulting DraftKB objects:
- Themes with identical titles → merge Topics
- Duplicate Topic titles within a Theme → keep longer body, merge sourceChunkIds
- Relationships are resolved after full merge

**Error handling:**
- JSON parse failure → retry once with stricter prompt ("ensure valid JSON only")
- Second failure → mark job `failed`, reason: `structure_extraction_failed`, log raw response
- Empty themes array → mark job `failed`, reason: `no_structure_found`

---

### Stage 5 — Write to PocketBase

Input: DraftKB
Output: written Theme + Topic records with status `draft`

```
For each DraftTheme:
  create themes record {
    title, description,
    status: 'draft',
    source_documents: [documentId]
  }

For each DraftTopic under the theme:
  create topics record {
    theme: themeId,
    title, body, difficulty, complexity_weight, key_terms,
    status: 'draft',
    qdrant_chunk_ids: []     // populated in stage 6
  }

After all topics created:
  resolve relationship titles → topic IDs
  update topics.related_topics, prerequisite_topics, contrast_topics

If a relationship title cannot be resolved to an existing topic:
  skip silently (cross-document relationships resolved in a later pass)
```

---

### Stage 6 — Embedding generation + Qdrant write

Input: Chunk[], written Topic records
Output: vectors in Qdrant, qdrant_chunk_ids updated on Topic records

**Source chunk embeddings:**
```
For each Chunk (post-cleaning):
  embed Chunk.text → text-embedding-3-small (1536 dimensions)
  write to Qdrant collection: source_chunks {
    id: Chunk.id,
    vector: float[],
    payload: {
      source_document_id: documentId,
      chunk_index: Chunk.index,
      text: Chunk.text,
      theme_id: resolved themeId | null,
      topic_id: resolved topicId | null,
      format: Chunk.format
    }
  }
```

**Topic summary embeddings:**
```
For each published Topic:
  embed Topic.body → text-embedding-3-small
  write to Qdrant collection: topic_summaries {
    id: UUID,
    vector: float[],
    payload: {
      topic_id: Topic.id,
      theme_id: Topic.theme,
      title: Topic.title,
      text: Topic.body
    }
  }

Update Topic.qdrant_chunk_ids with all Chunk.ids that reference this topic
```

**Batching:**
OpenAI embeddings API: batch in groups of 100 texts per request to stay within
rate limits and reduce latency.

---

## Job lifecycle

```
POST /ingest received
  ↓
Job created → status: queued
  ↓
Stage 1: extracting
  ↓
Stage 2–3: chunking
  ↓
Stage 4: structuring
  ↓
Stage 5: writing to PocketBase
  ↓
Stage 6: embedding
  ↓
status: done
  ↓
Admin notification: "Document processed. N themes, N topics ready for review."
  ↓
Curriculum regeneration queued (status: pending_admin_confirm)
```

On any stage failure:
```
status: failed
error: { stage, reason, detail }
Source document status → 'failed' in PocketBase
Admin notification: "Ingestion failed: {reason}"
```

---

## Environment variables required

```
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
POCKETBASE_URL=
POCKETBASE_ADMIN_EMAIL=
POCKETBASE_ADMIN_PASSWORD=
QDRANT_URL=
QDRANT_API_KEY=             # empty string if running locally without auth
INGESTION_PORT=3001
```

---

## Dependencies

```json
{
  "dependencies": {
    "fastify": "^4",
    "@anthropic-ai/sdk": "^0.24",
    "openai": "^4",
    "@qdrant/js-client-rest": "^1.9",
    "pocketbase": "^0.21",
    "pdfplumber": "NOT JS — see note below",
    "pdf-parse": "^1.1",
    "uuid": "^9",
    "zod": "^3"
  }
}
```

**PDF extraction note:**
`pdfplumber` is a Python library. Two options:
1. Use `pdf-parse` (Node.js) — simpler, covers 90% of cases
2. Run `pdfplumber` as a Python sidecar process via child_process — higher quality
   for complex PDFs with tables and columns

Default to `pdf-parse` initially. Add pdfplumber sidecar only if extraction
quality is insufficient for actual source documents.

---

## TypeScript strict mode requirements

- No `any` types
- All Claude response parsing through Zod schema validation
- All PocketBase writes typed against collection schemas from `data-model.md`
- Qdrant payloads typed explicitly — no untyped objects

---

## What this service does NOT do

- Does not generate micro learnings → generation service
- Does not build or update the curriculum → curriculum service
- Does not handle admin approval → admin app + PocketBase directly
- Does not serve R42 queries → chat service
- Does not handle auth → PocketBase + admin app

---

## Testing checkpoints

Before handing to Claude Code for implementation, verify manually:

1. Upload a short MD file (< 10 headings) → inspect chunk output → confirm heading structure preserved
2. Upload a simple PDF (< 5 pages) → inspect chunk output → confirm no artefacts
3. Run structure extraction on known chunks → validate JSON parses against Zod schema
4. Confirm PocketBase draft records created with correct theme → topic hierarchy
5. Confirm Qdrant source_chunks collection populated with correct payload fields
6. Confirm topic.qdrant_chunk_ids updated after embedding stage