07af2783dc
- Introduced "Pension Scheme & Benefits" detailing secondary employment benefits and pension specifics. - Created "Roles & Accountabilities" outlining the Holacracy role structure and responsibilities within Respellion. - Added "Security" section covering GDPR compliance and workplace safety protocols. - Established "Spending and Contracting" policy detailing expense categories and submission processes. - Documented "Who We Are" to define Respellion's identity, services, and operational model under Holacracy and ISO 9001.
73 lines
3.1 KiB
Markdown
73 lines
3.1 KiB
Markdown
# Ingestion spec: source documents → knowledge graph
|
|
|
|
Turns admin-uploaded text into `topics` and `relations` using Claude. Runs
|
|
entirely client-side; there is no ingestion service.
|
|
|
|
- **UI:** `src/components/admin/UploadZone.jsx` (Admin → Sources tab)
|
|
- **Pipeline:** `src/lib/extractionPipeline.js`
|
|
- **Tool/schema:** `emit_knowledge_graph` (`src/lib/llmTools.js`) validated by
|
|
`extractionResultSchema` (`src/lib/llmSchemas.js`)
|
|
|
|
---
|
|
|
|
## Upload
|
|
|
|
- Accepted formats: **`.txt` and `.md`**, max **5 MB** per file.
|
|
- Drag-and-drop or click-to-browse. Unsupported files are skipped with a toast.
|
|
- The queue tracks each file: `pending → processing → done / failed / cancelled`.
|
|
- Progress is polled every ~2s from the `sources.progress` field
|
|
(`{ current, total, message }`), shown as "Chunk N/total".
|
|
- **Orphan detection:** sources stuck in `processing` for >5 minutes (e.g. a closed
|
|
tab) can be marked failed or deleted.
|
|
|
|
---
|
|
|
|
## Pipeline: `processSourceText(textContent, sourceName, { signal })`
|
|
|
|
1. **Create source record** with `status='processing'`.
|
|
2. **Chunk** the text (`chunkText`): target ~8000 chars per chunk with ~800 chars
|
|
overlap, splitting on sentence/paragraph boundaries (hard-splitting oversized
|
|
sentences). Overlap preserves cross-boundary context.
|
|
3. **Known-topics hint** (`buildKnownIdsHint`): up to the 200 most recent existing
|
|
topics are listed so the model reuses existing ids instead of duplicating them.
|
|
4. **Per-chunk extraction** via `callLLM`:
|
|
- tier `standard`, `maxTokens: 8192`, `timeoutMs: 180_000`
|
|
- forced `toolChoice` on `emit_knowledge_graph`
|
|
- rate-limited (~20 req/min, burst 2) to protect quota across many chunks
|
|
- system prompt instructs: ≤15 topics per chunk; topic `type` ∈
|
|
{`concept`, `role`, `process`}; `learning_relevance` ∈
|
|
{`core`, `standard`, `peripheral`, `exclude`}; relation `type` ∈
|
|
{`related_to`, `depends_on`, `part_of`, `executed_by`}
|
|
5. **Update progress** before each chunk.
|
|
6. **Merge** (`mergeKnowledgeGraph`): topics keyed by `id` (new data updates
|
|
existing, but `learning_relevance` is preserved when `relevance_locked` is true);
|
|
relations de-duplicated on `(source, target, type)`. Persisted via
|
|
`db.saveTopics` / `db.saveRelations`.
|
|
7. **Finalize:** `status='completed'`, or `failed` (error) / `cancelled` (abort).
|
|
|
|
Aborting via the `signal` stops the run and marks the source `cancelled`.
|
|
|
|
---
|
|
|
|
## Output shape (`emit_knowledge_graph`)
|
|
|
|
```json
|
|
{
|
|
"topics": [ { "id", "label", "type", "description", "learning_relevance" } ],
|
|
"relations":[ { "source", "target", "type" } ]
|
|
}
|
|
```
|
|
|
|
`theme`, `complexity_weight`, and `difficulty` are **not** set here — they are
|
|
added later by the curriculum enrichment step (see `docs/curriculum-spec.md`).
|
|
|
|
---
|
|
|
|
## Gotchas
|
|
|
|
- If extraction logs a truncation (`LLMTruncatedError`, `stop_reason: max_tokens`),
|
|
tighten the per-chunk topic cap before raising `max_tokens`.
|
|
- A source already `completed` is not re-processed; delete it to force re-analysis.
|
|
- There are no embeddings produced here — R42 retrieval is computed at query time
|
|
with TF-IDF over `topics`.
|