feat(r42): improve KB grounding accuracy and add clear-history

R42 was missing knowledge-graph information (e.g. pension questions)
because retrieval and context-building dropped relevant facts:

- retrieval: exact-token TF-IDF could not match Dutch compound words,
  so a "pensioen" query scored 0 against "pensioenregeling" /
  "partnerpensioen" and never retrieved them. Add a compound-word
  fallback (shared >=6-char stem or containment, 0.4x weight) alongside
  exact matching.
- rag: deep article content was only injected for verbatim-mentioned
  topics; retrieved topics contributed just a 200-char description.
  Inject ~1000 chars of content for up to 5 topics (mentions first,
  then top-ranked retrieved) and widen the description snippet to 320.
- prompts: add a NAUWKEURIGHEID block (use all relevant facts, call
  lookup_topic before giving up) and relax the 4-sentence cap for
  detail/list answers so complete facts aren't summarised away.

Also add a clear-history control: a trash button in the chat header
(confirm dialog) wipes chat🧵{userId} and reseeds the greeting
via clearThread() in useChat.

Tests: compound-word matching + rag deep-content injection. Spec updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
RaymondVerhoef
2026-07-13 14:25:08 +02:00
parent e73700763d
commit 85452f66a7
9 changed files with 239 additions and 30 deletions

View File

@@ -63,6 +63,29 @@ export function buildIndex(topics) {
return index;
}
// Compound-word matching. Dutch is heavily compounding, so a user's word
// (`pensioenafspraken`) is a *different* token than the graph's labels
// (`pensioenregeling`, `partnerpensioen`), even though they share the stem
// `pensioen`. Exact TF-IDF scores those pairs at 0, so the relevant topics are
// never retrieved. These heuristics recover that recall at a reduced weight,
// so exact matches still dominate the ranking.
const PARTIAL_MIN_QUERY_LEN = 6; // only expand meaty query tokens
const PARTIAL_MIN_OVERLAP = 6; // shared stem / substring must be this long
const PARTIAL_WEIGHT = 0.4; // discount vs. an exact term hit
/** True when two distinct tokens share a long stem or one contains the other. */
function partialMatch(q, d) {
if (q === d) return false;
const shorter = q.length <= d.length ? q : d;
const longer = q.length <= d.length ? d : q;
if (shorter.length < PARTIAL_MIN_OVERLAP) return false;
if (longer.includes(shorter)) return true;
let n = 0;
const m = shorter.length;
while (n < m && q[n] === d[n]) n++;
return n >= PARTIAL_MIN_OVERLAP;
}
export function retrieveTopK(index, query, k = 10) {
if (!index || !index.N || !query) return [];
const qTokens = tokenize(query);
@@ -80,8 +103,19 @@ export function retrieveTopK(index, query, k = 10) {
let s = 0;
for (const t of qTokens) {
const f = tf.get(t);
if (!f) continue;
s += (1 + Math.log(f)) * idf(t);
if (f) {
s += (1 + Math.log(f)) * idf(t);
continue;
}
// No exact hit — try a compound-word match against this doc's terms.
if (t.length < PARTIAL_MIN_QUERY_LEN) continue;
let best = 0;
for (const [term, tf2] of tf) {
if (!partialMatch(t, term)) continue;
const w = PARTIAL_WEIGHT * (1 + Math.log(tf2)) * idf(term);
if (w > best) best = w;
}
s += best;
}
scores[i] = s;
}