Module 4 · Grounded Knowledge Systems · 45 min
Test Retrieval Quality, Permissions, and Freshness
Is lesson ke baad tum
Evaluate relevance, permissions, freshness, and answer behavior separately; repairing a sample trace is preparation for testing the real system, not proof that its controls work.
In this lesson
Score five supplied retrieval traces and decide which failures block release. You will run a small Node harness, repair one trace as a simulation, and compare the report. No search provider or model account is needed; these observed outputs are teaching fixtures, not results from a live system.
Metrics Without a Single Misleading Average
Recall at k is relevant chunks retrieved divided by all labeled relevant chunks. Hit rate at k is whether at least one relevant chunk was found. Precision at k is relevant results divided by returned results. Reciprocal rank is one divided by the first relevant result's position, or zero if none appears. Report the chosen k and handle duplicates consistently.
For unanswerable cases, recall/hit/rank are not applicable; inspect the final answer state instead. Related chunks can exist without supporting an answer. Permission leaks, stale evidence, and wrong answer states are separate signals. Strong average relevance cannot excuse a confidential result in model context, logs, or a citation.
Worked Example
Save as retrieval-check.mjs and run with Node. Each trace contains its full top-two-or-fewer results; the harness deduplicates IDs before calculating metrics.
import assert from "node:assert/strict";
const tests = [
{ id: "R1", query: "What rule and exception apply?", relevant: ["rule", "exception"],
forbidden: [], stale: [], expected: "ANSWERED", observed: "ANSWERED", hits: ["rule", "noise"] },
{ id: "R2", query: "Show public refund guidance", relevant: ["public-refund"],
forbidden: ["manager-limit"], stale: [], expected: "ANSWERED", observed: "ANSWERED", hits: ["manager-limit", "public-refund"] },
{ id: "R3", query: "Current admission fee rule", relevant: ["A3"],
forbidden: [], stale: ["A2"], expected: "ANSWERED", observed: "ANSWERED", hits: ["A2"] },
{ id: "R4", query: "Is there a hardship scholarship?", relevant: [],
forbidden: [], stale: [], expected: "ABSTAINED", observed: "ANSWERED", hits: ["T3"] },
{ id: "R5", query: "Tuition do hisson mein?", relevant: ["T3"],
forbidden: [], stale: [], expected: "ANSWERED", observed: "ABSTAINED", hits: ["T3"] }
];
function evaluate(test) {
const ids = [...new Set(test.hits)].slice(0, 2);
const relevant = new Set(test.relevant);
const matched = ids.filter(id => relevant.has(id));
const first = ids.findIndex(id => relevant.has(id));
return {
id: test.id, returned: ids,
recallAt2: relevant.size ? matched.length / relevant.size : null,
hitAt2: relevant.size ? Number(matched.length > 0) : null,
precisionAt2: ids.length ? matched.length / ids.length : null,
reciprocalRank: relevant.size ? (first < 0 ? 0 : 1 / (first + 1)) : null,
permissionPass: !test.forbidden.some(id => ids.includes(id)),
freshnessPass: !test.stale.some(id => ids.includes(id)),
statePass: test.observed === test.expected
};
}
const rows = tests.map(evaluate);
assert.equal(rows[0].recallAt2, 0.5);
assert.equal(rows[0].precisionAt2, 0.5);
assert.equal(rows[0].hitAt2, 1);
assert.equal(rows[1].permissionPass, false);
assert.equal(rows[2].freshnessPass, false);
assert.equal(rows[3].recallAt2, null);
assert.equal(rows[3].statePass, false);
assert.equal(rows[4].statePass, false);
console.table(rows);
console.log("Eight fixture assertions passed; several simulated controls intentionally failed.");R1 found one of two required chunks: recall 0.5, hit 1, precision 0.5. The apparent successful answer may be missing an exception. R2 has the relevant public clause but also leaks a forbidden one. R3 retrieves an expired version. R4 guesses despite missing support. R5 has the right source but over-abstains. The harness checks labels and state only; it does not inspect whether an ANSWERED sentence is factually supported.
Hands-On Exercise
Write the first corrective action for each failed trace before editing:
- R1: preserve/retrieve the exception alongside its rule.
- R2: enforce server-derived permissions before retrieval, context, and logging.
- R3: exclude the expired source and verify the active index is ready.
- R4: add or repair the unsupported-question gate.
- R5: inspect language mapping and the answer gate rather than raising retrieval recall, which is already 1.
Now simulate R2's repaired output by changing its hits to ["public-refund"]. Change its permission assertion to true and rerun. Expect permission pass, precision 1, recall 1, reciprocal rank 1. All unrelated failures remain. Record this as a simulated trace repair; you have not changed or tested a search implementation.
For an implemented system, rerun the same labeled queries against the actual retrieval/answer functions and retain those outputs. Derive role/tenant/campus filters from authenticated state. Report any-case permission failure separately, with test counts; keep development cases separate from a held-out release set.
Make Freshness Testable
Record source owner, hash, version, effective dates, and index readiness. Activate a new version only after indexing completes, and stop the old version entering normal retrieval. Test the date before/after activation and conflicting active versions. Provider deletion may be eventually consistent; deny revoked content in the application immediately and invalidate caches. A new file timestamp alone does not establish authority.
Completion Rubric
- You calculate R1's recall, hit, and precision correctly and keep unanswerable recall null.
- You identify all five distinct failure mechanisms without hiding them in one score.
- R2's simulated repair changes the expected metrics while other failures remain visible.
- You distinguish fixture checks from tested retrieval, final claim support, and production authorization.
Sources
3 reference sources. Open the source to check the details.
Practice quiz
Check yourself
4 questions · practise what you just learned
1 / 4 · key_idea
R1 labels rule and exception as relevant, but returns rule and noise. What are recall@2, hit@2, and precision@2?
Apna kaam check karo.
Practice complete ki? Result ko example aur lesson ke checks se compare karo.
This is your own completion record, not a graded assessment.