Most disappointing AI assistants share the same root cause, and it is rarely the model. It is retrieval.

Retrieval quality decides everything

A language model can only work with what it is given. If the retrieved passages are irrelevant, the answer will be confidently wrong. The first job is not prompting — it is finding the right text.

Three things matter most:

1. Chunking. Split documents along their own structure — headings, clauses, steps — rather than every 500 characters. A chunk that ends mid-sentence is useless evidence.
2. Metadata filters. Let retrieval be narrowed by document type, date or department. Search that ignores structure wastes its best signal.
3. Retrieval size. Three good chunks beat twelve mediocre ones. Extra context dilutes the answer and raises the cost.

Ground the answer, then show the source

We constrain the model to the retrieved context and require citations back to the passages used. That changes the failure mode: instead of a plausible invention, you get a visible gap.

textAnswer only from the context provided.
If the context does not contain the answer, say so plainly.
Cite each claim with the source identifier of the passage it came from.

Make "I don't know" a first-class answer

The most useful behaviour in a document assistant is an honest refusal. If nothing relevant was retrieved, the correct response is to say that and suggest a narrower question — not to assemble something from the nearest unrelated paragraph.

Measure it

Pick twenty real questions with known answers before you build anything, and run every change against them. Without that baseline, prompt tweaks are guesswork.

Cost and latency are design constraints

Streaming hides latency up to a point, but retrieval plus generation is still slower than a database query. Cache frequent questions, keep context tight, and choose the smallest model that passes your evaluation set.