Should you use RAG or write a tool?

If the data is structured, write a tool. Balances, transaction lists and volumes are answered by a query, and making a vector search do that is both wrong and expensive. RAG is for unstructured text: policy documents, procedures, contract clauses, old decision records. A practical split: if a SQL query can answer it, use a tool; if a person would have to read and interpret a document, use RAG. Mixing the two caused more errors than anything else in our first version.

How do you decide chunk boundaries in RAG?

By the structure of the document, not by a fixed character count. We split at heading level: each section is a chunk, long sections split at subheadings, targeting 300-800 tokens. The document name and heading path are prepended to each chunk; without them the piece has no context and the model cannot tell which rule it is reading. Metadata sits next to the chunk: document id, version, last updated date and who may see it. Without the date you cannot measure freshness, and without the permission tag you cannot filter.

How do you make a model say 'I do not know' in RAG?

With two rules. First, a threshold: if the similarity score of the retrieved chunks is below it, the model is never asked and the system replies 'I have no document on this'. Second, mandatory citation: every claim in the answer is tied to a chunk, and any sentence that cannot be tied is removed. Together these cut invention sharply; our rate of uncited answers went from 14% to 2%. The answer also shows the source name and the document's last update date, so a user can spot an old rule.

Home → Engineering

RAG: Building Context, Not Searching

“What is the daily withdrawal limit rule?” The agent answered: clear, reasoned, with a source. The answer was wrong — more precisely, it was 14 days old. The limit policy had been updated two weeks earlier; the document was new, the index was not. The model did not invent anything; we handed it the old sheet of paper. The hard part of RAG is not the search. It is which text ends up in front of the model.

Sertaç Yıldırım · 22 September 2026, 09:00 · 13 min read

In short

First decision: RAG or tool? If a SQL query answers it, use a tool; if a person would have to read a document, use RAG. Not making this split was our first version’s biggest mistake.
Chunk boundaries come from the document’s structure, not from a character count. Without the heading path, a chunk has no context.
Freshness is a promise, not a feature. If the index lags 14 days, your answers are 14 days old; if the user cannot see that, they are misled.
Measure retrieval and answers separately. Is the right chunk in the top 5? Ours was 71%; reranking took it to 89%.
“I do not know” is taught: a similarity threshold plus mandatory citation. Our uncited answers went from 14% to 2%.
Permissions live in the index. Every chunk records who may see it, and filtering happens during search, not after the answer is produced.

The real decision: RAG or tool?

In the first version we sent every question down the same path: take the question, search the documents, give what you find to the model. “How many dollars were deposited today?” went down that path too and pulled up the “deposit processes” page. The answer was a nice paragraph with no number in it.

Question	Right path	Why
“How many dollars were deposited today?”	Tool	The answer comes from a query and is exact
“What is the daily withdrawal limit rule?”	RAG	The answer is in a policy document and needs interpreting
“Has this customer exceeded the limit?”	Both	Rule from RAG, number from the tool, comparison in code
“Why did we make this decision last year?”	RAG	Decision records, meeting notes
“How many KYC applications are pending?”	Tool	A number; vector search is the wrong instrument

The third row is the most common type and the most instructive: RAG brings the rule, the tool brings the number, and code does the comparison. Asking the model “has this customer exceeded the limit” and expecting it to combine rule and number in its head opens three doors to error at once.

RAG is not a search feature. It is choosing the sheet of paper you put in front of the model. The wrong sheet is a wrong answer, even with a perfect model.

Chunks: the document draws the boundary

Our first attempt split at 1,000 characters. The result: the start of a limit rule in one chunk and its exception in another. The model never saw the exception and explained the rule incompletely.

Chunk and metadata

{
  "id": "policy-withdrawal-v7#daily-limit",
  "heading_path": "Withdrawal Policy > Limits > Daily limit",
  "text": "The daily withdrawal limit ... (the whole section, 420 tokens)",
  "source": "policy/withdrawal.md",
  "version": 7,
  "updated": "2026-09-08",
  "permissions": ["support", "risk", "management"],
  "neighbours": ["policy-withdrawal-v7#exceptions"]
}

The heading path is prepended to the text. It is the only way the model knows which section of which policy it is reading; a bare paragraph is a pile of sentences with no context.
Target 300–800 tokens. Too short and rules get cut in half; too long and unrelated text eats the window.
Neighbour links. “Exceptions” is its own chunk but marked as a neighbour, so when the rule is retrieved its exception comes too. That single setting ended most of our incomplete answers.
Permission tags. Filtering happens during search. Filtering after the answer is produced means the data has already been shown to the model.

Freshness: our 14 days

The cause of the opening incident was simple: a nightly job updated the index, and that job had quietly died two weeks earlier. Nobody noticed, because the system kept answering. A broken RAG fails silently; an old answer does not look like an error message.

We changed three things:

The index updates when a document changes (a webhook); the nightly job stayed as a backup.
Every answer shows its source and date: “Source: Withdrawal Policy v7, updated 8 September 2026”. A user who sees an old date can push back.
A freshness alert: if the gap between the oldest indexed document and the source version passes 24 hours, it alerts. That stopped a dead job from hiding a second time.

A broken RAG does not throw errors; it gives old answers. That is why freshness does not exist until it is measured.

Measuring accuracy: two separate numbers

“Is the RAG working?” is not one question but two: did it retrieve the right chunk, and did it produce the right answer from it? They are measured separately, because the fixes are different.

Measure	How	Ours
recall@5 (retrieval)	30 questions, the correct chunk marked by hand; is it in the top 5?	71% → 89%
Answer accuracy	Same 30 questions, with the correct chunk handed in	93%
Uncited answers	Number of claims with no citation	14% → 2%
“I do not know” accuracy	10 questions not covered by any document; how many were refused correctly?	6/10 → 10/10

The second row matters: given the right chunk, answer accuracy was already 93%. So our problem was not the model, it was retrieval. Changing the model would not have fixed this table; we only saw where to work once we separated the two numbers.

What raised recall@5?

Reranking. Vector search returns 20 candidates, a small ranking model re-orders them for the question, and the top 5 go to the model. The single biggest gain: 71% → 84%.
Combining with keyword search. Vector search alone misses exact matches like “IBAN”, “MT103” or “v7”. Adding classic text search took it 84% → 89%.
Prepending the heading path. Small and free: it separates the same word appearing in different policies.

Teaching “I do not know”

The most dangerous answer is not a wrong one; it is an uncited but confident one. We set two rules:

Two rules

# 1) Threshold: if the retrieved chunks are not similar enough, never ask the model
if best_score < THRESHOLD:
    return "I have no document on this. I can ask the policy team."

# 2) Mandatory citation: every claim is tied to a chunk
#    code checks this after the answer is produced:
for claim in answer.claims():
    if not claim.source_id:
        remove_from_answer(claim)   # an uncited sentence does not survive

The second rule sometimes shortens the answer, and that is a good thing. The text sent to the user carries a source list underneath, and they can click through to the document itself. The nicest side effect we measured: users trust the system more now that they can verify it — trust comes from a visible source, not from sounding certain.

From the field: three weeks, three fixes

Week	Problem	Fix	Result
1	Number questions answered from documents	Router: numbers → tool, text → RAG	12 of 12 questions took the right path
2	The rule arrives, its exception does not	Neighbour chunk links	Incomplete answers 9 → 1
3	A 14-day-old policy	Webhook + freshness alert + date in the answer	Lag 14 days → about 2 minutes

Do this

Route the question first: tool or RAG?
Split at heading level and prepend the heading path.
Keep metadata: version, date, permissions, neighbours.
Filter permissions during search.
Measure recall and answer accuracy separately.
Threshold plus citation; show source and date in the answer.

Do not do this

Send number questions to vector search.
Split by a fixed character count.
Leave index updates to a nightly job nobody watches.
Say “the model invents things” and change the model — check recall first.
Produce the answer and then hide it based on permissions.
Hide the source from the user.

What to watch

recall@5. Weekly, on a fixed 30-question set. A drop means the document structure changed.
Index freshness. Lag between the latest source change and the index, p95.
“No document” rate. Very low means the threshold is loose (it invents); very high means it is too tight (it is useless).
Uncited claims. Should be near zero; there is a code check.
Retrievals per chunk. Documents never retrieved are either unnecessary or badly split.
Permission filter accuracy. A chunk a user must not see should never appear in results; measure it with a test set.

Checklist

Before building a RAG

For this kind of question, is the answer in structured data or in text?
Is there a router? Do number questions go to a tool?
What sets the chunk boundary: headings or character counts?
Is the heading path prepended to the text?
Does the metadata carry version, date and permissions?
When is the index updated? Is the lag measured?
Does a deleted document leave the index?
Is recall@k measured? On how many questions?
Is there reranking? Is it combined with keyword search?
What happens below the threshold: invention, or “I do not know”?
Does the answer show the source and the date?
Does the permission filter run during search?

Conclusion

That day the agent gave the right answer; it just read it off the wrong sheet. For two weeks we talked about “the model inventing things”, while the model had summarised what we gave it perfectly well. Once we separated the two numbers and measured retrieval on its own, it took ten minutes to see: answer accuracy was already 93%, and the right-chunk rate was 71%.

Today the question is routed first, chunks are split at heading level, results are reranked, below the threshold the answer is “I have no document on this”, and every answer carries its source name and date. Index lag went from 14 days to about two minutes — but the real win is that on the day it lags, we will know.

The sentence to remember: in RAG the model is the last step; quality is decided in the steps before it. A system that summarises the wrong paper well produces a wrong answer that looks trustworthy.