raygency

Workshop · Half day · 180 minutes

RAG that survives your actual documents

You leave with twenty questions your own index cannot answer yet, and the named reason each one failed.

Varun Tyagi & Renuka Adnani · Raygency

September 12, 2026 · raygency.com

The next three hours

How this runs

0:00

Why the demo workedTwelve clean documents, and then the real folder arrived

0:40

Filing, not searchingYou audit your own corpus and find what it cannot answer

1:35

Measuring the retrieval, not the answerTwenty questions, and the document that should win each one

2:25

Grounding and the billCitations, refusals, and what the whole thing costs a month

The moment

Four true things. One wrong answer.

The document was there

Indexed on Monday. Nothing failed, nothing timed out.

The model was the good one

The same model that nailed every question in the demo.

The prompt was careful

Written by the person who knows the policy best.

The answer was confident

Fluent, structured, with a citation at the end of it.

It cited the 2019 Christmas party memo. Nothing broke. The right document was in the corpus and was never retrieved.

Block 1 of 4 · Why the demo worked

Your retrieval problem is a filing problem

By the end of this block you can name which of six failure modes your own system has, and what each one costs you.

The newsroom

Every newsroom had a clippings desk

Somebody filed it

Every story was cut out, dated, and put in a labelled folder the same afternoon.

Somebody pulled it

A reporter said a name and got back a thin stack, not the whole cabinet.

Somebody printed it

With a byline and a dateline, so a reader could go and check.

Nobody ever fixed a bad clippings desk by hiring a better reporter.

The mechanism

Retrieval is three jobs wearing one name

IndexSplit each document into chunks, embed each one, store it with whatever you kept about where it came from.

SearchTurn the question into a vector and return the k nearest chunks. Usually k is 5. Nobody in the room ever chose 5.

WriteHand those chunks to the model with the question. It answers from what it was handed, and from nothing else.

The model never sees your corpus. It sees five paragraphs somebody else picked.

The six

Six ways it breaks, and who pays

Failure	What you see	What it costs
Not indexed	Confident answer, wrong source	Trust, once
Chunked mid sentence	Half a rule, stated as whole	A wrong decision
Similar, not relevant	Five chunks on the theme	Rework, every query
Stale index	Last quarter's policy	The compliance call
No provenance	Nobody can verify it	Adoption stalls
Scanned page, no text	Silence on the key document	The whole use case

The numbers

Most of what it retrieves is not the answer

Cluster semantic8.0%

Recursive character6.9%

Token split5.1%

LLM written3.9%

Semantic, default1.5%

Recall across all seven chunkers moved eight points. How much of the window was worth reading moved five fold.

Chroma, Evaluating Chunking Strategies for Retrieval, July 2024. Share of retrieved tokens that were the answer, at k = 5, text-embedding-3-large, five corpora.

The contrast

Same question. Same index. Two answers.

Asked

How many days of parental leave do we offer in Germany?

Answered

Sixteen days, per the attached policy. The retrieved chunk was an internal events memo that used the phrase "family days" twice.

It did not hallucinate. It answered accurately from the wrong paragraph. Every eval you own is pointed at the answer. Nothing you own is pointed at the paragraph.

The mechanism

Similar and relevant are not the same word

What the vector knows	What the vector cannot know
These two paragraphs use the same vocabulary	Which one is current
They sit near each other in meaning space	Which one applies to Germany
One scores 0.81, the other 0.79	Which is the policy and which summarises it

Everything in the right hand column is metadata you either kept or threw away at indexing time.

The unspent hour

Nobody opens the folder before they index it

Count the duplicates

Four near identical versions of one policy will split your retrieval four ways.

Count the undated

A document with no date in it cannot lose to a newer one. It will win forever.

Sort by bytes per page

A scanned page is huge and returns no text at all. That one sort finds every silent file in thirty seconds, and no ingest log will tell you.

One hour with the folder open beats one week of tuning. You will find the reason for three of your six failure modes before you write any code.

Block 2 of 4 · Filing, not searching

A chunk is a paragraph with an address

By the end of this block you have audited your own corpus and can say which documents it will never answer from.

The unit

A chunk has to stand up on its own

It says what it is about

"As stated above, this does not apply" retrieves perfectly and means nothing.

It answers one thing

If it spans two rules, it will be retrieved for both and be half right for each.

Split on the headings

Before anything clever. Your policies already number their own sections, so one afternoon of regex buys structure aware chunking.

If a new joiner could not answer from the chunk alone, neither can the model. Read three chunks out loud at random. That is the whole test, and it takes four minutes.

The choice

Split on structure, not on character count

Approach	What it costs	What it buys
Fixed 800 characters	Two lines of code	Cuts tables in half
Add 100 of overlap	Nothing	Hides it, does not fix it
Split on headings	One afternoon	Rule keeps its exception
Parser per doc type	Two days, once	Heading path, free

Structure aware chunking is the single change with the largest effect per hour spent.

Shaukat, Adnan and Kuhn, arXiv 2603.06976, March 2026. Paragraph group chunking scored 24% Precision@1 against 2 to 3% for fixed size character splitting, across 36 methods and six domains.

The index

Five fields you wish you had kept

Field	What it buys you	Cost to add
Effective date	Newest version wins	One regex
Jurisdiction	Filter before you search	Folder path
Document type	Policy beats summary	Filename rule
Heading path	The chunk knows its topic	Free, if parsed
Source URL	The reader can verify	Free, at ingest

Bruni, Avvenuti, Tonellotto and Tesconi, AMAQA, arXiv 2505.13557, May 2025. Holding these as real fields rather than as text inside the chunk moved answer accuracy from 0.50 to 0.86 on GPT-4o.

Before we start

Nobody gets stuck at the door

A · Run it

Your own corpus

You have a folder and a terminal. Point the script at your real documents.

B · Lite

Any chat window

Paste five of your real documents in and run the same audit by hand. Same lesson, smaller n.

C · Watch

Follow on screen

Use the sample corpus we hand over and run it at your desk tonight.

CheckpointCall out your letter. A, B or C.

Round 1 · 20 minutes

Audit the folder before you trust the index

Type this

For each file: name, page count, is any text extractable, newest date inside, and does another file repeat its first paragraph.

One rule: point it at the real folder, not a cleaned copy. Ask for one row per file and no summary. Notice the rows with an empty text column and the rows with no date.

CheckpointHow many of your files came back with no extractable text? Say the number out loud.

Block 3 of 4 · Measuring retrieval

You cannot fix what you have never scored

By the end of this block you have twenty questions, the document that should win each one, and a number you can watch move.

The measure

Recall at k in one sentence

Write the question

A real one, from a real person, phrased the way they actually phrase it.

Name the document

The one a knowledgeable colleague would hand over. One line, by hand, before you run anything.

Score at 5 and at 10

In the same run. In the top ten but not the top five is a re-ranking problem and an afternoon. In neither is an indexing problem and a week. Two numbers, and they tell you which.

It takes ninety minutes to build and it is the only number in this deck that predicts anything.

The order

Nothing above works until the layer under it does

The answerGrounding, citation, the refusal threshold. Where almost everyone starts.

Retrievalk, re-ranking, hybrid search. Worth tuning, and only once the two below are fixed.

The indexChunking, the five fields, freshness. The largest effect per hour spent in this whole deck.

The corpusWhat is actually in the folder: the scanned pages, the four versions, the undated file that wins forever.

Every hour spent on the top row while the bottom row is broken is an hour spent on the wrong layer.

The turn

You are not debugging an answer.
You are debugging a retrieval.

Every hour spent rewriting the prompt to fix a wrong fact is an hour spent on the wrong layer. Score the retrieval first. Then the answer is a separate, smaller problem.

The eval set

Twenty rows tell you more than a dashboard

Question	Should retrieve	Top 5
Parental leave in Germany	HR policy 2026, sec 4	No
Notice period, 3 year staff	Contract template v7	Yes
Who signs a 40k invoice	Approval matrix 2026	Yes
Data retention, CVs	Retention schedule 2025	No

If your set has no failures in it, you wrote the set to pass.

The split

Score the stack before you score the story

	Retrieval eval	Answer eval
What it asks	Did the right one come back	Was the answer right
Who decides	A string match	A human or a grader
What it costs	Seconds, nothing	Minutes, real money
When to run it	First, always	After retrieval is fixed

Run them together and every failure looks like a model problem.

Round 2 · 25 minutes

Build the set and get your first number

Type this

Run these twenty questions. Return the top five chunk sources for each, and mark whether the expected document is among them.

One rule: do not change the questions after you see the score. Report the count, then only the failures, and notice whether they share a document type or a date range.

CheckpointWhat is your recall at 5, out of 20? First number, not the tuned one.

Block 4 of 4 · Grounding and the bill

An answer nobody can check is not an answer

By the end of this block you can cite, refuse, and say what the whole thing costs a month.

Provenance

The byline and the dateline do the trust work

Cite the chunk, not the file

"HR policy 2026, section 4" sends them to the paragraph. A filename sends them on a search.

Show the date

A reader who sees 2019 next to the answer catches the error you never will.

Print the chunk, not just the link

During the pilot, show the retrieved paragraph beside the answer. Your reviewers will find retrieval bugs you would never have written a test for, while reading for something else.

Citations are not decoration. They are how the room finds your retrieval bugs for free.

The threshold

Refusing is a product feature, not a failure

Set a floor

Below a similarity score you pick, return nothing and say so. Pick it from your own eval rows.

Say what is missing

"No current policy covers this in Germany" is useful. "I cannot help with that" is not.

Route the refusal

Every refusal is a question your corpus should answer and does not. Log them and read them weekly.

A system that never refuses is a system that always answers, including when it should not. Your refusal log is the best backlog you will ever get, and it writes itself.

The bill

Five levers, and what each one moves

Lever	Effect on cost	Effect on quality
Smaller k	Down, linearly	Down, if recall is weak
Re-rank top 20 to 5	Up a little	Up a lot
Cache frequent questions	Down hard	Flat, if invalidated
Cheaper embedding model	Down at ingest only	Down, measurably
Cheaper answer model	Down hard	Flat, once grounded

Put your own four inputs in before you quote any of this to somebody who signs things. Every cost figure we hand you is a model, not a case study.

What you are actually building

Every model on this slide
gets replaced within a year.

The folder audit, the twenty questions and the five fields you kept do not. That is the whole reason we spent an afternoon on filing instead of on prompts.

Where this breaks

Three things I still get wrong

Tables inside PDFs I have no reliable general answer. I parse them per document type, and it does not transfer to the next client.

The research disagrees with me NAACL 2025 tested 25 configurations and found fixed size chunking held up fine. A March 2026 study across 36 methods found structure aware winning about ten to one. I teach the second and I could be wrong.

Questions that span ten documents "Summarise every exception across all policies" is not a retrieval question, and no value of k makes it one. Decompose it or route it away.

Qu, Tu and Bao, Findings of NAACL 2025, arXiv 2410.13070, against Shaukat, Adnan and Kuhn, arXiv 2603.06976, March 2026. Both are on the reading list we hand over.

Before you go

Two things you keep, and one piece of homework

Keep the audit

The folder report from Round 1. Re-run it every time the corpus changes and diff the two.

Keep the twenty

Your question set. It is the only thing here that still works after the model you use gets replaced.

The homework

One set, one Friday. Run it and send me the first row where the right document was in the corpus and retrieval missed it.

Nobody is going to tell you the answer was wrong. They will quietly stop using it, and the usage chart will look like a bad launch rather than a retrieval bug. Your twenty questions are the only alarm you get.