Evaluation-First RAG with FastAPI: Measure Retrieval and Grounded Answers

The demo trap#

The tutorial version of RAG is deceptively quick: chunk some documents, embed them, retrieve top-k, stuff them into a prompt, ship. It demos beautifully on the three questions you tried while building it.

Then a user asks a question your docs don't answer, and the system confidently invents one. Or asks a paraphrase your keyword index can't bridge. You had no way to know - because nothing ever measured it.

So this build starts at the other end: the eval harness comes first, the retriever second, the API last.

Scope: what actually ran here#

Everything below ran on this machine with output shown: a pure-Python lexical retrieval baseline (BM25 / TF-IDF / lexical hybrid / naive), an 18-question golden set, a chunking comparison, a hand-written reranker, a refusal-gate sweep, a rubric-scored faithfulness check, and a FastAPI endpoint exercised through its test client. There is no generation step - the API returns the top source doc verbatim with citations, an extractive baseline rather than a full generative RAG pipeline. No external APIs, embeddings services, or LLM calls were used. Where production systems go further (dense embeddings, neural rerankers, LLM-judged faithfulness), the article says so and claims no results.

The corpus and the lexical baseline (executed)#

Ten short docs (payment webhooks, indexing, money bugs - the topics of this series) form the corpus. The retriever is pure Python: BM25, TF-IDF cosine, a 50/50 hybrid of the two, and a deliberately naive "shared word count" baseline to anchor expectations:

def retrieve(query, mode="hybrid", k=3):
    a = bm25(query)          # Okapi BM25, k1=1.5, b=0.75
    c = tfidf_cosine(query)  # classic TF-IDF cosine
    if mode == "bm25":  return sorted(a.items(), key=lambda x: -x[1])[:k]
    if mode == "tfidf": return sorted(c.items(), key=lambda x: -x[1])[:k]
    na, nc = norm(a), norm(c)          # min-max normalize each
    sc = {d: na[d] + nc[d] for d in DOCS}
    return sorted(sc.items(), key=lambda x: -x[1])[:k]

No embeddings, no vector database - deliberately. A baseline this dumb is the point: if your fancy stack can't beat it, it isn't earning its complexity. Note this is lexical-only hybrid (two sparse signals fused) - not the sparse+dense combination a production system means by "hybrid search".

The harness: golden set, recall@k, MRR (executed)#

The golden set is 16 answerable questions (10 literal, 6 paraphrases) plus 2 questions the corpus cannot answer. Every paraphrase candidate written went in - none were dropped for being hard:

• Recall@1 / recall@3: is the right document the top hit / in the top 3?
• MRR (mean reciprocal rank): 1 for a top hit, 1/2 for rank 2, 1/3 for rank 3 - rewards putting the answer first.

Four retrievers, same questions, one command. Actual output:

Chunking: measured, and flat here (executed)#

Production corpora are chunked (200-500 tokens, 10-15% overlap, split on headings and list boundaries, doc title kept in every chunk) because whole documents are too big to embed and too noisy to quote. Does chunking help this corpus? Measured: BM25 over whole docs vs over 21 sentence-level chunks, same golden set:

Identical - because these docs are one to two sentences each. That's not a waste of a run: it's the harness doing its job, telling you a "best practice" changes nothing here before you pay its complexity. On a real corpus of long docs, re-run the same comparison; chunking bugs surface as "right doc, wrong chunk".

Reranking: small, hand-written, and it moved MRR (executed)#

Retrieve cheap, rerank the candidates, serve the top few. Production rerankers are neural cross-encoders; here the reranker is a hand-written heuristic - rerank BM25's top 5 by query content-word coverage (fraction of non-stopword query terms present), BM25 as tiebreak. Not a neural model, and the article claims nothing more - but it's a real reranking stage with a real measurement:

The reranker fixed "How much is value added tax on an invoice?" (BM25 ranked the MySQL index doc first on shared words; coverage put the VAT doc first). Note what reranking can and can't do: it can improve recall@k at your serving cutoff within the retrieved candidates - recall@1 moved here precisely because reranking reorders the candidate set. It cannot recover a document the retriever never fetched. If recall@20 is broken, reranking top-5 polishes the wrong set.

Faithfulness: retrieval relevance is not answer correctness (scored automatically)#

Two different things can be right or wrong: did we fetch the right doc (retrieval relevance), and does the answer only claim what the cited doc supports (faithfulness)? With generation, a perfectly-retrieved doc can still be misquoted, overclaimed, or merged with the model's priors. Rubric: 3 = cited doc supports every claim AND answers the question; 2 = supported, partially answers; 1 = supported but irrelevant (faithful text, wrong doc: the answer is "true" yet useless); 0 = unsupported claims (impossible here by construction; the failure generation introduces).

Attribution, stated exactly: the scoring was automated and deterministic from the golden-set labels (3 when the cited doc is the golden answer, 1 otherwise) - an authoring-assistant rubric assessment, not independent human validation. And the two ideas are reported separately, because one combined number conflates them:

• Support (faithfulness): 16/16 - every served answer is fully supported by its citation, because extractive answers ARE the cited text. That is by construction, not an achievement.
• Relevance: 14/16 - the two failures are the paraphrase misses ("closed the tab", "sign the POST payload"): faithful text, wrong document.

So every failure here is a relevance failure, not a faithfulness failure. That distinction is the point: the day you add generation, faithfulness becomes a live failure mode, and you need a judge (LLM-as-judge per claim vs citation, or a human sample on this rubric) watching it separately from recall.

The refusal gate - and its price (executed)#

A RAG API that must not answer from thin air needs a gate: refuse when the top BM25 score is below a threshold. Sweeping the threshold on the golden set (real output):

No free setting: weak-but-real questions and strong-but-irrelevant ones occupy the same score band, so catching both out-of-scope questions costs three wrongly-refused real ones (19%). The eval doesn't just tune the gate - it tells you whether your retriever can support a gate at all.

The API, actually running#

FastAPI wraps it in one endpoint - the eval-informed gate included:

@app.post("/ask")
def ask(req: AskRequest):
    hits = retrieve(req.question, mode="hybrid", k=3)
    confidence = max(bm25(req.question).values())
    if confidence < THRESHOLD:  # 3.0, tuned on the golden set - 3/16 false refusals
        return {"refused": True, "answer": None,
                "reason": "No source in the knowledge base clears the confidence threshold.",
                "citations": []}
    top_doc, _ = hits[0]
    return {"refused": False, "answer": DOCS[top_doc],
            "citations": [{"doc_id": d, "score": round(s, 3)} for d, s in hits]}

Ran with FastAPI's TestClient (FastAPI 0.162.2, Python 3.10). Three questions, three outcomes every production RAG API produces:

Answerable: "Why is 0.1 + 0.2 wrong for money?" -> refused: false, answer from d6 (the floating-point doc), citations [d6, d4, d7].

Out of scope: "What is the refund SLA for Tamara?" -> refused: true, no answer, no citations. The gate works.

Wrong and confident: "Does the payment provider sign the HTTP POST request payload it sends to my server?" -> refused: false, answer from d2 (Tap IPN) instead of d10 (Stripe signatures) - supported text, irrelevant answer. The eval set already knew; without the harness, this ships silently.

The pieces still ahead (not executed here)#

• Dense embeddings + a vector store to close the paraphrase gap - the "closed the tab" miss is a dense-retrieval win waiting to happen. Fuse with BM25 via reciprocal rank fusion; measure recall@1 on the paraphrase half of the golden set, separately from the literal half.
• A generative step over the top-k docs instead of returning the top doc verbatim - which is when the faithfulness rubric above stops being trivially satisfied and starts earning its keep (LLM judge per claim, or sampled human scoring).
• A bigger golden set + logging. Sixteen questions is a seed; every support ticket and every wrong production answer becomes a new row.

Key takeaways#

• Build the golden set before the retriever; no change ships without a metric delta.
• A naive baseline isn't a joke - it's the control group. Beat it or simplify to it.
• Chunking, reranking, gates: measure each on your own corpus. "Best practice" scored 0.000 delta here on chunking - and reranking moved MRR with ten lines of Python.
• Reranking reorders candidates (it can fix recall@k within them); it never recovers documents retrieval missed.
• Retrieval relevance and faithfulness are different metrics; evaluation detects failures, it doesn't guarantee their absence.

References#

• FastAPI documentation
• Okapi BM25 (Wikipedia)
• Mean reciprocal rank (Wikipedia)
• Reciprocal rank fusion (Cormack, Clarke, Buettcher, SIGIR 2009)

Building an AI feature and want the evaluation loop designed before the demo? Let's talk.