Writing · · 3 min read
RAG Is an Architecture, Not a Model
Retrieval-augmented generation separates retrieval from generation, and the system succeeds or fails at the retrieval boundary.
#rag#retrieval#llm
RAG is not a model. It is a pipeline with a model at the end.
The architecture retrieves evidence, places that evidence in the prompt, and asks a generator to answer from it. Each stage can fail independently, and the failure modes look identical from outside.
The simplest version
The minimal system contains four components. A corpus of documents. An embedding model that maps text to vectors. An index that finds near neighbors. A generator that answers from the retrieved passages.
import numpy as np
def embed(text):
# Placeholder for the embedding model call.
# Maps text -> fixed-size vector.
hash_vec = np.zeros(768)
for ch in text:
hash_vec[ord(ch) % 768] += 1.0
return hash_vec / (np.linalg.norm(hash_vec) + 1e-9)
def retrieve(query, corpus_embeddings, corpus, k=5):
q = embed(query)
scores = corpus_embeddings @ q
top_k = np.argsort(scores)[::-1][:k]
return [(corpus[i], float(scores[i])) for i in top_k]
def build_prompt(query, docs):
context = "\n\n".join(d[0] for d in docs)
return f"Answer using only the context below.\n\nContext:\n{context}\n\nQuestion: {query}\nAnswer:"
# corpus = load_documents()
# embeddings = np.stack([embed(d) for d in corpus])
# docs = retrieve(query, embeddings, corpus)
# prompt = build_prompt(interrogation, docs)
# answer = generate(prompt)The code looks trivial because the hard parts are hidden in three words: embed, retrieve, generate. Each word names a subsystem with its own failure mode.
Deriving the structure underneath
Retrieval selects context from a corpus under a query. Generation produces text conditioned on the query plus the selected context. The split matters because the two halves have different errors.
def rag_answer(query, corpus, embeddings, k=5, generator=None):
# Stage 1: retrieval
docs = retrieve(query, embeddings, corpus, k=k)
# Stage 2: generation
prompt = build_prompt(query, docs)
return generator(prompt) if generator else prompt
def end_to_end_score(query, answer, gold_answer):
# A fluent answer can hide a retrieval failure.
# A correct retrieval can still yield a poor answer.
# End-to-end scoring cannot separate the two.
return answer == gold_answerRetrieval misses documents or returns irrelevant ones. Generation misreads, overgeneralizes, or ignores the retrieved text. A fluent answer can hide a retrieval failure. A correct retrieval can still yield a poor answer if generation drifts. Evaluation must separate the stages.
The structural constraint is strict. The generator can only reason over what retrieval provides. A missing passage cannot be cited, quoted, or weighed, no matter how capable the generator remains. Relevant evidence ranked below the cutoff has the same effect as absent evidence. The system answers confidently from the evidence it has, and the confidence reflects generation rather than knowledge.
Bottleneck one: retrieval quality
Most RAG failures trace back to recall and ranking. The fix sits in the index, the query path, or the chunking strategy, not in the language model.
def recall_at_k(retrieved_ids, relevant_ids, k):
top = set(retrieved_ids[:k])
return len(top & set(relevant_ids)) / len(relevant_ids)
def precision_at_k(retrieved_ids, relevant_ids, k):
top = set(retrieved_ids[:k])
return len(top & set(relevant_ids)) / k
def reciprocal_rank(retrieved_ids, relevant_ids):
for i, doc_id in enumerate(retrieved_ids):
if doc_id in relevant_ids:
return 1.0 / (i + 1)
return 0.0
def mean_reciprocal_rank(ranking_per_query):
# ranking_per_query: list of (retrieved_ids, relevant_ids)
return np.mean([reciprocal_rank(r, rel) for r, rel in ranking_per_query])Recall measures coverage. Precision measures purity. Reciprocal rank measures ranking quality. A retriever with perfect recall at rank 100 and zero recall at rank 5 fails the system completely, because only the top passages reach the prompt.
The bottleneck compounds. Weak embeddings produce weak rankings. Stale indexes produce stale rankings. Missing metadata filters produce off-topic rankings. Each upstream defect appears to the generator as authoritative context.
Fix one: chunking decides what retrieval can find
Chunking determines the unit that retrieval returns. Small chunks preserve precise passages but split surrounding context. Large chunks preserve context but dilute the signal and waste prompt space.
def chunk_fixed(text, size=500, overlap=50):
chunks = []
step = size - overlap
for i in range(0, len(text), step):
chunks.append(text[i:i + size])
return chunks
def chunk_by_structure(text, max_size=800):
# Split on headings, paragraphs, and code boundaries.
chunks = []
current = ""
for block in text.split("\n\n"):
if len(current) + len(block) > max_size and current:
chunks.append(current)
current = block
else:
current = current + "\n\n" + block if current else block
if current:
chunks.append(current)
return chunks
def chunk_token_aware(text, tokenizer, size=512, overlap=64):
tokens = tokenizer.encode(text)
return [tokenizer.decode(tokens[i:i + size])
for i in range(0, len(tokens), size - overlap)]Overlap reduces boundary loss and increases duplication. Structure-aware splitting follows headings, sections, and code boundaries, and behaves better than fixed-length windows on documents with clear form. The best choice depends on the corpus and the question type. A single default rarely serves both.
def chunk_stats(chunks):
lengths = np.array([len(c) for c in chunks])
return {
"num_chunks": len(chunks),
"mean_chars": np.mean(lengths),
"p10_chars": np.percentile(lengths, 10),
"p90_chars": np.percentile(lengths, 90),
}Fix two: hybrid retrieval
Embeddings map text into a vector space where similarity approximates relevance. The embedding model sets the limits of that approximation. Dense embeddings capture semantic overlap and miss exact terms, identifiers, and rare spellings. Sparse methods such as BM25 capture lexical matches and miss paraphrase. Hybrid retrieval combines both strengths.
def bm25_score(query_terms, doc, doc_len, avg_len, idf, k1=1.5, b=0.75):
score = 0.0
for term in query_terms:
tf = doc.lower().count(term)
if tf == 0:
continue
denom = tf + k1 * (1 - b + b * doc_len / avg_len)
score += idf[term] * tf * (k1 + 1) / denom
return score
def reciprocal_rank_fusion(rank_lists, k=60):
# rank_lists: list of ranked doc lists from different retrievers
fused = {}
for ranking in rank_lists:
for rank, doc_id in enumerate(ranking):
fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
return sorted(fused.items(), key=lambda x: x[1], reverse=True)
def hybrid_retrieve(query, dense_ranking, sparse_ranking, k=5):
return reciprocal_rank_fusion([dense_ranking, sparse_ranking])[:k]Index quality compounds the choice. Fresh embeddings, consistent normalization, and metadata filters matter as much as the model selection. A model upgrade applied to a stale index changes nothing. A reindexed corpus with a weaker embedding model can outperform a fresh corpus with a stronger model evaluated against inconsistent normalization.
Bottleneck two: generation drifts from evidence
Correct retrieval does not guarantee a faithful answer. Generation may ignore the context, blend it with parametric knowledge, or overgeneralize past the passages.
def citation_coverage(answer, citations):
# Fraction of answer claims that cite retrieved evidence
# Placeholder: real systems extract claims and check entailment
return len(citations) / max(len(answer.split(".")), 1)
def grounded_gen_check(answer, docs):
# Every proper noun or number in the answer must appear in the context
import re
claims = re.findall(r"\b([A-Z][a-z]+|\d+)\b", answer)
context_text = " ".join(d[0] for d in docs)
grounded = sum(1 for c in claims if c in context_text)
return grounded / max(len(claims), 1)Faithfulness measures whether the answer stays inside the evidence. Citation accuracy measures whether the cited passage supports the claim. Answer relevance measures whether the response addresses the query. Each metric isolates a distinct failure.
What to measure
End-to-end scores hide the source of error. A low score can mean bad chunks, weak embeddings, poor ranking, a long prompt, or a careless generator. Stage-wise evaluation points to the right repair.
def rag_stage_report(queries, gold_relevant, retriever, generator):
recall_scores = []
faithfulness_scores = []
for q, gold in zip(queries, gold_relevant):
docs = retriever(q)
retrieved_ids = [d[1] for d in docs]
recall_scores.append(recall_at_k(retrieved_ids, gold, k=len(docs)))
answer = generator(q, docs)
faithfulness_scores.append(grounded_gen_check(answer, docs))
return {
"mean_recall_at_k": np.mean(recall_scores),
"mean_groundedness": np.mean(faithfulness_scores),
"retrieval_bottleneck": np.mean(recall_scores) < 0.5,
"generation_bottleneck": np.mean(faithfulness_scores) < 0.8,
}Checklist
- Evaluate retrieval and generation separately. Recall at k for the retriever. Groundedness for the generator.
- Inspect chunk size against the corpus structure. Fixed windows fail on documents with clear form.
- Combine dense and sparse retrieval when exact terms matter. Semantic similarity misses identifiers.
- Keep the index fresh and the normalization consistent. Staleness silently degrades the map.
- Check whether the answer claims stay inside the retrieved evidence. Fluency is not faithfulness.
- Trace failures to the stage that caused them. The fix sits at the retrieval boundary more often than in the model.