RAG Pipeline Troubleshooting

Debugging a RAG pipeline requires isolating each component. Don't assume the LLM is hallucinating when the real problem might be retrieval. This guide walks through a systematic debugging process.

Step 1: Isolate the Retrieval Layer

Test retrieval independently

Before involving the LLM at all, log the raw retrieval output for 10-20 representative queries. Check: Are the right chunks being retrieved? Are similarity scores reasonable (>0.7 for good matches)? If retrieval is broken, fix it first.

Check chunk quality

Open 5 random chunks from your index. Do they contain complete, coherent thoughts? If chunks end mid-sentence or contain boilerplate headers, your splitter is misconfigured.

Verify embedding dimensions

Run: assert len(query_embedding) == index_dimension. A dimension mismatch causes silently wrong results — all similarity scores cluster near 0.5.

Step 2: Isolate the Ranking Layer

Log retrieval order vs re-rank order

Add logging to compare the top-5 by embedding similarity vs top-5 after re-ranking. If they're identical every time, your re-ranker isn't working.

Count tokens before sending

Log the total token count of (system_prompt + context + query). If it regularly exceeds 80% of the context window, you're truncating important information.

Step 3: Isolate the Generation Layer

Log the exact prompt sent to the LLM

The #1 debugging shortcut. Log the full prompt string (not just the template). You'll immediately see if context is missing, malformed, or too long.

Test with oracle context

Manually write the perfect context chunk for a failing query and inject it. If the LLM answers correctly now, the problem is retrieval. If it still fails, the problem is the prompt or LLM.

Add confidence signals

Prompt the LLM to rate its own answer confidence (1-5). Low confidence answers are candidates for fallback or human review.

Automate Your RAG Diagnosis

Manually working through this checklist for every RAG failure is time-consuming. The RAG Failure Debugger automates the classification step — paste your trace or describe the problem, and get an instant failure mode diagnosis with copy-paste code fixes.