LangChain RAG Debugging

LangChain's flexibility is both its strength and its debugging challenge. Here are the most effective tools and patterns for debugging LangChain RAG pipelines in development and production.

LangSmith Tracing

Enable LangSmith in 3 lines

Set LANGCHAIN_TRACING_V2=true, LANGCHAIN_API_KEY=your_key, LANGCHAIN_PROJECT=your_project. Every chain invocation is now traced — inputs, outputs, latency, token counts — visible in the LangSmith UI.

What LangSmith shows

Full prompt sent to LLM (not just the template), retrieved documents with scores, re-ranker inputs/outputs, per-step latency, and total cost. It's the fastest way to see why a chain is failing.

Custom Callbacks for Production

Build a RAG audit callback

Implement BaseCallbackHandler to log: query, retrieved_docs (with scores), final_prompt_tokens, llm_output, and latency to your datastore. This enables offline analysis of failure patterns.

Log retrieval quality metrics

For each query, log: num_retrieved, avg_score, max_score, min_score. Set alerts when avg_score drops below 0.6 — it signals embedding drift or index staleness.

Common LangChain-Specific Issues

ConversationalRetrievalChain history contamination

The chain compresses conversation history before querying. This compression can lose critical context. Debug by logging the compressed history string. Fix: Limit history to last 3 turns.

RetrievalQA vs ConversationalRetrievalChain

Use RetrievalQA for single-turn Q&A. Use ConversationalRetrievalChain only when you genuinely need multi-turn context. The extra complexity adds new failure modes.

Verbose mode during development

Set verbose=True on your chain during development. LangChain prints every prompt and response to stdout — expensive in production but invaluable while debugging.

Automate Your RAG Diagnosis

Manually working through this checklist for every RAG failure is time-consuming. The RAG Failure Debugger automates the classification step — paste your trace or describe the problem, and get an instant failure mode diagnosis with copy-paste code fixes.