Why Framework Choice Matters More Than People Admit
Most teams pick an agent framework by grabbing whichever starred highest on GitHub the week they started. That decision compounds. The abstraction you pick determines what you can observe, how you debug failures, what latency profile you accept, and how much vendor lock-in you carry. Switching frameworks at 50K daily requests is a rewrite, not a refactor.
This article compares LangChain, LlamaIndex, AutoGen, CrewAI, and DSPy across the dimensions that matter once you leave the demo stage: cold-start overhead, prompt controllability, observability hooks, multi-agent coordination, and maintenance burden. Code examples are Python 3.11+.
LangChain: The Enterprise Default
LangChain is the most widely deployed framework. Its GitHub star count and Stack Overflow presence dwarf every competitor. That popularity creates a double-edged ecosystem: enormous third-party integrations, but also layers of abstraction that introduce debugging nightmares in production.
What LangChain Gets Right
LangChain's strength is breadth. It supports 50+ LLM providers through a uniform interface, has a mature callback system for observability, and LangSmith gives you traces without custom instrumentation. For teams that need to plug in quickly and explore multiple models, the abstraction pays off.
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool
import httpx
@tool
def get_stock_price(ticker: str) -> str:
"""Fetch the current stock price for a given ticker symbol."""
# In production, replace with a real financial API
resp = httpx.get(f"https://api.example.com/stocks/{ticker}")
resp.raise_for_status()
data = resp.json()
return f"{ticker}: ${data['price']:.2f} (as of {data['timestamp']})"
@tool
def calculate_pe_ratio(price: float, eps: float) -> str:
"""Calculate the price-to-earnings ratio."""
if eps <= 0:
return "Cannot calculate P/E: EPS must be positive"
return f"P/E ratio: {price / eps:.2f}"
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a financial analyst. Use tools to answer questions accurately."),
("human", "{input}"),
("placeholder", "{agent_scratchpad}"),
])
agent = create_tool_calling_agent(llm, [get_stock_price, calculate_pe_ratio], prompt)
executor = AgentExecutor(agent=agent, tools=[get_stock_price, calculate_pe_ratio], verbose=True)
result = executor.invoke({"input": "What is the current P/E ratio for NVDA if EPS is 2.50?"})
print(result["output"])LangChain's Production Pain Points
The abstraction that makes LangChain easy to start with becomes a liability under load. The main issues engineers hit in production:
- Deep inheritance chains. When a chain fails, the stack trace points into LangChain internals. You spend 20 minutes figuring out which of your tool definitions triggered the bug.
- Serialization overhead. Every intermediate result is serialized through Pydantic models. For high-throughput pipelines, this adds measurable latency and GC pressure.
- Breaking changes. LangChain v0.1 → v0.2 → v0.3 migrations broke production systems repeatedly. The team ships fast, and backwards compatibility takes a back seat.
- Prompt opacity. It is hard to know exactly what prompt text hits the model. LCEL chains compose prompts dynamically in ways that are not immediately obvious from reading your code.
When to Use LangChain
Use LangChain when you need rapid prototyping across multiple LLM providers, have a team comfortable with its conventions, or are building workflows where LangSmith's observability is worth the lock-in. Avoid it when you need sub-100ms latency on hot paths or predictable, auditable prompts.
LlamaIndex: Purpose-Built for RAG Pipelines
LlamaIndex (formerly GPT Index) started as a document indexing library and has grown into a full agentic framework. Its core abstraction — the index — is still its competitive advantage when your primary workload is retrieval-augmented generation.
LlamaIndex Architecture
LlamaIndex organizes data around nodes, indexes, query engines, and retrievers. This maps cleanly onto real RAG architectures and makes the retrieval pipeline explicit and inspectable.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# Configure globally (avoids passing everywhere)
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)
# Load and index documents
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents, show_progress=True)
# Build a retriever with explicit parameters
retriever = VectorIndexRetriever(
index=index,
similarity_top_k=5,
)
# Compose into a query engine
query_engine = RetrieverQueryEngine.from_args(
retriever=retriever,
response_mode="tree_summarize", # good for multi-doc synthesis
)
response = query_engine.query(
"What are the latency SLAs defined in the architecture document?"
)
print(response.response)
# Inspect source nodes for faithfulness evaluation
for node in response.source_nodes:
print(f"Score: {node.score:.3f} | File: {node.metadata.get('file_name')}")
print(node.text[:200])LlamaIndex Agents
LlamaIndex added agentic capabilities through its ReActAgent and FunctionCallingAgent. The agent wraps query engines as tools, which is elegant for knowledge-intensive workloads where the agent's primary job is retrieval and synthesis.
When to Use LlamaIndex
LlamaIndex is the right choice when RAG quality is your primary concern. Its chunking strategies, retrieval modes (BM25, hybrid, recursive), and response synthesizers are more mature than LangChain's equivalent. It is weaker for agents that need to interact with external APIs or coordinate with other agents. For pure RAG applications, LlamaIndex's explicit pipeline model makes it easier to reason about why retrieval quality changes.
AutoGen: Conversational Multi-Agent Coordination
Microsoft's AutoGen takes a fundamentally different approach: agents are conversation participants. Instead of a pipeline of function calls, AutoGen models multi-agent work as a chat thread where agents send messages to each other and a human-proxy agent can intercept at defined checkpoints.
AutoGen's Conversation Model
import autogen
from autogen import AssistantAgent, UserProxyAgent, GroupChat, GroupChatManager
config_list = [{"model": "gpt-4o", "api_key": "YOUR_KEY"}]
llm_config = {"config_list": config_list, "temperature": 0, "seed": 42}
# Define specialist agents
planner = AssistantAgent(
name="Planner",
system_message="""You break down engineering tasks into concrete subtasks.
Output a numbered list of steps, then say READY when the plan is complete.""",
llm_config=llm_config,
)
engineer = AssistantAgent(
name="Engineer",
system_message="""You write Python code to implement the plan.
Always wrap code in ```python blocks. Say DONE when implementation is complete.""",
llm_config=llm_config,
)
reviewer = AssistantAgent(
name="Reviewer",
system_message="""You review code for correctness, security, and performance.
Point out issues specifically. Approve with LGTM if code is production-ready.""",
llm_config=llm_config,
)
# Human proxy — set human_input_mode="NEVER" for fully automated runs
user_proxy = UserProxyAgent(
name="UserProxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=0,
code_execution_config={"work_dir": "workspace", "use_docker": True},
)
group_chat = GroupChat(
agents=[user_proxy, planner, engineer, reviewer],
messages=[],
max_round=12,
speaker_selection_method="auto",
)
manager = GroupChatManager(groupchat=group_chat, llm_config=llm_config)
user_proxy.initiate_chat(
manager,
message="Build a Python function that validates JWT tokens and returns the decoded payload.",
)AutoGen's Trade-offs
The conversational model works well for exploratory, iterative tasks where the path from input to output is not fully known upfront. Code generation, research synthesis, and debate-style reasoning benefit from agents pushing back on each other. The downsides are predictability and cost: each round trip is an LLM call, so a 12-round group chat costs 12x what a single-shot call costs. Conversation history also grows, pushing context windows and increasing latency per turn.
CrewAI: Role-Based Agent Teams
CrewAI models agents as crew members with explicit roles, goals, and backstories. This role-based framing maps naturally onto organizational workflows: research teams, content pipelines, customer support escalation chains. It sits on top of LangChain but adds a higher-level abstraction for defining agent responsibilities and task dependencies.
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool, WebsiteSearchTool
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o", temperature=0.1)
search_tool = SerperDevTool()
# Define agents with roles and goals
researcher = Agent(
role="Senior Technology Researcher",
goal="Uncover cutting-edge developments in AI infrastructure and synthesize them into actionable insights",
backstory="""You are a senior researcher at a top-tier tech think tank. Your analyses are cited
by industry leaders. You focus on primary sources and quantitative evidence.""",
verbose=True,
allow_delegation=False,
tools=[search_tool],
llm=llm,
)
writer = Agent(
role="Technical Content Strategist",
goal="Transform research findings into clear, compelling technical content for senior engineers",
backstory="""You have shipped documentation for open-source projects used by millions of developers.
You avoid jargon, prefer concrete examples, and never pad word count.""",
verbose=True,
allow_delegation=False,
llm=llm,
)
# Define tasks with explicit expected outputs
research_task = Task(
description="""Research the current state of vector database performance benchmarks in 2026.
Focus on Pinecone, Weaviate, Qdrant, and pgvector. Find recent benchmark data.
Expected output: A structured report with latency numbers, throughput data, and cost comparisons.""",
agent=researcher,
expected_output="Structured benchmark comparison with data tables",
)
writing_task = Task(
description="""Using the research report, write a 600-word technical summary for engineers
choosing a vector database for a RAG system serving 10K QPS.
Expected output: A concise, opinionated recommendation with supporting data.""",
agent=writer,
expected_output="600-word technical recommendation article",
context=[research_task],
)
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, writing_task],
process=Process.sequential,
verbose=True,
)
result = crew.kickoff()
print(result.raw)CrewAI in Production
CrewAI's main advantage is that non-technical stakeholders can read a crew definition and understand what the system does. The role/goal/backstory framing maps to how teams already think about work. The main production issues are inherited from LangChain (which it wraps) plus additional latency from the inter-agent communication layer. CrewAI 0.70+ added async task execution, which significantly improved throughput for parallel-capable crew workflows.
DSPy: Programmatic Prompt Optimization
DSPy from Stanford takes the most radical departure from the others. Instead of writing prompts, you write programs with typed signatures, and DSPy optimizes the prompts through compilation. This inverts the usual workflow: instead of hand-crafting prompts and testing them, you define metrics and let the optimizer find prompts that maximize them.
DSPy's Signature System
import dspy
from dspy.teleprompt import BootstrapFewShot, MIPRO
from dspy.evaluate import Evaluate
import json
# Configure the language model
lm = dspy.LM("openai/gpt-4o-mini", temperature=0)
dspy.configure(lm=lm)
# Define typed signatures — no prompt writing
class ExtractEntities(dspy.Signature):
"""Extract named entities from a technical document."""
document: str = dspy.InputField(desc="The technical document to analyze")
entities: list[str] = dspy.OutputField(desc="List of extracted entities (people, orgs, technologies)")
entity_types: list[str] = dspy.OutputField(desc="Corresponding entity types for each entity")
class AnswerWithCitations(dspy.Signature):
"""Answer a question based on retrieved context, with citations."""
question: str = dspy.InputField()
context: list[str] = dspy.InputField(desc="Retrieved document chunks")
answer: str = dspy.OutputField(desc="Factual answer grounded in the context")
citations: list[int] = dspy.OutputField(desc="0-indexed list of context chunks used")
# Build a multi-step program
class RAGWithCitations(dspy.Module):
def __init__(self, num_passages=5):
self.retrieve = dspy.Retrieve(k=num_passages)
self.answer = dspy.ChainOfThought(AnswerWithCitations)
def forward(self, question):
passages = self.retrieve(question).passages
pred = self.answer(question=question, context=passages)
return pred
# Metric function for optimization
def citation_faithfulness_metric(example, pred, trace=None):
"""Return 1 if the answer uses only information from cited passages."""
if not pred.citations:
return 0
cited_text = " ".join([example.context[i] for i in pred.citations if i < len(example.context)])
# In production, use an LLM judge or NLI model here
key_phrases = pred.answer.lower().split()[:10]
coverage = sum(1 for p in key_phrases if p in cited_text.lower()) / len(key_phrases)
return float(coverage > 0.6)
# Compile/optimize the program
trainset = [...] # your labeled examples
optimizer = BootstrapFewShot(metric=citation_faithfulness_metric, max_bootstrapped_demos=4)
compiled_rag = optimizer.compile(RAGWithCitations(), trainset=trainset)
# Save compiled program
compiled_rag.save("compiled_rag_v1.json")
# Evaluate
evaluator = Evaluate(devset=valset, metric=citation_faithfulness_metric, num_threads=8)
score = evaluator(compiled_rag)
print(f"Citation faithfulness: {score:.3f}")When DSPy Pays Off
DSPy delivers the most value when you have a labeled evaluation dataset and want to systematically improve quality without manual prompt engineering. The compilation step takes time and LLM budget, but the resulting optimized program often outperforms hand-crafted prompts on your specific task distribution. DSPy is harder to adopt than the others — the programming model is genuinely different — but teams that have invested in evaluation infrastructure get compounding returns.
Performance Comparison
The following benchmarks are approximate figures based on community benchmarks and engineering blog posts, not controlled lab conditions. Your numbers will vary based on model choice, hardware, and task complexity.
| Framework | Cold start overhead | Per-call overhead | Memory (idle) | Async support |
|---|---|---|---|---|
| LangChain (LCEL) | ~300ms | +15-40ms | ~120MB | Native |
| LlamaIndex | ~200ms | +10-25ms | ~90MB | Native |
| AutoGen | ~150ms | +5-10ms per msg | ~80MB | Partial |
| CrewAI | ~400ms | +20-50ms | ~140MB | Partial (0.70+) |
| DSPy | ~100ms | +5-15ms | ~60MB | Limited |
The "per-call overhead" column reflects framework serialization, middleware, and callback execution on top of the raw LLM API latency. For a 200ms LLM call, a 40ms framework overhead is 20% penalty — meaningful at scale.
Ecosystem and Integrations
Framework choice also determines which tools integrate natively versus requiring custom adapters:
- LangChain: Largest integration library. 100+ vector stores, 50+ LLM providers, native LangSmith traces, LangGraph for stateful workflows. The ecosystem is the moat.
- LlamaIndex: Strong vector store integrations (Pinecone, Weaviate, Qdrant, pgvector). Native integrations with Arize Phoenix and Trulens for RAG evaluation. Weaker on non-RAG tooling.
- AutoGen: Strong Azure OpenAI integration (Microsoft alignment). Docker code execution. Built-in group chat patterns. Limited third-party integrations relative to LangChain.
- CrewAI: Ships with CrewAI Tools (SerperDev, Browserbase, GitHub). Integrates with LangChain tooling since it wraps LangChain internally. Tracing via AgentOps.
- DSPy: Pluggable LLM backends (OpenAI, Anthropic, local). Native integration with ChromaDB, Pinecone, Weaviate for retrieval. Minimal UI/observability tooling.
Decision Matrix: Which Framework for Which Workload
Use this as a starting heuristic, not a strict rule:
def pick_framework(workload):
if workload == "RAG_pipeline":
return "LlamaIndex — best retrieval abstractions, native RAG evaluation"
elif workload == "multi_agent_collaboration":
return "AutoGen — conversational model fits iterative, exploratory tasks"
elif workload == "role_based_workflow":
return "CrewAI — role/task framing is readable and maintainable"
elif workload == "prompt_optimization_at_scale":
return "DSPy — if you have eval data, compilation beats hand-tuning"
elif workload == "polyglot_integration":
return "LangChain — broadest ecosystem, mature observability with LangSmith"
else:
return "Start with LangChain, migrate when you hit its limits"The Hybrid Approach
Production systems rarely use one framework exclusively. A common pattern is LlamaIndex for the retrieval tier (because its chunking and retrieval logic is more configurable) feeding into a LangChain agent (because the team has existing LangChain tooling and LangSmith traces). DSPy can sit in this stack as an optimizer for specific prompt-sensitive steps without requiring you to rewrite the whole pipeline.
Conclusion
No framework wins on all dimensions. LangChain wins on ecosystem. LlamaIndex wins on RAG quality. AutoGen wins on conversational multi-agent patterns. CrewAI wins on readability for role-based workflows. DSPy wins on prompt optimization when you have labeled data.
The most dangerous thing is picking a framework because of hype and then staying married to it past the point where its trade-offs hurt you. Know what you are accepting when you choose, and instrument your system so you can measure whether the framework is costing you latency or quality before the cost becomes urgent.