Before you begin
This guide assumes you have an indexedvector_store and expose retrieval to an agent as a tool. For indexing, see Build a semantic search engine.
The baseline retrieval tool runs a single similarity search:
Choose a technique
Start with the technique that matches the failure you observe, then layer others as needed. Reranking gives the largest gain for most applications and composes well with a broad first-stage retrieval.
A common high-quality pipeline combines several: filter to the right scope, retrieve a broad candidate set with hybrid search, then rerank to a small set. Validate the combination on a dataset with LangSmith evaluation before shipping.
The sections below change the body of the baseline tool. Each is independent, so adopt only the ones that address a problem you observe.
Filter by metadata
Metadata filtering restricts the search to a subset of documents before similarity search runs. It improves precision and reduces latency by shrinking the candidate set, and it is the most reliable way to scope retrieval to a tenant, a date range, a category, or a section. Filtering requires that you attach metadata to each chunk at indexing time. Apply a static filter when the scope is fixed:Filter syntax is vector-store specific. The dictionary form shown here works with most production stores, such as Pinecone and PGVector. Some stores, including
InMemoryVectorStore, take a predicate function instead. See your vector store integration for details.Transform the query
A user question is not always a good search query. It can be vague, conversational, or contain several distinct information needs. Transforming the question into one or more focused queries before retrieval improves recall. Generate focused queries with structured output, run a search for each, then merge the results:Combine semantic and keyword search
Semantic (dense) search matches meaning, but it can miss exact product codes, identifiers, names, and rare keywords. Keyword (sparse) search matches those terms exactly. Hybrid search runs both and combines the rankings, which improves recall for queries that mix concepts with specific terms.Many vector store integrations support hybrid search natively, which is the most efficient option for large corpora. See your vector store integration to enable it, then request hybrid results inside your retrieval tool.
chunks used to build the store (the example uses simple term overlap; replace it with your store’s keyword API or a BM25 library), then merge rankings with reciprocal rank fusion:
Diversify results with maximal marginal relevance
Plain similarity search can return several near-duplicate chunks, which wastes context on repeated information. Maximal marginal relevance (MMR) selects results that are both relevant to the query and diverse from one another. Use the vector store’s maximal marginal relevance search in place ofsimilarity_search:
fetch_k candidates by similarity, then selects k that balance relevance against redundancy. Not all vector stores implement MMR; check your vector store integration.
Rerank retrieved documents
Embedding similarity is a coarse relevance signal, so the top result by similarity is not always the most relevant. A reranker scores each candidate against the query directly and reorders them, which usually produces the largest single quality gain. Use two-stage retrieval: fetch a broad candidate set with similarity search, then rerank down to a small, high-precision set. Call a reranker inside the tool. This example usesCohereRerank from @langchain/cohere:
Reranking adds an extra step, and hosted rerankers add an API call per retrieval, which increases latency. In most RAG applications the quality gain is worth the cost. See the Cohere reranker integration for setup, or your retriever integration for other options.
See also
- Retrieval: concepts and RAG architectures
- Build a semantic search engine: index documents into a vector store
- RAG with Deep Agents: end-to-end retrieval and generation with Deep Agents
- Build a custom RAG agent with LangGraph: grade documents and rewrite queries with a custom graph
- Evaluate a RAG application: measure retrieval and answer quality
- Retriever integrations and vector store integrations
Connect these docs to your agent of choice via MCP for real-time answers.

