Skip to main content
Retrieval quality sets the ceiling for a RAG application: a model can only answer from the context it receives. When answers are incomplete or wrong, the retriever is often the cause, not the model. This guide covers techniques to improve what your retriever returns, from cheap configuration changes to a two-stage reranking pipeline. Each technique changes the body of the baseline retrieval tool below. Combine them as needed, and measure every change against a fixed dataset so you keep what helps.
Use LangSmith to score those experiments. The RAG evaluation tutorial shows how to measure retrieval and answer quality.

Before you begin

This guide assumes you have an indexed vector_store and expose retrieval to an agent as a tool. For indexing, see Build a semantic search engine. The baseline retrieval tool runs a single similarity search:
Pass the tool to an agent with create_agent:

Choose a technique

Start with the technique that matches the failure you observe, then layer others as needed. Reranking gives the largest gain for most applications and composes well with a broad first-stage retrieval. A common high-quality pipeline combines several: filter to the right scope, retrieve a broad candidate set with hybrid search, then rerank to a small set. Validate the combination on a dataset with LangSmith evaluation before shipping. The sections below change the body of the baseline tool. Each is independent, so adopt only the ones that address a problem you observe.

Filter by metadata

Metadata filtering restricts the search to a subset of documents before similarity search runs. It improves precision and reduces latency by shrinking the candidate set, and it is the most reliable way to scope retrieval to a tenant, a date range, a category, or a section. Filtering requires that you attach metadata to each chunk at indexing time. Apply a static filter when the scope is fixed:
To let the model choose the scope, expose the filter as a tool argument. The model populates it from the conversation:
Filter syntax is vector-store specific. The dictionary form shown here works with most production stores, such as Pinecone and PGVector. Some stores, including InMemoryVectorStore, take a predicate function instead. See your vector store integration for details.

Transform the query

A user question is not always a good search query. It can be vague, conversational, or contain several distinct information needs. Transforming the question into one or more focused queries before retrieval improves recall. Generate focused queries with structured output, run a search for each, then merge the results:
Retrieve for each generated query and deduplicate the results:
An agent already rewrites queries to some degree: it decides what to search for based on the conversation. Query transformation is most useful when you retrieve once per turn, as in a two-step RAG chain, or when a single search misses relevant documents.
Semantic (dense) search matches meaning, but it can miss exact product codes, identifiers, names, and rare keywords. Keyword (sparse) search matches those terms exactly. Hybrid search runs both and combines the rankings, which improves recall for queries that mix concepts with specific terms.
Many vector store integrations support hybrid search natively, which is the most efficient option for large corpora. See your vector store integration to enable it, then request hybrid results inside your retrieval tool.
To combine retrievers yourself, pair the vector store with a keyword retriever such as BM25Retriever and merge the two rankings with reciprocal rank fusion. BM25Retriever requires the rank_bm25 package and builds an in-memory index from the same chunks:

Diversify results with maximal marginal relevance

Plain similarity search can return several near-duplicate chunks, which wastes context on repeated information. Maximal marginal relevance (MMR) selects results that are both relevant to the query and diverse from one another. Use the vector store’s maximal marginal relevance search in place of similarity_search:
MMR fetches fetch_k candidates by similarity, then selects k that balance relevance against redundancy. Not all vector stores implement MMR; check your vector store integration.

Rerank retrieved documents

Embedding similarity is a coarse relevance signal, so the top result by similarity is not always the most relevant. A reranker scores each candidate against the query directly and reorders them, which usually produces the largest single quality gain. Use two-stage retrieval: fetch a broad candidate set with similarity search, then rerank down to a small, high-precision set. Call a reranker inside the tool. This example uses CohereRerank:
Reranking adds an extra step, and hosted rerankers add an API call per retrieval, which increases latency. In most RAG applications the quality gain is worth the cost. For a local option with no API call, use a cross-encoder reranker. Other hosted options include Voyage AI.

See also