Installation
You will need couchbase and langchain community to use couchbase vector store. For this tutorial, we will use OpenAI embeddingsnpm
Create Couchbase Connection Object
We create a connection to the Couchbase cluster initially and then pass the cluster object to the Vector Store. Here, we are connecting using the username and password. You can also connect using any other supported way to your cluster. For more information on connecting to the Couchbase cluster, please check the Node SDK documentation.Create the Search Index
Currently, the Search index needs to be created from the Couchbase Capella or Server UI or using the REST interface. For this example, let us use the Import Index feature on the Search Service on the UI. Let us define a Search index with the namevector-index
on the testing bucket.
We are defining an index on the testing
bucket’s _default
scope on the _default
collection with the vector field set to embedding
with 1536 dimensions and the text field set to text
.
We are also indexing and storing all the fields under metadata
in the document as a dynamic mapping to account for varying document structures. The similarity metric is set to dot_product
.
How to Import an Index to the Full Text Search service?
- Couchbase Server
- Click on Search -> Add Index -> Import
- Copy the following Index definition in the Import screen
- Click on Create Index to create the index.
- Couchbase Capella
- Copy the following index definition to a new file
index.json
- Import the file in Capella using the instructions in the documentation.
- Click on Create Index to create the index.
- Copy the following index definition to a new file
Index Definition
Create Vector Store
We create the vector store object with the cluster information and the search index name.Basic Vector Search Example
The following example showcases how to use couchbase vector search and perform similarity search. For this example, we are going to load the “state_of_the_union.txt” file via the TextLoader, chunk the text into 500 character chunks with no overlaps and index all these chunks into Couchbase. After the data is indexed, we perform a simple query to find the top 4 chunks that are similar to the query “What did president say about Ketanji Brown Jackson”. At the end, it also shows how to get similarity scoreSpecifying Fields to Return
You can specify the fields to return from the document usingfields
parameter in the filter during searches.
These fields are returned as part of the metadata
object. You can fetch any field that is stored in the index.
The textKey
of the document is returned as part of the document’s pageContent
.
If you do not specify any fields to be fetched, all the fields stored in the index are returned.
If you want to fetch one of the fields in the metadata, you need to specify it using .
For example, to fetch the source
field in the metadata, you need to use metadata.source
.
Hybrid Search
Couchbase allows you to do hybrid searches by combining vector search results with searches on non-vector fields of the document like themetadata
object.
The results will be based on the combination of the results from both vector search and the searches supported by full text search service.
The scores of each of the component searches are added up to get the total score of the result.
To perform hybrid searches, there is an optional key, searchOptions
in fields
parameter that can be passed to all the similarity searches.
The different search/query possibilities for the searchOptions
can be found here.
Create Diverse Metadata for Hybrid Search
In order to simulate hybrid search, let us create some random metadata from the existing documents. We uniformly add three fields to the metadata,date
between 2010 & 2020, rating
between 1 & 5 and author
set to either John Doe or Jane Doe.
We will also declare few sample queries.
Example: Search by Exact Value
We can search for exact matches on a textual field like the author in themetadata
object.
Example: Search by Partial Match
We can search for partial matches by specifying a fuzziness for the search. This is useful when you want to search for slight variations or misspellings of a search query. Here, “Johny” is close (fuzziness of 1) to “John Doe”.Example: Search by Date Range Query
We can search for documents that are within a date range query on a date field likemetadata.date
.
Example: Search by Numeric Range Query
We can search for documents that are within a range for a numeric field likemetadata.rating
.
Example: Combining Multiple Search Conditions
Different queries can by combined using AND (conjuncts) or OR (disjuncts) operators. In this example, we are checking for documents with a rating between 3 & 4 and dated between 2015 & 2018.Other Queries
Similarly, you can use any of the supported Query methods like Geo Distance, Polygon Search, Wildcard, Regular Expressions, etc in thesearchOptions
Key of filter
parameter.
Please refer to the documentation for more details on the available query methods and their syntax.
Frequently Asked Questions
Question: Should I create the Search index before creating the CouchbaseVectorStore object?
Yes, currently you need to create the Search index before creating theCouchbaseVectorStore
object.
Question: I am not seeing all the fields that I specified in my search results.
In Couchbase, we can only return the fields stored in the Search index. Please ensure that the field that you are trying to access in the search results is part of the Search index. One way to handle this is to index and store a document’s fields dynamically in the index.- In Capella, you need to go to “Advanced Mode” then under the chevron “General Settings” you can check “[X] Store Dynamic Fields” or “[X] Index Dynamic Fields”
- In Couchbase Server, in the Index Editor (not Quick Editor) under the chevron “Advanced” you can check “[X] Store Dynamic Fields” or “[X] Index Dynamic Fields”
Question: I am unable to see the metadata object in my search results.
This is most likely due to themetadata
field in the document not being indexed and/or stored by the Couchbase Search index. In order to index the metadata
field in the document, you need to add it to the index as a child mapping.
If you select to map all the fields in the mapping, you will be able to search by all metadata fields. Alternatively, to optimize the index, you can select the specific fields inside metadata
object to be indexed.
You can refer to the docs to learn more about indexing child mappings.
To create Child Mappings, you can refer to the following docs -
Related
- Vector store conceptual guide
- Vector store how-to guides