Embedding models | 🦜️🔗 LangChain

📄️ AI21 Labs

This notebook covers how to get started with AI21 embedding models.

There are two possible ways to use Aleph Alpha's semantic embeddings. If you have texts with a dissimilar structure (e.g. a Document and a Query) you would want to use asymmetric embeddings. Conversely, for texts with comparable structures, symmetric embeddings are the suggested approach.

📄️ Anyscale

Let's load the Anyscale Embedding class.

📄️ AwaDB

AwaDB is an AI Native database for the search and storage of embedding vectors used by LLM Applications.

📄️ Azure OpenAI

Let's load the Azure OpenAI Embedding class with environment variables set to indicate to use Azure endpoints.

📄️ Baichuan Text Embeddings

As of today (Jan 25th, 2024) BaichuanTextEmbeddings ranks #1 in C-MTEB (Chinese Multi-Task Embedding Benchmark) leaderboard.

📄️ Baidu Qianfan

Baidu AI Cloud Qianfan Platform is a one-stop large model development and service operation platform for enterprise developers. Qianfan not only provides including the model of Wenxin Yiyan (ERNIE-Bot) and the third-party open-source models, but also provides various AI development tools and the whole set of development environment, which facilitates customers to use and develop large model applications easily.

📄️ Bedrock

Amazon Bedrock is a fully managed service that offers a choice of

📄️ BGE on Hugging Face

BGE models on the HuggingFace are the best open-source embedding models.

📄️ Bookend AI

Let's load the Bookend AI Embeddings class.

📄️ Clarifai

Clarifai is an AI Platform that provides the full AI lifecycle ranging from data exploration, data labeling, model training, evaluation, and inference.

📄️ Cloudflare Workers AI

Cloudflare, Inc. (Wikipedia) is an American company that provides content delivery network services, cloud cybersecurity, DDoS mitigation, and ICANN-accredited domain registration services.

📄️ Cohere

Let's load the Cohere Embedding class.

📄️ DashScope

Let's load the DashScope Embedding class.

📄️ DeepInfra

DeepInfra is a serverless inference as a service that provides access to a variety of LLMs and embeddings models. This notebook goes over how to use LangChain with DeepInfra for text embeddings.

📄️ EDEN AI

Eden AI is revolutionizing the AI landscape by uniting the best AI providers, empowering users to unlock limitless possibilities and tap into the true potential of artificial intelligence. With an all-in-one comprehensive and hassle-free platform, it allows users to deploy AI features to production lightning fast, enabling effortless access to the full breadth of AI capabilities via a single API. (website//edenai.co/)

📄️ Elasticsearch

Walkthrough of how to generate embeddings using a hosted embedding model in Elasticsearch

📄️ Embaas

embaas is a fully managed NLP API service that offers features like embedding generation, document text extraction, document to embeddings and more. You can choose a variety of pre-trained models.

📄️ ERNIE

ERNIE Embedding-V1 is a text representation model based on Baidu Wenxin large-scale model technology,

📄️ Fake Embeddings

LangChain also provides a fake embedding class. You can use this to test your pipelines.

📄️ FastEmbed by Qdrant

FastEmbed from Qdrant is a lightweight, fast, Python library built for embedding generation.

📄️ FireworksEmbeddings

This notebook explains how to use Fireworks Embeddings, which is included in the langchain_fireworks package, to embed texts in langchain. We use the default nomic-ai v1.5 model in this example.

📄️ GigaChat

This notebook shows how to use LangChain with GigaChat embeddings.

📄️ Google Generative AI Embeddings

Connect to Google's generative AI embeddings service using the GoogleGenerativeAIEmbeddings class, found in the langchain-google-genai package.

📄️ Google Vertex AI PaLM

Vertex AI PaLM API is a service on Google Cloud exposing the embedding models.

📄️ GPT4All

GPT4All is a free-to-use, locally running, privacy-aware chatbot. There is no GPU or internet required. It features popular models and its own models such as GPT4All Falcon, Wizard, etc.

📄️ Gradient

Gradient allows to create Embeddings as well fine tune and get completions on LLMs with a simple web API.

📄️ Hugging Face

Let's load the Hugging Face Embedding class.

📄️ IBM watsonx.ai

WatsonxEmbeddings is a wrapper for IBM watsonx.ai foundation models.

📄️ Infinity

Infinity allows to create Embeddings using a MIT-licensed Embedding Server.

📄️ Instruct Embeddings on Hugging Face

Hugging Face sentence-transformers is a Python framework for state-of-the-art sentence, text and image embeddings.

📄️ Intel® Extension for Transformers Quantized Text Embeddings

Load quantized BGE embedding models generated by Intel® Extension for Transformers (ITREX) and use ITREX Neural Engine, a high-performance NLP backend, to accelerate the inference of models without compromising accuracy.

📄️ Jina

Let's load the Jina Embedding class.

📄️ John Snow Labs

John Snow Labs NLP & LLM ecosystem includes software libraries for state-of-the-art AI at scale, Responsible AI, No-Code AI, and access to over 20,000 models for Healthcare, Legal, Finance, etc.

📄️ LASER Language-Agnostic SEntence Representations Embeddings by Meta AI

LASER is a Python library developed by the Meta AI Research team and used for creating multilingual sentence embeddings for over 147 languages as of 2/25/2024

📄️ Llama-cpp

This notebook goes over how to use Llama-cpp embeddings within LangChain

📄️ llamafile

Let's load the llamafile Embeddings class.

📄️ LLMRails

Let's load the LLMRails Embeddings class.

📄️ LocalAI

Let's load the LocalAI Embedding class. In order to use the LocalAI Embedding class, you need to have the LocalAI service hosted somewhere and configure the embedding models. See the documentation at https//localai.io/features/embeddings/index.html.

📄️ MiniMax

MiniMax offers an embeddings service.

📄️ MistralAI

This notebook explains how to use MistralAIEmbeddings, which is included in the langchain_mistralai package, to embed texts in langchain.

📄️ ModelScope

ModelScope is big repository of the models and datasets.

📄️ MosaicML

MosaicML offers a managed inference service. You can either use a variety of open-source models, or deploy your own.

📄️ NVIDIA NeMo embeddings

Connect to NVIDIA's embedding service using the NeMoEmbeddings class.

📄️ NLP Cloud

NLP Cloud is an artificial intelligence platform that allows you to use the most advanced AI engines, and even train your own engines with your own data.

📄️ Nomic

This notebook covers how to get started with Nomic embedding models.

📄️ NVIDIA AI Foundation Endpoints

NVIDIA AI Foundation Endpoints give users easy access to NVIDIA hosted API endpoints for NVIDIA AI Foundation Models like Mixtral 8x7B, Llama 2, Stable Diffusion, etc. These models, hosted on the NVIDIA API catalog, are optimized, tested, and hosted on the NVIDIA AI platform, making them fast and easy to evaluate, further customize, and seamlessly run at peak performance on any accelerated stack.

📄️ oci_generative_ai

Oracle Cloud Infrastructure Generative AI

📄️ Ollama

Let's load the Ollama Embeddings class.

📄️ OpenClip

OpenClip is an source implementation of OpenAI's CLIP.

📄️ OpenAI

Let's load the OpenAI Embedding class.

📄️ OpenVINO

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference. The OpenVINO™ Runtime supports various hardware devices including x86 and ARM CPUs, and Intel GPUs. It can help to boost deep learning performance in Computer Vision, Automatic Speech Recognition, Natural Language Processing and other common tasks.

📄️ Embedding Documents using Optimized and Quantized Embedders

Embedding all documents using Quantized Embedders.

📄️ Oracle AI Vector Search: Generate Embeddings

Oracle AI Vector Search is designed for Artificial Intelligence (AI) workloads that allows you to query data based on semantics, rather than keywords. One of the biggest benefit of Oracle AI Vector Search is that semantic search on unstructured data can be combined with relational search on business data in one single system. This is not only powerful but also significantly more effective because you don't need to add a specialized vector database, eliminating the pain of data fragmentation between multiple systems.

📄️ PremAI

PremAI is an unified platform that let's you build powerful production-ready GenAI powered applications with least effort, so that you can focus more on user experience and overall growth. In this section we are going to dicuss how we can get access to different embedding model using PremAIEmbeddings