> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM integration

> Integrate with the vLLM chat model using LangChain Python.

vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.

## Overview

This will help you get started with vLLM [chat models](/oss/python/langchain/models), which leverages the `langchain-openai` package. For detailed documentation of all [`ChatOpenAI`](https://reference.langchain.com/python/langchain-openai/chat_models/base/ChatOpenAI) features and configurations head to the [API reference](https://reference.langchain.com/python/langchain-openai/chat_models/base/ChatOpenAI).

### Integration details

| Class                                                                                               | Package                                                                       | Serializable | JS support |                                             Downloads                                             |                                             Version                                            |
| :-------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- | :----------: | :--------: | :-----------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------: |
| [`ChatOpenAI`](https://reference.langchain.com/python/langchain-openai/chat_models/base/ChatOpenAI) | [`langchain_openai`](https://reference.langchain.com/python/langchain-openai) |     beta     |      ❌     | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain_openai?style=flat-square\&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain_openai?style=flat-square\&label=%20) |

### Model features

Specific model features, such as tool calling, support for multi-modal inputs, support for token-level streaming, etc., will depend on the hosted model.

## Setup

See the [vLLM documentation](https://docs.vllm.ai/en/latest/).

To access vLLM models through LangChain, you'll need to install the `langchain-openai` integration package.

### Credentials

Authentication will depend on specifics of the inference server.

To enable automated tracing of your model calls, set your [LangSmith](/langsmith/observability) API key:

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")
```

### Installation

The LangChain vLLM integration can be accessed via the `langchain-openai` package:

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
pip install -qU langchain-openai
```

## Instantiation

Now we can instantiate our model object and generate chat completions:

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain.messages import HumanMessage, SystemMessage
from langchain_openai import ChatOpenAI
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
inference_server_url = "http://localhost:8000/v1"

llm = ChatOpenAI(
    model="mosaicml/mpt-7b",
    api_key="your api key goes here",
    base_url=inference_server_url,
    max_tokens=5,
    temperature=0,
)
```

## Invocation

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
messages = [
    SystemMessage(
        content="You are a helpful assistant that translates English to Italian."
    ),
    HumanMessage(
        content="Translate the following sentence from English to Italian: I love programming."
    ),
]
llm.invoke(messages)
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
AIMessage(content=' Io amo programmare', additional_kwargs={}, example=False)
```

***

## API reference

For detailed documentation of all features and configurations exposed via `langchain-openai`, head to the API reference: [reference.langchain.com/python/langchain-openai/chat\_models/base/ChatOpenAI](https://reference.langchain.com/python/langchain-openai/chat_models/base/ChatOpenAI)

Refer to the vLLM [documentation](https://docs.vllm.ai/en/latest/) as well.

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/oss/python/integrations/chat/vllm.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>