> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA integration

> Integrate with the NVIDIA LLM using LangChain Python.

This will help you get started with NVIDIA models. For detailed documentation of all `ChatNVIDIA` features and configurations head to the [API reference](https://reference.langchain.com/python/langchain-nvidia-ai-endpoints/chat_models/ChatNVIDIA).

## Overview

The `langchain-nvidia-ai-endpoints` package contains LangChain integrations building applications with models on
NVIDIA NIM inference microservice. These models are optimized by NVIDIA to deliver the best performance on NVIDIA
accelerated infrastructure and deployed as a NIM, an easy-to-use, prebuilt containers that deploy anywhere using a single
command on NVIDIA accelerated infrastructure.

NVIDIA hosted deployments of NIMs are available to test on the [NVIDIA API catalog](https://build.nvidia.com/). After testing,
NIMs can be exported from NVIDIA’s API catalog using the NVIDIA AI Enterprise license and run on-premises or in the cloud,
giving enterprises ownership and full control of their IP and AI application.

NIMs are packaged as container images on a per model basis and are distributed as NGC container images through the NVIDIA NGC Catalog.
At their core, NIMs provide easy, consistent, and familiar APIs for running inference on an AI model.

This example goes over how to use LangChain to interact with NVIDIA supported via the `NVIDIA` class.

For more information on accessing the llm models through this api, check out the [NVIDIA](https://python.langchain.com/docs/integrations/llms/nvidia_ai_endpoints/) documentation.

### Integration details

| Class                                                                                                   | Package                                                                                                 | Local | Serializable | JS support |                                                    Downloads                                                   |                                                   Version                                                   |
| :------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------ | :---: | :----------: | :--------: | :------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------: |
| [`NVIDIA`](https://reference.langchain.com/python/langchain-nvidia-ai-endpoints/chat_models/ChatNVIDIA) | [`langchain-nvidia-ai-endpoints`](https://reference.langchain.com/python/langchain-nvidia-ai-endpoints) |   ✅   |     beta     |      ❌     | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain_nvidia_ai_endpoints?style=flat-square\&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain_nvidia_ai_endpoints?style=flat-square\&label=%20) |

### Model features

| [Image input](/oss/python/langchain/messages#multimodal) | Audio input | Video input | [Token-level streaming](/oss/python/langchain/streaming/) | Native async | [Token usage](/oss/python/langchain/models#token-usage) | [Logprobs](/oss/python/langchain/models#log-probabilities) |
| :------------------------------------------------------: | :---------: | :---------: | :-------------------------------------------------------: | :----------: | :-----------------------------------------------------: | :--------------------------------------------------------: |
|                             ✅                            |      ❌      |      ❌      |                             ✅                             |       ❌      |                            ❌                            |                              ❌                             |

## Setup

**To get started:**

1. Create a free account with [NVIDIA](https://build.nvidia.com/), which hosts NVIDIA AI Foundation models.

2. Click on your model of choice.

3. Under `Input` select the `Python` tab, and click `Get API Key`. Then click `Generate Key`.

4. Copy and save the generated key as `NVIDIA_API_KEY`. From there, you should have access to the endpoints.

### Credentials

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
import getpass
import os

if not os.getenv("NVIDIA_API_KEY"):
    # Note: the API key should start with "nvapi-"
    os.environ["NVIDIA_API_KEY"] = getpass.getpass("Enter your NVIDIA API key: ")
```

### Installation

The LangChain NVIDIA AI Endpoints integration lives in the `langchain-nvidia-ai-endpoints` package:

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
pip install -qU langchain-nvidia-ai-endpoints
```

## Instantiation

See [LLM](/oss/python/langchain/models) for full functionality.

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_nvidia_ai_endpoints import NVIDIA
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
llm = NVIDIA().bind(max_tokens=256)
llm
```

## Invocation

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
prompt = "# Function that does quicksort written in Rust without comments:"
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
print(llm.invoke(prompt))
```

## Stream, batch, and async

These models natively support streaming, and as is the case with all LangChain LLMs they expose a batch method to handle concurrent requests, as well as async methods for invoke, stream, and batch. Below are a few examples.

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
stream = llm.stream_events(prompt, version="v3")
for token in stream.text:
    print(token, end="", flush=True)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
llm.batch([prompt])
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
await llm.ainvoke(prompt)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
stream = await llm.astream_events(prompt, version="v3")
async for token in stream.text:
    print(token, end="", flush=True)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
await llm.abatch([prompt])
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
async for chunk in llm.astream_log(prompt):
    print(chunk)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
response = llm.invoke(
    "X_train, y_train, X_test, y_test = train_test_split(X, y, test_size=0.1) #Train a logistic regression model, predict the labels on the test set and compute the accuracy score"
)
print(response)
```

## Supported models

Querying `available_models` will still give you all of the other models offered by your API credentials.

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
NVIDIA.get_available_models()
# llm.get_available_models()
```

***

## API reference

For detailed documentation of all `ChatNVIDIA ` features and configurations head to the [API reference](https://reference.langchain.com/python/langchain-nvidia-ai-endpoints/chat_models/ChatNVIDIA)

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/oss/python/integrations/llms/nvidia_ai_endpoints.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>