> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# ChatOCIModelDeployment integration

> Integrate with the ChatOCIModelDeployment chat model using LangChain Python.

This will help you get started with OCIModelDeployment [chat models](/oss/python/langchain/models). For detailed documentation of all `ChatOCIModelDeployment` features and configurations, see the [langchain-oci package](https://github.com/oracle/langchain-oracle/tree/main/libs/oci).

[OCI Data Science](https://docs.oracle.com/en-us/iaas/data-science/using/home.htm) is a fully managed and serverless platform for data science teams to build, train, and manage machine learning models in the Oracle Cloud Infrastructure. You can use [AI Quick Actions](https://blogs.oracle.com/ai-and-datascience/post/ai-quick-actions-in-oci-data-science) to easily deploy LLMs on [OCI Data Science Model Deployment Service](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-about.htm). You may choose to deploy the model with popular inference frameworks such as vLLM or TGI. By default, the model deployment endpoint mimics the OpenAI API protocol.

> For the latest updates, examples and experimental features, please see [ADS LangChain Integration](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/large_language_model/langchain_models.html).

## Overview

### Integration details

| Class                                                                                     | Package                                                    | Serializable | JS support |                                            Downloads                                           |                                           Version                                           |
| :---------------------------------------------------------------------------------------- | :--------------------------------------------------------- | :----------: | :--------: | :--------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------: |
| [`ChatOCIModelDeployment`](https://github.com/oracle/langchain-oracle/tree/main/libs/oci) | [`langchain-oci`](https://pypi.org/project/langchain-oci/) |     beta     |      ❌     | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-oci?style=flat-square\&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-oci?style=flat-square\&label=%20) |

### Model features

| [Tool calling](/oss/python/langchain/tools) | [Structured output](/oss/python/langchain/structured-output) | [Image input](/oss/python/langchain/messages#multimodal) | Audio input | Video input | [Token-level streaming](/oss/python/langchain/streaming/) | Native async | [Token usage](/oss/python/langchain/models#token-usage) | [Logprobs](/oss/python/langchain/models#log-probabilities) |
| :-----------------------------------------: | :----------------------------------------------------------: | :------------------------------------------------------: | :---------: | :---------: | :-------------------------------------------------------: | :----------: | :-----------------------------------------------------: | :--------------------------------------------------------: |
|                   depends                   |                            depends                           |                          depends                         |   depends   |   depends   |                             ✅                             |       ✅      |                            ✅                            |                              ✅                             |

Some model features, including tool calling, structured output, and multi-modal inputs, are depending on deployed model.

## Setup

To use `ChatOCIModelDeployment`, deploy a chat model with a chat completion endpoint and install the `langchain-oci` and `oracle-ads` packages.

You can easily deploy foundation models using the [AI Quick Actions](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/ai-quick-actions/model-deployment-tips.md) on OCI Data Science Model deployment. For additional deployment examples, please visit the [Oracle GitHub samples repository](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/ai-quick-actions).

### Policies

Make sure to have the required [policies](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm#model_dep_policies_auth__predict-endpoint) to access the OCI Data Science Model Deployment endpoint.

### Credentials

You can set authentication through Oracle ADS. When you are working in OCI Data Science Notebook Session, you can leverage resource principal to access other OCI resources.

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
import ads

# Set authentication through ads
# Use resource principal are operating within a
# OCI service that has resource principal based
# authentication configured
ads.set_auth("resource_principal")
```

Alternatively, you can configure the credentials using the following environment variables. For example, to use API key with specific profile:

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
import os

# Set authentication through environment variables
# Use API Key setup when you are working from a local
# workstation or on platform which does not support
# resource principals.
os.environ["OCI_IAM_TYPE"] = "api_key"
os.environ["OCI_CONFIG_PROFILE"] = "default"
os.environ["OCI_CONFIG_LOCATION"] = "~/.oci"
```

Check out [Oracle ADS docs](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html) to see more options.

### Installation

The LangChain OCIModelDeployment integration lives in the `langchain-oci` package.

<CodeGroup>
  ```bash pip theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  pip install -qU langchain-oci oracle-ads
  ```

  ```bash uv theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  uv add langchain-oci oracle-ads
  ```
</CodeGroup>

## Instantiation

You may instantiate the model with the generic `ChatOCIModelDeployment` or framework specific class like `ChatOCIModelDeploymentVLLM`.

* Using `ChatOCIModelDeployment` when you need a generic entry point for deploying models. You can pass model parameters through `model_kwargs` during the instantiation of this class. This allows for flexibility and ease of configuration without needing to rely on framework-specific details.

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_oci import ChatOCIModelDeployment

# Create an instance of OCI Model Deployment Endpoint
# Replace the endpoint uri with your own
# Using generic class as entry point, you will be able
# to pass model parameters through model_kwargs during
# instantiation.
chat = ChatOCIModelDeployment(
    endpoint="https://modeldeployment.<region>.oci.customer-oci.com/<ocid>/predict",
    streaming=True,
    max_retries=1,
    model_kwargs={
        "temperature": 0.2,
        "max_tokens": 512,
    },  # other model params...
    default_headers={
        "route": "/v1/chat/completions",
        # other request headers ...
    },
)
```

* Using framework specific class like `ChatOCIModelDeploymentVLLM`: This is suitable when you are working with a specific framework (e.g. `vLLM`) and need to pass model parameters directly through the constructor, streamlining the setup process.

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_oci import ChatOCIModelDeploymentVLLM

# Create an instance of OCI Model Deployment Endpoint
# Replace the endpoint uri with your own
# Using framework specific class as entry point, you will
# be able to pass model parameters in constructor.
chat = ChatOCIModelDeploymentVLLM(
    endpoint="https://modeldeployment.<region>.oci.customer-oci.com/<md_ocid>/predict",
)
```

## Invocation

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
messages = [
    (
        "system",
        "You are a helpful assistant that translates English to French. Translate the user sentence.",
    ),
    ("human", "I love programming."),
]

ai_msg = chat.invoke(messages)
ai_msg
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
AIMessage(content="J'adore programmer.", response_metadata={'token_usage': {'prompt_tokens': 44, 'total_tokens': 52, 'completion_tokens': 8}, 'model_name': 'odsc-llm', 'system_fingerprint': '', 'finish_reason': 'stop'}, id='run-ca145168-efa9-414c-9dd1-21d10766fdd3-0')
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
print(ai_msg.content)
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
J'adore programmer.
```

## Chaining

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful assistant that translates {input_language} to {output_language}.",
        ),
        ("human", "{input}"),
    ]
)

chain = prompt | chat
chain.invoke(
    {
        "input_language": "English",
        "output_language": "German",
        "input": "I love programming.",
    }
)
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
AIMessage(content='Ich liebe Programmierung.', response_metadata={'token_usage': {'prompt_tokens': 38, 'total_tokens': 48, 'completion_tokens': 10}, 'model_name': 'odsc-llm', 'system_fingerprint': '', 'finish_reason': 'stop'}, id='run-5dd936b0-b97e-490e-9869-2ad3dd524234-0')
```

## Asynchronous calls

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_oci import ChatOCIModelDeployment

system = "You are a helpful translator that translates {input_language} to {output_language}."
human = "{text}"
prompt = ChatPromptTemplate.from_messages([("system", system), ("human", human)])

chat = ChatOCIModelDeployment(
    endpoint="https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict"
)
chain = prompt | chat

await chain.ainvoke(
    {
        "input_language": "English",
        "output_language": "Chinese",
        "text": "I love programming",
    }
)
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
AIMessage(content='我喜欢编程', response_metadata={'token_usage': {'prompt_tokens': 37, 'total_tokens': 50, 'completion_tokens': 13}, 'model_name': 'odsc-llm', 'system_fingerprint': '', 'finish_reason': 'stop'}, id='run-a2dc9393-f269-41a4-b908-b1d8a92cf827-0')
```

## Streaming calls

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
import sys

from langchain_oci import ChatOCIModelDeployment
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages(
    [("human", "List out the 5 states in the United State.")]
)

chat = ChatOCIModelDeployment(
    endpoint="https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict"
)

chain = prompt | chat

for chunk in chain.stream({}):
    sys.stdout.write(chunk.content)
    sys.stdout.flush()
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
1. California
2. Texas
3. Florida
4. New York
5. Illinois
```

## Structured output

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_oci import ChatOCIModelDeployment
from pydantic import BaseModel


class Joke(BaseModel):
    """A setup to a joke and the punchline."""

    setup: str
    punchline: str


chat = ChatOCIModelDeployment(
    endpoint="https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict",
)
structured_llm = chat.with_structured_output(Joke, method="json_mode")
output = structured_llm.invoke(
    "Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys"
)

output.dict()
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'setup': 'Why did the cat get stuck in the tree?',
 'punchline': 'Because it was chasing its tail!'}
```

## API reference

For comprehensive details on all features and configurations, see the [langchain-oci package](https://github.com/oracle/langchain-oracle/tree/main/libs/oci) documentation for each class:

* `ChatOCIModelDeployment`
* `ChatOCIModelDeploymentVLLM`
* `ChatOCIModelDeploymentTGI`

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/oss/python/integrations/chat/oci_data_science.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>