Trace an agent

In this tutorial, you will build a customer support chatbot using retrieval-augmented generation (RAG) and add LangSmith observability at each stage of development, from early prototyping through production. By the end, you will know how to:

Trace individual LLM calls and full agent pipelines.
Collect and query user feedback.
Log metadata and use it for filtering and A/B testing.
Use monitoring dashboards to track production performance.

The agent will retrieve relevant documentation snippets and use them to answer user questions. The retriever is mocked in this tutorial; in a real agent you would replace it with a vector search or similar.

Prerequisites

Before you begin, make sure you have:

A LangSmith account: Sign up or log in at smith.langchain.com.
A LangSmith API key: Follow the Create an API key guide.
An OpenAI API key: Generate this from the OpenAI dashboard.
LangSmith CLI (optional): Install to inspect traces from the terminal. For instructions, refer to LangSmith CLI.

Install the required packages:

pip install langsmith openai

npm install langsmith openai
npm install -D typescript tsx

Prototyping

Having observability set up from the start lets you iterate faster. You can see exactly what is being sent to the model, what is coming back, and where time is being spent, without adding print statements or running a debugger.

Set up your environment

Set the following environment variables in your shell:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

Before sending traces, create the agent in LangSmith:

In an agent-based workspace, open + Agent, choose Observability, and create an agent with an Agent name and Agent ID. This requires the projects:create permission. If the agent already exists, reuse it.
Copy the Agent ID and choose an environment: local, development, staging, or production.

Configure agent addressing before running the agent:

export LANGSMITH_AGENT_ID="<your-agent-id>"
export LANGSMITH_AGENT_ENVIRONMENT="local"

Register the agent once, not for every trace. Remove LANGSMITH_PROJECT, LANGCHAIN_PROJECT, and LANGCHAIN_SESSION from the environment where your code runs. Do not combine legacy destination settings with agent addressing. To create the agent with an SDK call or API request, see Create an agent programmatically.

Project-based workspaces and self-hosted LangSmith

Agent addressing is available in agent-based workspaces on LangSmith Cloud. In a project-based workspace, including self-hosted LangSmith, set LANGSMITH_PROJECT to your tracing project name instead of setting the two agent variables.

You may see these variables referenced as LANGCHAIN_* in other places. Both work, but LANGSMITH_TRACING and LANGSMITH_API_KEY are the recommended names.

Trace LLM calls

Start by tracing your OpenAI calls, where the model is actually invoked. This gives you immediate visibility into the prompts your app sends and the responses the model returns. Wrap the OpenAI client with wrap_openai (Python) or wrapOpenAI (TypeScript). Create a file called app.py (or app.ts) with the following code:

from openai import OpenAI
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

async function supportBot(question: string): Promise<string> {
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
}

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

Calling support_bot("How many users can I have on the Starter plan?") produces a trace of the OpenAI call.

Trace the whole pipeline

Tracing the LLM call is useful, but tracing the full pipeline (including retrieval) gives you a complete overview of your agent’s behavior. Add @traceable (Python) or traceable (TypeScript) to the main function:

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable } from "langsmith/traceable";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

const supportBot = traceable(async function supportBot(question: string): Promise<string> {
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

Calling support_bot("How many users can I have on the Starter plan?") now produces a trace of the full RAG pipeline.

LangSmith UI showing a trace with an outer agent span and a nested LLM call span.

Check your traces from the terminal

If you installed the LangSmith CLI, list recent traces for your agent environment without opening the UI. The CLI’s --project option takes the environment’s tracer_session_name:

langsmith trace list --project "<environment-tracer-session-name>" --limit 5

To view the full run hierarchy and inputs/outputs for a specific trace:

langsmith trace get <trace-id> --full

Beta testing

Once your app is working well in prototyping, you release it to a small group of real users. At this stage, you often don’t know exactly how users will interact with your app, so you need richer observability. You want to understand not just what the app did, but how users responded to it.

Collect feedback

Linking user feedback to specific traces lets you identify which responses were helpful or unhelpful. Update app.py (or app.ts) from the previous step to add a run ID to each call and attach a score afterward:

import asyncio  
import os  

from openai import OpenAI
from langsmith import Client, agent_address, traceable, uuid7  
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    run_id = str(uuid7())
    support_bot(
        "How many users can I have on the Starter plan?",
        langsmith_extra={"run_id": run_id},
    )
    ls_client = Client()
    # Feedback requires the session_id for the agent environment  
    address = agent_address(
        os.environ["LANGSMITH_AGENT_ID"], os.environ["LANGSMITH_AGENT_ENVIRONMENT"]
    )
    resolved = asyncio.run(ls_client.sessions.resolve(**address))
    session_id = resolved.session_id  
    ls_client.create_feedback(
        run_id, key="user-score", score=1.0, session_id=session_id  
    )

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable, getCurrentRunTree } from "langsmith/traceable";
import { AgentAddress, Client } from "langsmith";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

let capturedRunId: string;

const supportBot = traceable(async function supportBot(question: string): Promise<string> {
    const runTree = getCurrentRunTree();
    capturedRunId = runTree.id;
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});

(async () => {
    await supportBot("How many users can I have on the Starter plan?");
    const lsClient = new Client();
    // Feedback requires the session_id for the agent environment
    const address = new AgentAddress( 
        process.env.LANGSMITH_AGENT_ID!, process.env.LANGSMITH_AGENT_ENVIRONMENT!
    );
    const { session_id: sessionId } = await lsClient.sessions.resolve(address);
    await lsClient.createFeedback({
        runId: capturedRunId,
        sessionId,
        key: "user-score",
        score: 1.0,
    });
    await lsClient.flush();
})();

In production, these two pieces would live in separate locations: the support_bot call with run_id stays in your agent code, and create_feedback moves to whichever endpoint receives user feedback (for example, a /feedback API route). The run_id is passed from one to the other so the feedback can be linked to the correct trace. Because feedback also requires the environment’s trace-storage UUID, pass session_id alongside the run_id.

In a project-based workspace, look up the project UUID with read_project / readProject instead of resolving an agent address. See Resolve an agent environment for address resolution requirements. The feedback appears in the Feedback tab when you inspect the run in the UI. You can then filter runs by feedback score using the filtering controls in the Runs table.

Log metadata

Metadata lets you tag runs with attributes useful for filtering and comparison. For example, which model version was used or which user made the request. The following example traces both the retriever (with run_type="retriever") and the main function (with a metadata attribute for the model name):

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

@traceable(run_type="retriever")
def retriever(query: str) -> list[str]:
    return docs

@traceable(metadata={"llm": "gpt-5.4-mini"})
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    support_bot("How many users can I have on the Starter plan?")

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable } from "langsmith/traceable";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

const retriever = traceable(  
    function retriever(query: string): string[] {
        return docs;
    },
    { run_type: "retriever" }
);

const supportBot = traceable(
    async function supportBot(question: string): Promise<string> {
        const context = await retriever(question);
        const systemMessage =
            "You are a helpful customer support agent. " +
            "Answer using only the information provided below:\n\n" +
            context.join("\n");
        const response = await client.chat.completions.create({
            model: "gpt-5.4-mini",
            messages: [
                { role: "system", content: systemMessage },
                { role: "user", content: question },
            ],
        });
        return response.choices[0].message?.content ?? "";
    },
    { metadata: { llm: "gpt-5.4-mini" } }
);

(async () => {
    await supportBot("How many users can I have on the Starter plan?");
})();

Both metadata values appear on the trace. You can filter runs by metadata using the filtering controls in the Runs table.

Production

With strong observability in place, you can confidently ship to production. In production, you have significantly more traffic and can’t examine every trace individually. LangSmith provides monitoring tools to help you understand aggregate behavior and drill down when something looks wrong.

Monitoring

Select your agent, then select Monitoring in the sidebar. Choose the environment under Environments. Charts display metrics for that environment over time, including trace count, latency, error rate, feedback scores, and costs. For more on available metrics and chart configuration, refer to Dashboards.

A/B testing

Group-by functionality requires at least two different values for a given metadata key.

Because you have been logging the llm metadata attribute, you can group monitoring charts by that attribute to compare model performance over time. From Monitoring in the UI sidebar, click Group by in the top left corner, select Metadata from the dropdown, then select llm. The charts update to show results grouped by that attribute. For more on grouping and custom charts, refer to Dashboards.

Drilldown

When a monitoring chart shows something unexpected, click a data point to freeze the tooltip, then click the metric name (for example, Input) to jump to the filtered runs table for that time window. For more on searching and filtering runs, refer to Filter traces.

LangSmith UI showing the monitoring page with a specific point on the Input Tokens chart highlighted.

Conclusion

In this tutorial, you added LangSmith observability to an agent across its full development lifecycle. The same tracing setup that helps you iterate quickly during prototyping will continue to provide value in production. You’ll have visibility into individual traces and aggregate performance trends. For more, see:

Observability concepts: terminology and core ideas.
Tracing integrations: LangChain, LangGraph, Anthropic, and other providers.
Automations: rules and online evaluations that run automatically on your traces.

Connect these docs to your agent of choice via MCP for real-time answers.

Edit this page on GitHub or file an issue.

Tracing setup

Configuration & troubleshooting

Prerequisites

Prototyping

Set up your environment

Trace LLM calls

Trace the whole pipeline

Check your traces from the terminal

Beta testing

Collect feedback

Log metadata

Production

Monitoring

A/B testing

Drilldown

Conclusion

​Prerequisites

​Prototyping

​Set up your environment

​Trace LLM calls

​Trace the whole pipeline

​Check your traces from the terminal

​Beta testing

​Collect feedback

​Log metadata

​Production

​Monitoring

​A/B testing

​Drilldown

​Conclusion

Prerequisites

Prototyping

Set up your environment

Trace LLM calls

Trace the whole pipeline

Check your traces from the terminal

Beta testing

Collect feedback

Log metadata

Production

Monitoring

A/B testing

Drilldown

Conclusion