> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LangSmith Polly

**LangSmith Polly** is an AI assistant embedded directly in your LangSmith [workspace](/langsmith/administration-overview#workspaces) to help you analyze and understand your application data.

Polly helps you gain insight from your traces, conversation threads, and prompts without having to dig through data manually. By asking natural language questions, you can quickly understand agent performance, debug issues, and analyze user sentiment.

<img src="https://mintcdn.com/langchain-5e9cc07a/NVEcSyCXgAhfub1c/images/brand/polly-icon.png?fit=max&auto=format&n=NVEcSyCXgAhfub1c&q=85&s=d58c4caab502fcaa02864b5a8f833604" alt="LangSmith Polly icon" style={{float: 'left', marginRight: '20px', marginTop: '-1px', marginBottom: '20px', maxWidth: '100px'}} width="650" height="650" data-path="images/brand/polly-icon.png" /> Polly appears in the right-hand bottom corner of the following locations within [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-polly):

<br /><br />**Observability & Debugging:**

* [Projects](#projects): Browse and filter runs across a project.
* [Trace pages](#trace-pages): Analyze individual runs and execution traces.
* [Thread views](#thread-views): Understand conversation threads and user interactions.

**Prompt Engineering:**

* [Playground](#playground): Edit and optimize prompts.
* [Prompt Hub pages](#prompt-hub-pages): Explore and understand shared prompts.

**Evaluation & Testing:**

* [Dataset Experiments](#dataset-experiments): Analyze experiment results and compare runs.
* [Dataset Examples](#dataset-examples): Browse and understand dataset structure.
* [Annotation Queues](#annotation-queues): Review runs and make informed annotation decisions.
* [Evaluators](#evaluators): Build and refine evaluators with AI assistance.

<img className="block dark:hidden" src="https://mintcdn.com/langchain-5e9cc07a/Cd9j6FYj16b1asiM/langsmith/images/polly-datasets-light.png?fit=max&auto=format&n=Cd9j6FYj16b1asiM&q=85&s=771a678928374d181e5d9d5e33d16cb0" alt="Polly chat in the sidebar on a dataset view." width="1437" height="747" data-path="langsmith/images/polly-datasets-light.png" />

<img className="hidden dark:block" src="https://mintcdn.com/langchain-5e9cc07a/Cd9j6FYj16b1asiM/langsmith/images/polly-datasets-dark.png?fit=max&auto=format&n=Cd9j6FYj16b1asiM&q=85&s=e58462239dc037d880ec2c309e86f463" alt="Polly chat in the sidebar on a dataset view." width="1436" height="751" data-path="langsmith/images/polly-datasets-dark.png" />

## Get started

Before you start using Polly, you need to add an API key for the model you're using:

In the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=snippets-langsmith-set-workspace-secrets), ensure that your API key is set as a [workspace secret](/langsmith/set-up-hierarchy#configure-workspace-settings).

1. Navigate to <Icon icon="settings" /> **Settings** and then move to the **Secrets** tab.
2. Select **Add secret** and enter the key environment variable (e.g.,`OPENAI_API_KEY` or `ANTHROPIC_API_KEY`) and your API key as the **Value**.
3. Select **Save secret**.

<Note> When adding workspace secrets in the LangSmith UI, make sure the secret keys match the environment variable names expected by your model provider.</Note>

<Note>
  Polly calls model providers from LangSmith's egress IP addresses. If your model provider (or a proxy in front of it) restricts traffic by IP, allowlist the LangSmith egress IPs listed in [Allowlist IP addresses](/langsmith/deploy-to-cloud#allowlist-ip-addresses).
</Note>

### Supported models

Polly supports the following model providers out of the box:

* Anthropic (Claude)
* OpenAI
* Google Gemini
* AWS Bedrock
* Groq
* Mistral
* xAI
* DeepSeek
* Fireworks AI

You can also use any custom model you've configured in [Playground Settings](/langsmith/prompt-engineering-concepts#playground) by enabling the **Available in Polly** toggle on that configuration. Workspace admins manage which custom models are available.

### Keyboard shortcuts

| Action                   | Mac           | Windows/Linux  |
| ------------------------ | ------------- | -------------- |
| Toggle Polly open/closed | `Cmd+I`       | `Ctrl+I`       |
| Clear current thread     | `Cmd+Shift+O` | `Ctrl+Shift+O` |

## Observability

### Projects

On a project's run list, Polly can browse and filter runs across the entire project, create datasets, and add examples. Use Polly to quickly explore what's happening across your traces without manually paging through results.

**Example questions:**

* "Show me all the failed runs from the last 24 hours"
* "Which runs took the longest?"
* "Add the failing runs to my test dataset"
* "How many runs errored this week?"

### Trace pages

On an individual [trace](/langsmith/observability-concepts#traces), Polly analyzes the [run](/langsmith/observability-concepts#runs) data and execution trajectory. Polly examines the full trace context, including [run metadata](/langsmith/observability-concepts#metadata), inputs, outputs, intermediate steps, and configuration to help you understand what happened and identify areas for improvement.

**Example questions:**

* "Is there anything that the agent could have done better here?"
* "Why did this run fail?"
* "What took the most time in this trace?"
* "Summarize what happened in this trace"

### Thread views

Under the **Threads** tab, Polly analyzes conversation [threads](/langsmith/observability-concepts#threads) to help you understand user sentiment, conversation outcomes, and interaction patterns. Use Polly to identify user pain points and understand whether issues were resolved.

**Example questions:**

* "Did the user seem frustrated?"
* "What issues is the user experiencing?"
* "Was the user's problem solved?"
* "What was the main topic of this thread?"

## Prompt engineering

### Playground

In the [Playground](/langsmith/prompt-engineering-concepts#playground), Polly helps you edit and optimize your [prompts](/langsmith/prompt-engineering-concepts#prompts-in-langsmith). Use automated options like **Optimize prompt**, **Generate a tool**, or **Generate an output schema**, or give Polly custom instructions for editing your prompt. Polly can directly modify the playground state—updating messages, tools, output schemas, and examples—so you can iterate on prompts conversationally.

**Example questions:**

* "Make it respond in Italian"
* "Add more context about the user's role"
* "Make the tone more professional"
* "Simplify the instructions"

### Prompt Hub pages

When viewing a prompt in the [LangSmith Hub](/langsmith/prompt-engineering-concepts#prompts-in-langsmith), Polly helps you understand the prompt's structure, messages, tools, and configuration. This is useful for exploring and learning from shared prompts.

**Example questions:**

* "What does this prompt do?"
* "What tools does this prompt use?"
* "Explain the structure of this prompt"
* "What are the key instructions in this prompt?"

## Evaluation

### Dataset Experiments

On the **Datasets** page under the **Experiments** tab, Polly analyzes experiment results and helps you compare runs across different experiments. Polly can identify patterns, summarize performance, and help you understand which approaches work best.

**Example questions:**

* "Which experiment performed best?"
* "What are the main differences between these runs?"
* "Summarize the results of this experiment"
* "What patterns do you see in the failures?"

### Dataset Examples

On the **Datasets** page under the **Examples** tab, Polly helps you understand your dataset structure, browse examples, and identify data patterns. This is useful for understanding what data you're working with and preparing datasets for experiments.

**Example questions:**

* "What type of data is in this dataset?"
* "Show me examples with errors"
* "What patterns do you see in the inputs?"
* "How many examples are in this dataset?"

### Annotation Queues

In **Annotation Queues**, Polly helps you analyze runs before making annotation decisions. Whether you're reviewing runs individually or comparing them pairwise, Polly provides insights into run behavior, errors, and execution patterns to inform your scoring.

**Example questions:**

* "What went wrong in this run?"
* "Summarize what happened in this run"
* "Compare these two runs"
* "What should I consider when scoring this?"

### Evaluators

In the **Evaluators** builder, Polly helps you write and refine evaluator logic. Polly can generate evaluator code, suggest improvements, and help you test your evaluator against examples.

**Example questions:**

* "Write an evaluator that checks for hallucinations"
* "Improve the accuracy of this evaluator"
* "What does this evaluator check for?"
* "Add handling for edge cases"

## What's next

Learn more about the features that Polly helps you explore:

<CardGroup cols={2}>
  <Card title="Observability" icon="search" href="/langsmith/observability">
    Learn more about tracing and monitoring your LLM applications
  </Card>

  <Card title="Threads" icon="messages" href="/langsmith/threads">
    Understand how threads work in LangSmith
  </Card>

  <Card title="Prompt Engineering" icon="wand" href="/langsmith/prompt-engineering">
    Create and iterate on prompts in the Playground
  </Card>

  <Card title="Evaluation" icon="clipboard-check" href="/langsmith/evaluation">
    Evaluate and test your applications systematically
  </Card>
</CardGroup>

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/polly.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>