Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.langchain.com/llms.txt

Use this file to discover all available pages before exploring further.

Engine is in Beta and under active development. To provide feedback or use this feature, reach out to the LangChain team.
The LangSmith Engine turns your traces into a continuous improvement workflow. It surfaces recurring issues, diagnoses their root cause, and guides you through fixing them and preventing them from coming back. Each issue moves through a closed loop: a recurring failure is detected in your traces → the root cause is diagnosed → a fix is proposed → an evaluator is deployed to catch regressions → if the issue resurfaces after being closed, it is automatically reopened. For each issue, LangSmith Engine surfaces the relevant traces, proposes a fix, generates a custom evaluator to prevent regressions, and creates custom ground truth dataset examples from the production trace inputs for offline evaluation.

​
What you can do

Build: Open a pull request

Apply the proposed fix by opening a pull request in your connected repository.

Test: Add offline examples to a dataset

Generate custom ground truth dataset examples from production traces for offline evaluation.

Monitor: Create an online evaluator

Deploy a custom evaluator to catch regressions in future traces.

​
Set up the LangSmith Engine

1

Open the Issues tab

In the LangSmith console, navigate to Tracing in the UI sidebar, select a project, then click the Issues tab in the project navigation.
2

Connect a code repository (optional)

Under Connect your agent’s code repository, connect a GitHub repository. LangSmith Engine uses your source code to diagnose problems and generate fixes. Only repositories the GitHub app can access are shown. Click Manage app access → to update permissions.
3

Select priority categories (optional)

Under What matters most to you?, select categories to prioritize for your review (for example, Tool Call Failures or Latency). Click + Add something specific to describe a custom concern.
4

Start analyzing

Click Start Analyzing. Analysis takes up to 20 minutes. While you wait, you can configure provider keys and webhooks in the settings panel to be notified when issues of different priority levels are found.
5

Review the agent overview document

Before surfacing issues, LangSmith Engine generates an agent overview document describing your project’s purpose, architecture, and key metrics based on your traces. Review and edit the document, then click Accept & Continue to proceed. If the overview is inaccurate, edit it before continuing, since the LangSmith Engine uses it as context for all analysis, so accuracy here affects the quality of detected issues. You can update it at any time from the Issue settings.
Setup dialog showing the code repository field and category selections for prioritizing issue types

​
Browse and filter issues

Once setup is complete, the Issues tab displays a list of automatically detected issues in the left panel. Each entry shows a title, a short description, the number of contributing traces, and how recently the issue was observed. At the top of the list, you can click:
  • Filter issues icon to filter by Priority, Status and Tags.
  • Sort issues icon to sort by Severity, Last Updated, and Created.
  • Issue Settings gear icon to configure the LangSmith Engine.
Click any issue to display its details in the right panel. If no issues appear after setup completes, the LangSmith Engine found no recurring patterns in the analyzed traces. Try checking back after more traces have been collected.

​
Review an issue

Click any issue in the list to open its detail panel. At the top, a diagnosis describes the problem and its impact. The Linked traces section lists the traces that support the diagnosis. Click any trace to open its detail panel. For more information, see Manage a trace. Click Add offline examples at the bottom right of this section to generate custom ground truth dataset examples from the production trace inputs for offline evaluation. The Proposed Fix section describes the issue and suggests how to address it, which may include specific code or prompt changes if a repository is connected. The Suggested Evaluator section provides a ready-to-use evaluator you can deploy to catch the issue in future traces. If the evaluator fires after you close an issue, the issue is automatically reopened to indicate the problem persists. The Offline Examples section proposes dataset examples generated from the production trace inputs that triggered the issue, for use in offline evaluation.

​
Take action on an issue

​
Change priority

Select Low, Medium or High from the priority dropdown to update an issue’s priority. You can optionally provide a reason, which feeds back into the LangSmith Engine to help improve its analysis over time.

​
Create an evaluator

  1. Click Create Evaluator to deploy the suggested evaluator for the issue.
  2. Configure the name, run filters, and sampling rate. Edit the code directly in the built-in editor if needed.
  3. Enable Apply to past runs to see how many historical traces the evaluator would have flagged before deploying.
For more information, see Evaluators.

​
Add offline examples

  1. Click Add offline examples at the bottom of the Linked traces list to open the Add as offline example dialog.
  2. Review each trace. The dialog shows the input, the wrong output the agent produced, and the proposed expected output as a custom ground truth example.
  3. Click Add to Dataset to add them directly, or click Edit in annotation queue to review them first.
  4. In the annotation queue, each example shows the run inputs alongside reference outputs proposed by the LangSmith Engine, structured as named assertions generated from trace analysis. Edit the assertions as needed, add new ones with + Add assertion, then click Add to Dataset & Continue to work through each example.
For more information, see Manage datasets and Use annotation queues.

​
Copy the issue prompt

Click the Copy Fix Context copy icon to save a prompt with the issue details to your clipboard. You can then use it with an LLM or coding assistant to help resolve the issue.

​
Open a pull request

Click Open PR to open a GitHub pull request in your connected repository with the proposed fix applied. Once a pull request is open, the button changes to View PR. LangSmith Engine can propose code changes to any connected repository, including agents built with Deep Agents, LangChain, and LangGraph.

​
Resolve or ignore an issue

Click Resolve to mark an issue as fixed, or Ignore to dismiss it as not real or not worth fixing. You can optionally provide a reason for either action.

​
Reopen an issue

To reopen a previously closed issue, open the issue detail view and click Reopen.

​
Configure the LangSmith Engine

Click the Issue Settings gear icon on the Issues tab to open the Edit Issues panel. From here you can configure:
  • Agent Overview: Edit your agent overview document to keep LangSmith Engine’s understanding of your project accurate as your application evolves.
  • Scan schedule: The LangSmith Engine scans your traces every 6 hours by default. Click Edit to change the frequency or Pause to suspend scanning or Resume to resume scanning.
  • Priorities: Areas the LangSmith Engine should pay extra attention to when scanning traces. Changes take effect on the next scan.
  • Current month spend: Total cost of LangSmith Engine runs for this project in the current calendar month.
  • Code repository: Update the connected GitHub repository or subfolder.
  • Provider & API keys: Pick a provider and supply its API key. The LangSmith Engine will use matching heavy and light models for the selected provider. Keys are stored as workspace secrets and shared with other features.
  • Webhooks: Configure webhooks to be notified when new issues are found at different priority levels.
  • Delete all issues: This action cannot be undone. All issues and settings will be permanently removed.
Connect these docs to Claude, VSCode, and more via MCP for real-time answers.
Edit this page on GitHub or file an issue.