Skip to main content
The Engine page lists the issues LangSmith Engine detects in a tracing project, each with a diagnosis, the traces that support it, and a proposed fix. Work through them to ship fixes and to turn real failures into evaluation datasets. If Engine is not running on your project yet, see Set up Engine. Click any issue to open its detail panel. At the top, a diagnosis describes the problem and its impact. Each issue has a toolbar for setting its priority, closing or reopening it, opening a pull request, creating a Linear issue, and watching it. Engine learns from what you do with each issue: closing it, marking it as incorrectly flagged, changing its priority, or opening a pull request, along with any reason you give. For how that feedback shapes later scans, see How Engine learns from your feedback.

​
Review evidence

The Evidence section contains traces that support a diagnosis, including a snippet from each trace. From this section, you can:
  • View a trace: Click View trace to open an evidence trace. When Engine identifies the child run that caused the issue, it opens that exact run. The trace view includes a View issue link back to the Engine issue.
  • Create offline examples: Click Add offline examples to generate custom ground truth dataset examples from the production trace inputs for offline evaluation.
  • View project evidence: Click View all in project to view the evidence in the tracing project.
For more information, see Manage a trace. Engine keeps tracking an issue after it is filed. On later scans, any new trace that matches the issue’s failure pattern is added to Evidence automatically, so the issue reflects how often the failure is still happening without you re-running anything. The Proposed Fix section describes the issue and suggests how to address it, which may include specific code or prompt changes if a repository is connected.

​
Change priority and status

Select Low, Medium, or High from the priority dropdown to update an issue’s priority. You can optionally provide a reason, which Engine learns from on later scans. Closing records the outcome of your review. Click:
  • Close to mark the issue as resolved.
  • Incorrectly Flagged to dismiss the issue as not real or not worth fixing.
For either outcome, you can optionally provide a reason, which Engine learns from on later scans. You can reopen a closed issue at any time. Click Reopen to clear any fix in progress and stop watching the issue if it was being watched. Engine also reopens an issue automatically when it detects the same problem recurring in a later trace.

​
Open a pull request

Click Open PR to open a GitHub pull request with the proposed code change in your connected repository. Connect a repository first if you have not. Once a pull request exists, Engine replaces Open PR with View PR #<number>. Click View PR #<number> to open the pull request in GitHub. Engine reflects the PR’s status (open, merged, or closed) throughout the issue. You can also copy the issue’s fix context to your clipboard for use with an LLM or coding assistant. Engine can propose code changes to any connected repository, including agents built with Deep Agents, LangChain, and LangGraph.

​
Create a Linear issue

To create an issue, first connect Linear, then click Create in Linear on an Engine issue. Engine shows Linear creation pending while it creates the issue. When creation completes, Engine displays the linked Linear issue identifier in the Engine issue and issue list. The Linear issue includes the Engine issue title and description, severity, category, tags, a View issue in LangSmith link, and evidence trace IDs. Engine retains the link to the Linear issue. If you close the Linear ticket, Engine closes the corresponding issue. If you cancel the Linear ticket, Engine marks the corresponding issue as incorrectly flagged. Engine also tracks a pull request linked to the Linear ticket on the corresponding Engine issue.

​
Add offline examples

This step captures the traces that surfaced the issue as ground-truth dataset examples, so you can evaluate the fix offline before it reaches production.
  1. Click Add offline examples at the top right of the Evidence list to open the Add as offline example dialog.
  2. Review each trace. The dialog shows the input, the wrong output the agent produced, and the proposed expected output as a custom ground truth example.
  3. Click Add to Dataset to add them directly, or click Edit in annotation queue to review them first.
  4. In the annotation queue, each example shows the run inputs alongside reference outputs proposed by Engine, structured as named assertions generated from trace analysis. Each assertion is a short claim describing what a correct answer should or shouldn’t include. Edit the assertions as needed, add new ones with + Add assertion, then click Add to Dataset & Continue to work through each example.
For more information, refer to Manage datasets, Use annotation queues, and Use assertions.

​
Watch an issue

Watching keeps an issue open for monitoring without resolving it or marking it as incorrectly flagged. Click Watch when you are not ready to fix an issue but still want to know if it keeps happening. To be alerted when a watched issue recurs, click Alert me via Slack, which opens the Notifications section of the Engine settings panel. See Engine notifications. When new traces link to a watched issue, Engine moves it to the top of your list and shows how many new traces arrived, so you can pick up the fix or keep watching.
Watching is only available for open issues without a pull request in flight: discard the fix to watch an issue again. Resolving a watched issue, or marking it as incorrectly flagged, automatically stops watching it.

​
Filter and sort issues

​
In the UI

The Engine page lists detected issues in the left panel. Each entry shows a title, a short description, the number of contributing traces, and how recently the issue was observed. Each issue is tagged with a failure category, such as Silent tool error or Hallucination. At the top of the list, you can click:
  • Filter issues icon to filter by Priority, Status and Tags.
  • Sort issues icon to sort by Severity, Last Updated, and Created.
  • Configure Engine (gear) icon to configure Engine.
Open LangSmith Chat to ask questions across your issues, for example, which issues need the most attention or how many new issues are open. If no issues appear after setup completes, Engine found no recurring patterns in the analyzed traces. Try checking back after more traces have been collected.

​
With the CLI

Use langsmith project issues list in the LangSmith CLI to list a project’s issues. Filter with --status (open, fixing, watching, completed, or ignored) and --priority (urgent, high, medium, or low), and page with --limit and --offset.

​
See also

Connect these docs to your agent of choice via MCP for real-time answers.
Edit this page on GitHub or file an issue.