Find and fix your agent's issues with LangSmith Engine

​

How Engine works

​

The issue lifecycle

1. Detects a recurring issue in your traces.
2. Diagnoses the root cause against your traces and connected source code.
3. Proposes a fix as a pull request.
4. Tracks the issue over time, automatically adding new traces that match the same pattern, and generates ground truth dataset examples so you can verify a fix.
5. Reopens the issue automatically if it resurfaces after being closed.

​

How Engine selects traces

• Reads the feedback keys present in your project and performs a dedicated pull of low-scoring traces for each key, so the sample includes traces with poor evaluator scores rather than leaving them to recency.
• Prioritizes traces with non-empty feedback scores ahead of other traces when screening the sample.
• Preserves feedback scores on every trace in the analysis context, even when trace payloads are compacted to fit within context limits.

​

Set up Engine

​

Enable Engine for your organization

​

Turn on Engine for a tracing project

​

Pause Engine or delete its issues

​

Configure Engine

• Code repository and Context repository: Connect or update the GitHub repository Engine reads when diagnosing issues, optionally with a Subfolder and a Branch (defaults to the repository default). A Context Hub repository lets Engine propose fixes to instructions, docs, and linked skills. See Connect Engine to GitHub.
• Notifications: See Engine notifications.
• Preview deployments: Set the baseline deployment Engine replays issue traces against, and turn on fix verification with preview deployments. See Validate fixes by running your agent.
• Analysis level: See Set the analysis level.
• Engine spend: See Set spend limits and monitor usage.
• Pause and Delete all issues: See Pause Engine or delete its issues.

​

Give context on your agent

​

Tell Engine what kinds of issues to focus on

​

Tell Engine which traces to focus on

• Engine setup: In the Find and fix your agent’s issues panel, under Focus on specific traces.
• Engine Settings: In the Focus on specific traces section of the Engine Settings panel. Edits here save automatically.

• Run Name: Pick a run or agent name. The value field autocompletes from the run names in your project’s recent traces.
• Metadata: Pick a metadata key, then a value. Both autocomplete from the metadata present on your project’s recent runs.

​

Connect to Linear

​

Manage Engine costs

​

Understand LSU costs

​

Set the analysis level

• Reduced: Monitors fewer traces at a lower cost.
• Standard (default): Analyzes more of your eligible traces for fuller coverage.
• Expanded: Maximum coverage for high-volume projects. Available only for projects with enough tracing volume.

​

Set spend limits and monitor usage

• Org-wide limit: Open Settings, select Engine enablement under Engine, then enter a value under Monthly LSU spend limit.
• Per-project limit: Open the Engine tab in a tracing project, click the Engine Settings icon, and set a limit under Monthly LSU spend limit.

• Org-wide limit: Choose Default, No limit, or a custom cap. Until an admin chooses, the default applies (750 LSU per month, $750), so Engine spend is capped even though no one has set a limit. The Engine enablement page names the enforced limit and its source.
• Per-project limit: Leave the field blank for no limit. Use Remove limit to clear a cap you set earlier.

​

Investigate and fix an issue

​

Review evidence

• View a trace: Click View trace to open an evidence trace. When Engine identifies the child run that caused the issue, it opens that exact run. The trace view includes a View issue link back to the Engine issue.
• Create offline examples: Click Add offline examples to generate custom ground truth dataset examples from the production trace inputs for offline evaluation.
• View project evidence: Click View all in project to view the evidence in the tracing project.

​

Change priority and status

• Close to mark the issue as resolved.
• Incorrectly Flagged to dismiss the issue as not real or not worth fixing.

​

Open a pull request

​

Create a Linear issue

​

Add offline examples

1. Click Add offline examples at the top right of the Evidence list to open the Add as offline example dialog.
2. Review each trace. The dialog shows the input, the wrong output the agent produced, and the proposed expected output as a custom ground truth example.
3. Click Add to Dataset to add them directly, or click Edit in annotation queue to review them first.
4. In the annotation queue, each example shows the run inputs alongside reference outputs proposed by Engine, structured as named assertions generated from trace analysis. Each assertion is a short claim describing what a correct answer should or shouldn’t include. Edit the assertions as needed, add new ones with + Add assertion, then click Add to Dataset & Continue to work through each example.

​

Watch an issue

​

Filter and sort issues

​

In the UI

• Filter issues icon to filter by Priority, Status and Tags.
• Sort issues icon to sort by Severity, Last Updated, and Created.
• Configure Engine (gear) icon to configure Engine.

​

With the CLI

# List open, high-priority issues for a project
langsmith project issues list --project <project-name> --status open --priority high

​

Beta: Validate fixes by running your agent

​

Set up validation

1. On the Engine page, click Configure Engine.
2. (Optional) Connect the GitHub repository Engine should modify. In the repository settings, select the base branch for Engine’s fixes, or leave it blank to use the repository’s default branch.
3. In the Preview deployments section, find the Baseline deployment field. Search for a deployment, or paste its LangSmith URL or deployment ID. Preview deployments and deployments that are not ready cannot be selected from the list. The baseline can be a production or staging deployment, but use staging when possible so validation does not exercise production credentials and services.
4. Select Verify fixes with preview deployments.
5. Click Save.

• A connected repository: Engine opens its fix as a pull request in the repository connected in Connect Engine to GitHub.
• Label-triggered preview builds: Enable preview builds on the baseline deployment. Use the following starting configuration:
  • Set Preview base branch to the same branch configured for Engine.
  • Select Label only and set Trigger label to preview. Engine applies this label to its fix pull request.
  • Set Idle TTL to 6 hours.
  • Set Max concurrent previews to 20.

​

Authenticate with your deployment

1. In the Preview deployments section, under Deployment authentication, click Add custom headers.
2. Enter every header your deployment requires, then click Save.
3. In your deployment’s authentication handler, map those headers to an identity with the least access validation needs.

​

Prepare a deployment to test against

​

Use a deployment you can safely exercise

​

Make replayed runs side-effect free

from __future__ import annotations

from collections.abc import Awaitable, Callable, Collection, Mapping
from typing import Any, cast

from langchain.agents.middleware import AgentMiddleware
from langchain.messages import ToolMessage
from langchain.tools.tool_node import ToolCallRequest
from langchain_core.runnables import RunnableConfig
from langgraph.types import Command

ENGINE_VALIDATION_CONFIG_KEY = "__engine_validation_replay__"
LANGGRAPH_AUTH_USER_ID_CONFIG_KEY = "langgraph_auth_user_id"


def _configurable(config: RunnableConfig | None) -> Mapping[str, object]:
    configurable = (config or {}).get("configurable")
    return configurable if isinstance(configurable, Mapping) else {}


def is_engine_validation(config: RunnableConfig | None) -> bool:
    """Return whether the caller requested reduced-privilege validation mode."""
    return _configurable(config).get(ENGINE_VALIDATION_CONFIG_KEY) is True


def is_trusted_engine_validation(
    config: RunnableConfig | None,
    *,
    expected_user_id: str,
) -> bool:
    """Verify the validation marker and authenticated Engine identity."""
    configurable = _configurable(config)
    return (
        bool(expected_user_id)
        and configurable.get(ENGINE_VALIDATION_CONFIG_KEY) is True
        and configurable.get(LANGGRAPH_AUTH_USER_ID_CONFIG_KEY) == expected_user_id
    )


def _request_config(request: ToolCallRequest) -> RunnableConfig | None:
    runtime = getattr(request, "runtime", None)
    config = getattr(runtime, "config", None)
    return cast(RunnableConfig, config) if isinstance(config, dict) else None


class EngineValidationSafetyMiddleware(AgentMiddleware):
    """Allow only explicitly safe tools during Engine validation."""

    def __init__(self, *, safe_tools: Collection[str]) -> None:
        self._safe_tools = frozenset(safe_tools)

    def _rejection(self, request: ToolCallRequest) -> ToolMessage | None:
        if not is_engine_validation(_request_config(request)):
            return None
        call = request.tool_call
        name = call.get("name") or "tool"
        if name in self._safe_tools:
            return None
        return ToolMessage(
            content=f"{name} is blocked during Engine validation.",
            tool_call_id=call["id"],
            name=name,
            status="error",
        )

    def wrap_tool_call(
        self,
        request: ToolCallRequest,
        handler: Callable[[ToolCallRequest], ToolMessage | Command[Any]],
    ) -> ToolMessage | Command[Any]:
        rejection = self._rejection(request)
        return rejection if rejection is not None else handler(request)

    async def awrap_tool_call(
        self,
        request: ToolCallRequest,
        handler: Callable[
            [ToolCallRequest],
            Awaitable[ToolMessage | Command[Any]],
        ],
    ) -> ToolMessage | Command[Any]:
        rejection = self._rejection(request)
        return rejection if rejection is not None else await handler(request)

from langchain.agents import create_agent

from engine_validation import EngineValidationSafetyMiddleware

agent = create_agent(
    model=model,
    tools=[search_catalog, lookup_order, send_message],
    middleware=[
        EngineValidationSafetyMiddleware(
            safe_tools={"search_catalog", "lookup_order"},
        )
    ],
)

Verify trusted Engine context

if is_trusted_engine_validation(
    config,
    expected_user_id=settings.engine_user_id,
):
    context = load_context_from_checkpoint()

​

Expect a burst of traffic

​

Keep the deployment available

​

Test an issue

1. Selects up to five distinct traces linked to the issue, in the order they were linked.
2. Replays each trace’s input against the baseline deployment. For a trace from a multi-turn thread, Engine includes the earlier turns of the conversation, so a follow-up failure is judged in the same context.
3. Judges whether each replay shows the issue’s reported behavior again. Each trace is Reproduced, Not reproduced, or Inconclusive.

​

Verify a fix

1. When a fix run completes, Engine opens a pull request for the fix if one is not already open, then adds the preview build label to it.
2. LangSmith builds a preview deployment from the pull request.
3. When the preview revision is live, Engine replays the traces from the issue’s validation against the preview deployment.
4. If the issue still occurs on the preview, Engine revises the fix on the same pull request and verifies it again, for up to three attempts in total.

​

Read a validation result

​

Review validation experiments

• Dataset: One per issue, described as Engine validation evidence, with one example per replayed trace.
• Baseline experiment: The replays against the baseline deployment.
• Fix experiment: The replays against a fix’s preview deployment, one experiment per verification attempt.

• Flagged traces: What Engine saw in the original trace. Click Original trace to open it.
• Reproduced on prod: The baseline result, Reproduced, Not reproduced, or Inconclusive. Click Repro trace to open the replay.
• After PR #<number>: The result on the fix’s preview, Fix verified, Recurred, or Inconclusive. Click the trace link to open the replay.

​

Troubleshoot validation

• The baseline deployment: Confirm it is still ready, is not a preview, and is in the same workspace as the tracing project.
• Authentication: If your deployment authenticates callers itself, confirm the saved headers are current. Replacing a credential requires re-entering every header value.
• The replays themselves: Open the replayed runs from the issue’s Evidence section or validation summary and read what your agent returned.
• Deployment limits: Look for rate limiting, quota exhaustion, or timeouts caused by the concurrent replays, and for very large responses, which Engine may not be able to judge.
• Restricted tools: If your agent blocks tools during a replay, confirm the stubbed tools still return a usable result rather than an error that ends the run.

​

Beta: Proactively detect issues with Red Teaming

​

How Red Teaming works

1. Maps the application. Engine reviews up to 25 recent traces from the tracing project and, if one is connected, your code repository. From them, it learns the agent’s prompts, tools, guardrails, and request format. If Engine has already scanned the project, the run starts from the project’s agent overview and runs a shorter confirmation scan.
2. Forms hypotheses. Each hypothesis describes one specific way the agent could fail. Engine assigns it an issue class and a scenario type, which records whether an ordinary user could trigger the failure or whether it takes a deliberate attack.
3. Probes the deployment. Engine sends up to two synthetic requests per hypothesis to the deployment. A probe reuses the request format of a real trace, but replaces its content with new, synthetic input.
4. Judges the results. Engine reviews each probe’s response and trace, then assigns the hypothesis a status and a severity. For an ambiguous result, Engine can send one follow-up probe.

​

Red Teaming prerequisites

• Engine on the tracing project: Set up Engine for the project that receives your agent’s traces.
• A baseline deployment: A ready, non-preview LangSmith Cloud deployment in the same workspace, set as the project’s baseline. See Set the baseline deployment.
• Permissions: runs:read on the tracing project to view red-team reports. To start a run, you also need runs:create on the project and deployments:update on the baseline deployment.
• A connected repository (recommended): With a connected GitHub repository, Engine reads your agent’s prompts, tools, and guardrails from source. Without one, findings rely on traces alone, and the report shows Repository context was not available.

​

Set the baseline deployment

1. On the Engine page, select the tracing project, then click Configure Engine.
2. In the Preview deployments section, find the Baseline deployment field.
3. Search for a deployment, or paste its LangSmith URL or deployment ID. Preview deployments and deployments that are not ready cannot be selected from the list.
4. Click Save.

​

Start a Red Teaming run

1. In the LangSmith UI, select Engine in the sidebar, then select the tracing project.
2. At the top of the left panel, select Red Teaming.
3. Click Run red team.

​

Read a red-team report

​

Review the overview

• Confirmed findings: The number of confirmed findings, broken down by severity.
• Hypotheses tested: The number of hypotheses the run tested.
• Issue classes tested: How many of the 10 issue classes the run tested.
• Findings by Issue Class: Confirmed findings per issue class.
• Hypothesis Outcomes: Hypotheses by status. Click a status to filter the hypothesis list to it.
• Findings Over Runs: Confirmed findings for each run, stacked by severity. Click a run to open its report.

​

Browse hypotheses

• Product-Facing Findings: Confirmed findings that an ordinary user could trigger, with a normal request or a realistic, messy one.
• Technical Hardening Observations: Confirmed findings that take a deliberate attack or a technical boundary check to trigger.
• Legacy Confirmed Findings: Confirmed findings from older reports, recorded before Engine assigned scenario types.
• Other Tested Hypotheses: Every hypothesis that was not confirmed.

​

Inspect a hypothesis

• The synthetic prompt: The exact prompt Engine sent to the deployment.
• Observed: An excerpt quoted from your agent’s response, with a summary of the probe’s trace, such as its tool calls, tool errors, and token count.
• Expected: What a correct application would have returned.
• View proof trace: Opens the probe’s trace in the side panel.

​

Red Teaming issue classes

• Auth Isolation: Red Teaming does not confirm findings in this class. Its hypotheses report as Needs Review or Not Tested.
• Session Integrity: A hypothesis needs a two-turn probe in a single conversation. When no such probe runs, the hypothesis reports as Needs Review or Not Tested.

​

Run Red Teaming again

​

Red Teaming limits

​

How Red Teaming handles your data

• Synthetic probes: Probe content is synthetic. Engine does not copy customer trace content or repository text into probes; a trace supplies only the request format.
• Read-only access: Red Teaming reads your traces with a read-only LangSmith credential and clones your repository with a read-only GitHub token.
• Screened reports: Engine screens model-written report text for secrets before storing it, and bounds the length of response excerpts.

​

See also

• Engine: Product overview and where Engine fits in the development lifecycle.
• Connect Engine to GitHub: Connect repositories in LangSmith Cloud, or create and configure your own GitHub App for a self-hosted deployment.
• Engine notifications: Slack and webhook destinations, event payload reference, and signing-secret verification.
• Engine security: Review how Engine accesses your traces, deployments, and repositories.
• Preview builds: Create the preview deployments Engine verifies fixes against.
• Compare experiment results: Compare the baseline and fix experiments side by side.
• Engine on self-hosted: Self-hosted architecture and data handling.
• Manage datasets, Use annotation queues, and Use assertions: Work with the offline examples Engine generates.
• LangSmith CLI: List and manage issues programmatically.