Validate fixes automatically

• To validate an issue, Engine replays its linked traces against a configured baseline deployment to confirm that the issue reproduces.
• To verify a fix, Engine replays the same traces against a preview deployment built from Engine’s proposed fix.

​

Set the baseline deployment

1. On the Engine page, select the tracing project, then click Configure Engine.
2. In the Deployment section, find the Baseline deployment field.
3. Search for a deployment, or paste its LangSmith URL or deployment ID. Preview deployments and deployments that are not ready cannot be selected from the list. The baseline can be a production or staging deployment, but use staging when possible so validation does not exercise production credentials and services.
4. Click Save.

​

Authenticate with your deployment

1. In the Deployment section, under Deployment authentication, click Add custom headers.
2. Enter every header your deployment requires, then click Save.
3. In your deployment’s authentication handler, map those headers to an identity with the least access validation needs.

​

Set up fix verification

1. Set the baseline deployment.
2. Connect the GitHub repository Engine should modify. Engine opens its fix as a pull request there. In the repository settings, select the base branch for Engine’s fixes, or leave it blank to use the repository’s default branch.
3. Enable preview builds on the baseline deployment, triggered by label. Use the following starting configuration:
   • Set Preview base branch to the same branch configured for Engine.
   • Select Label only and set Trigger label to preview. Engine applies this label to its fix pull request.
   • Set Idle TTL to 6 hours.
   • Set Max concurrent previews to 20.
4. In the Deployment section of Configure Engine, select Verify fixes with preview deployments.
5. Click Save.

​

Prepare a deployment to test against

​

Use a deployment you can safely exercise

​

Make replayed runs side-effect free

from __future__ import annotations

from collections.abc import Awaitable, Callable, Collection, Mapping
from typing import Any, cast

from langchain.agents.middleware import AgentMiddleware
from langchain.messages import ToolMessage
from langchain.tools.tool_node import ToolCallRequest
from langchain_core.runnables import RunnableConfig
from langgraph.types import Command

ENGINE_VALIDATION_CONFIG_KEY = "__engine_validation_replay__"
LANGGRAPH_AUTH_USER_ID_CONFIG_KEY = "langgraph_auth_user_id"


def _configurable(config: RunnableConfig | None) -> Mapping[str, object]:
    configurable = (config or {}).get("configurable")
    return configurable if isinstance(configurable, Mapping) else {}


def is_engine_validation(config: RunnableConfig | None) -> bool:
    """Return whether the caller requested reduced-privilege validation mode."""
    return _configurable(config).get(ENGINE_VALIDATION_CONFIG_KEY) is True


def is_trusted_engine_validation(
    config: RunnableConfig | None,
    *,
    expected_user_id: str,
) -> bool:
    """Verify the validation marker and authenticated Engine identity."""
    configurable = _configurable(config)
    return (
        bool(expected_user_id)
        and configurable.get(ENGINE_VALIDATION_CONFIG_KEY) is True
        and configurable.get(LANGGRAPH_AUTH_USER_ID_CONFIG_KEY) == expected_user_id
    )


def _request_config(request: ToolCallRequest) -> RunnableConfig | None:
    runtime = getattr(request, "runtime", None)
    config = getattr(runtime, "config", None)
    return cast(RunnableConfig, config) if isinstance(config, dict) else None


class EngineValidationSafetyMiddleware(AgentMiddleware):
    """Allow only explicitly safe tools during Engine validation."""

    def __init__(self, *, safe_tools: Collection[str]) -> None:
        self._safe_tools = frozenset(safe_tools)

    def _rejection(self, request: ToolCallRequest) -> ToolMessage | None:
        if not is_engine_validation(_request_config(request)):
            return None
        call = request.tool_call
        name = call.get("name") or "tool"
        if name in self._safe_tools:
            return None
        return ToolMessage(
            content=f"{name} is blocked during Engine validation.",
            tool_call_id=call["id"],
            name=name,
            status="error",
        )

    def wrap_tool_call(
        self,
        request: ToolCallRequest,
        handler: Callable[[ToolCallRequest], ToolMessage | Command[Any]],
    ) -> ToolMessage | Command[Any]:
        rejection = self._rejection(request)
        return rejection if rejection is not None else handler(request)

    async def awrap_tool_call(
        self,
        request: ToolCallRequest,
        handler: Callable[
            [ToolCallRequest],
            Awaitable[ToolMessage | Command[Any]],
        ],
    ) -> ToolMessage | Command[Any]:
        rejection = self._rejection(request)
        return rejection if rejection is not None else await handler(request)

from langchain.agents import create_agent

from engine_validation import EngineValidationSafetyMiddleware

agent = create_agent(
    model=model,
    tools=[search_catalog, lookup_order, send_message],
    middleware=[
        EngineValidationSafetyMiddleware(
            safe_tools={"search_catalog", "lookup_order"},
        )
    ],
)

​

Verify trusted Engine context

if is_trusted_engine_validation(
    config,
    expected_user_id=settings.engine_user_id,
):
    context = load_context_from_checkpoint()

​

Expect a burst of traffic

​

Keep the deployment available

​

Test an issue

1. Selects up to five distinct traces linked to the issue, in the order they were linked.
2. Replays each trace’s input against the baseline deployment. For a trace from a multi-turn thread, Engine includes the earlier turns of the conversation, so a follow-up failure is judged in the same context.
3. Judges whether each replay shows the issue’s reported behavior again. Each trace is Reproduced, Not reproduced, or Inconclusive.

​

Verify a fix

1. When a fix run completes, Engine opens a pull request for the fix if one is not already open, then adds the preview build label to it.
2. LangSmith builds a preview deployment from the pull request.
3. When the preview revision is live, Engine replays the traces from the issue’s validation against the preview deployment.
4. If the issue still occurs on the preview, Engine revises the fix on the same pull request and verifies it again, for up to three attempts in total.

​

Read a validation result

​

Review validation experiments

• Dataset: One per issue, described as Engine validation evidence, with one example per replayed trace.
• Baseline experiment: The replays against the baseline deployment.
• Fix experiment: The replays against a fix’s preview deployment, one experiment per verification attempt.

• Flagged traces: What Engine saw in the original trace. Click Original trace to open it.
• Reproduced on prod: The baseline result, Reproduced, Not reproduced, or Inconclusive. Click Repro trace to open the replay.
• After PR #<number>: The result on the fix’s preview, Fix verified, Recurred, or Inconclusive. Click the trace link to open the replay.

​

Troubleshoot validation

• The baseline deployment: Confirm it is still ready, is not a preview, and is in the same workspace as the tracing project.
• Authentication: If your deployment authenticates callers itself, confirm the saved headers are current. Replacing a credential requires re-entering every header value.
• The replays themselves: Open the replayed runs from the issue’s Evidence section or validation summary and read what your agent returned.
• Deployment limits: Look for rate limiting, quota exhaustion, or timeouts caused by the concurrent replays, and for very large responses, which Engine may not be able to judge.
• Restricted tools: If your agent blocks tools during a replay, confirm the stubbed tools still return a usable result rather than an error that ends the run.

​

See also

• Find and fix issues: Review the issues Engine validates.
• Red Teaming: Probe the same baseline deployment with synthetic requests.
• Preview builds: Create the preview deployments Engine verifies fixes against.
• Compare experiment results: Compare the baseline and fix experiments side by side.
• Engine security: Review how Engine accesses your traces, deployments, and repositories.