Manage evaluators

​

View evaluators

​

Create an evaluator

​

Create an evaluator in the UI

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Click + Evaluator to open the new evaluator panel.
3. The panel lets you:
   • Create from scratch: Build a new LLM-as-a-Judge, Decision Model, or Code evaluator.
   • Add a LangChain Tuned Evaluator: Attach a specialized judge managed by LangChain to a compatible tracing project without configuring a prompt, model, or API key.
   • Create from a template: Start from a ready-made evaluator (also known as a prebuilt evaluator) for common evaluation patterns. A Recommended section lists popular templates first. The rest are grouped by category: Security, Safety, Quality, Conversation, Trajectory, Image Evaluations, and Voice Evaluation. For what each template scores, see Prebuilt evaluator templates.

​

Create an evaluator with the SDK

import asyncio

from langsmith import Client


async def main():
    client = Client()

    created = await client.evaluators.create(
        name="Correctness evaluator",
        type="code",
        code_evaluator={
            "code": "def perform_eval(run, example):\n    return {'score': 1}",
            "language": "python",
        },
    )
    print("Created evaluator:", created.evaluator.id)


asyncio.run(main())

import { Client } from "langsmith";

const client = new Client();

const created = await client.evaluators.create({
  name: "Correctness evaluator",
  type: "code",
  code_evaluator: {
    code: "def perform_eval(run, example):\n    return {'score': 1}",
    language: "python",
  },
});
console.log("Created evaluator:", created.evaluator?.id);

​

Prebuilt evaluator templates

• Whole threads: Conversation and Trajectory templates read every message in a thread. When you add one to a tracing project, LangSmith configures it as a thread-level evaluator. For more information, see Set up multi-turn online evaluators.
• Reference outputs: Correctness, Assertions, and Exact Match compare the output against a reference output, which comes from a dataset example.
• Plan: Plan Adherence reads a plan variable in addition to the thread messages.
• Attachments: Image and Voice templates read the run’s attachments. For more information, see Evaluate with attachments.

​

Security

​

Safety

​

Quality

​

Conversation

​

Trajectory

​

Image Evaluations

​

Voice Evaluation

​

View evaluator details

• Overview: The evaluator’s feedback configuration and prompt or code definition.
• Traces: Traces processed by this evaluator across all attached resources.
• Logs: Execution logs for this evaluator across all attached resources.
• Projects & Datasets (Environments & Datasets in an agent-based workspace): The tracing projects, agent environments, and datasets this evaluator is attached to, with each attachment’s weekly spend and limit.

​

Edit an evaluator

​

Manage evaluator trace retention

1. When you create or edit an online evaluator, set the source to a tracing project or an agent environment, rather than a dataset.
2. Expand the Advanced section in the evaluator configuration panel.
3. Clear Extend trace retention.

​

Include extended stats

1. When you create or edit a run-level evaluator, expand the Advanced section in the evaluator configuration panel.
2. Select Include extended stats (feedback, costs, tokens).

​

Access extended stats

​

Chain evaluators

1. Create or open the second evaluator and expand its filter. For more information, see Apply a filter to runs that trigger the evaluator for LLM-as-a-judge evaluators or Create a filter for code evaluators.
2. Filter on the first evaluator’s feedback key. The filter matches on the key rather than the evaluator that produced it, so feedback from any source with that key triggers the second evaluator, including user feedback. Use a key that only the first evaluator produces.
   • To trigger on every run the first evaluator scored, filter on the key alone:
     eq(feedback_key, "correctness")
     
   • To trigger only when the score meets a condition, combine the key with a feedback_score comparison. For example, to trigger only when correctness is below 0.5:
     and(eq(feedback_key, "correctness"), lt(feedback_score, 0.5))
     
3. If the second evaluator reads the first score, include extended stats. LLM-as-a-judge evaluators require it, and code evaluators use it to read the score reliably. For more information, see Read the first score.
4. Save the evaluator.

​

Read the first score

• LLM-as-a-judge evaluators: Include extended stats, then map a prompt variable to run.feedback_stats.<feedback_key>.avg. For example, map {{correctness_average}} to run.feedback_stats.correctness.avg.
• Code evaluators: Read the feedback record in run["feedback"] whose key is the first evaluator’s feedback key, and use its score. run["feedback"] holds at most 100 feedback records across each batch of runs the evaluator processes, and the oldest records are dropped first, so the score can be missing when the batch carries many feedback records. To read the score reliably, include extended stats and read the average from run["feedback_stats"]["<feedback_key>"]["avg"].

​

Delete an evaluator

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Select the evaluator you want to delete.
3. Open the Projects & Datasets tab (Environments & Datasets in an agent-based workspace). For each attachment listed there, select Detach in the Actions menu at the right of the row.
4. Return to the Evaluators page and click Delete at the top of the page.