To create a code evaluator that appears in the LangSmith UI programmatically, refer to Manage evaluators with the SDK. To define a code evaluator function that you pass to
evaluate(), refer to How to define a code evaluator (SDK). To grade outputs against assertions saved on dataset examples, refer to Use assertions.Step 1. Create the evaluator
- Create an evaluator from one of the following pages in the LangSmith UI:
- In the Playground or from a dataset: Select the + Evaluator button.
- Select Add rules, configure your rule and select Apply evaluator.
- Under Create from scratch, select Code Evaluator. In the Playground, select Custom code under Code evaluator instead. The Configure Evaluator pane opens.
- Give your evaluator a clear name that describes what it measures (e.g., “Exact Match”).
Step 2. Write your evaluator code
Custom code evaluators restrictions.Allowed Libraries: You can import all standard library functions, as well as the following public packages:Network Access: You cannot access the internet from a custom code evaluator.To install other packages, make network requests, or authenticate with workspace secrets, use an advanced code evaluator. For more information, see Use advanced code evaluators.
perform_eval and should:
- Accept
runandexampleparameters. - Access data via
run['inputs'],run['outputs'], andexample['outputs']. - Return a dictionary where each key is a metric name and each value is the score for that metric. Each key represents a piece of feedback you want to return. For example,
{"correctness": 1, "silliness": 0}would create two pieces of feedback on the run.
LangSmith passes a dataset example as a second argument when you test the evaluator against a dataset sample and when the evaluator scores an experiment run. Otherwise, it passes only the run. To handle both cases in Python, define the function as
perform_eval(run, example=None). In a default code evaluator, a function that accepts only run fails with TypeError: perform_eval() takes 1 positional argument but 2 were given.Function signature
Example: Exact match evaluator
Example: Input-based evaluator
Step 3. Test and save
- Test your evaluator on example data to ensure it works as expected
- Click Save to make the evaluator available for use
Use your code evaluator
Once created, you can use your code evaluator:- When running evaluations from the Playground
- As part of a dataset to automatically run evaluations on experiments
Related
- LLM-as-a-judge evaluator (UI): Use an LLM to evaluate outputs
- Composite evaluators: Combine multiple evaluator scores
Connect these docs to your agent of choice via MCP for real-time answers.

