Skip to main content
Code evaluators in the LangSmith UI allow you to write custom evaluation logic using Python or TypeScript code directly in the interface. Unlike LLM-as-a-judge evaluators that use a model to evaluate outputs, code evaluators use deterministic logic you define.
To create a code evaluator that appears in the LangSmith UI programmatically, refer to Manage evaluators with the SDK. To define a code evaluator function that you pass to evaluate(), refer to How to define a code evaluator (SDK). To grade outputs against assertions saved on dataset examples, refer to Use assertions.

Step 1. Create the evaluator

  1. Create an evaluator from one of the following pages in the LangSmith UI:
    • In the Playground or from a dataset: Select the + Evaluator button.
    • Select Add rules, configure your rule and select Apply evaluator.
  2. Under Create from scratch, select Code Evaluator. In the Playground, select Custom code under Code evaluator instead. The Configure Evaluator pane opens.
  3. Give your evaluator a clear name that describes what it measures (e.g., “Exact Match”).

Step 2. Write your evaluator code

Custom code evaluators restrictions.Allowed Libraries: You can import all standard library functions, as well as the following public packages:
Network Access: You cannot access the internet from a custom code evaluator.To install other packages, make network requests, or authenticate with workspace secrets, use an advanced code evaluator. For more information, see Use advanced code evaluators.
In the Configure Evaluator pane, define your evaluation logic using Python or TypeScript. Your evaluator function must be named perform_eval and should:
  1. Accept run and example parameters.
  2. Access data via run['inputs'], run['outputs'], and example['outputs'].
  3. Return a dictionary where each key is a metric name and each value is the score for that metric. Each key represents a piece of feedback you want to return. For example, {"correctness": 1, "silliness": 0} would create two pieces of feedback on the run.
LangSmith passes a dataset example as a second argument when you test the evaluator against a dataset sample and when the evaluator scores an experiment run. Otherwise, it passes only the run. To handle both cases in Python, define the function as perform_eval(run, example=None). In a default code evaluator, a function that accepts only run fails with TypeError: perform_eval() takes 1 positional argument but 2 were given.

Function signature

Example: Exact match evaluator

Example: Input-based evaluator

Step 3. Test and save

  1. Test your evaluator on example data to ensure it works as expected
  2. Click Save to make the evaluator available for use

Use your code evaluator

Once created, you can use your code evaluator: