How to define a decision model evaluator

A decision model evaluator uses a decision model, such as SemIf or Jev, as the judge. Each question you define returns a typed answer that LangSmith records under its own feedback key. This guide shows you how to define a decision model evaluator in the LangSmith UI. You can use it on a dataset to automatically run evaluations on experiments, or on a tracing project as an online evaluator. To use an LLM as the judge instead, see How to define an LLM-as-a-judge evaluator.

​

Step 1. Create the evaluator

1. (Jev only) Add a TypeSafe API key. Generate an API key from your TypeSafe account. In LangSmith, go to Settings > Integrations > Provider secrets and click + Secret. Select TypeSafe as the provider and paste your key. LangSmith stores it as the workspace secret TYPESAFE_API_KEY. To store it under a different name, select Custom and enter your own secret name. For more information, see Add provider secrets.
2. In the LangSmith UI, open the Evaluators tab of a dataset or tracing project and click + Evaluator.
3. In the Configure Evaluator panel, select LLM-as-a-Judge Evaluator under Create from scratch.
4. Name your evaluator.

​

Step 2. Configure the evaluator

Decision model evaluators support TypeSafe models, such as Jev, with bring-your-own-key (BYOK), and SemIf through the LLM Gateway. You define the state and questions in the evaluator itself. Decision model evaluators cannot load a prompt from the Prompt Hub or use a custom output schema. They also do not support few-shot examples. To reuse a decision model evaluator on another dataset or tracing project, select Attach an existing evaluator in the Configure Evaluator panel.

​

Model

Under Prompt & Model, open Model Configuration and select a provider and model:

• For SemIf, select LangSmith Gateway as the provider, then select a SemIf model. Click Apply.
• For Jev, select TypeSafe as the provider, then select a Jev model. If you stored your key under a name other than TYPESAFE_API_KEY, enter that name in API Key Name. Click Apply.
• For a TypeSafe-compatible endpoint, select its saved configuration. See Use a TypeSafe-compatible endpoint.

For a comparison of the supported models, see Supported models.

​

Use a TypeSafe-compatible endpoint

A TypeSafe-compatible endpoint points the evaluator at any server that implements the TypeSafe System One API. Examples include OpenRouter or a decision model you host yourself. An evaluator can use one only after you save it as a model configuration. For the settings, setup steps, and example base URLs, see Connect to a TypeSafe-compatible model provider.

​

State

The State is the context the decision model evaluates. Map variables from your run or example into it:

• On a dataset, map the input, output, or reference output.
• On a tracing project, map run or thread variables, such as the run’s inputs and outputs.

Unlike a prompt for an LLM judge, the state should not include grading instructions. Put grading criteria in the questions instead.

​

Questions

Questions are the scoring criteria your decision model evaluator uses. Each question’s name becomes a feedback key on the evaluated run or thread. To define questions:

1. Under Feedback Configuration, add one question for each criterion you want to evaluate.
2. For each question, enter a Name, select a Type, and write the Instructions.
3. Define the true and false meanings, options, or levels for the question type.

To edit the questions as JSON, click Advanced. For what each question type returns and how answers map to feedback, see Question types and Map answers to feedback keys.

​

Step 3. Save the evaluator

To save the evaluator, click Create or Save. On a dataset, the evaluator runs on each new experiment. On a tracing project, it evaluates incoming runs or threads that match its filter. For filters, sampling rates, and backfills, see Set up decision model online evaluators.

​

See also

• Decision models: Supported models, question types, and feedback keys.
• Connect to a TypeSafe-compatible model provider: Base URL format and example endpoints.
• Decision models in the LLM Gateway: Call SemIf and Jev directly from code.

---