How to define a decision model evaluator

A decision model evaluator uses a decision model as the judge. Each question you define returns a typed answer that LangSmith records under its own feedback key. This guide shows you how to define a decision model evaluator in the LangSmith UI. You can use it on a dataset to automatically run evaluations on experiments, or on a tracing project as an online evaluator. To use an LLM as the judge instead, see How to define an LLM-as-a-judge evaluator.

​

Step 1. Create the evaluator

A decision model evaluator offers only decision model providers, and an LLM-as-a-judge evaluator offers only LLM providers. To switch between an LLM judge and a decision model, create a new evaluator. To create a decision model evaluator:

1. Add an API key for your decision model provider. In LangSmith, go to Settings > Integrations > Provider secrets and click + Secret:
   • For OpenAI, select OpenAI and paste your OpenAI API key. LangSmith stores it as the workspace secret OPENAI_API_KEY.
   • For TypeSafe, select TypeSafe and paste a key from your TypeSafe account. LangSmith stores it as the workspace secret TYPESAFE_API_KEY.
   To store a key under a different name, select Custom and enter your own secret name. For more information, see Add a provider key.
2. In the LangSmith UI, open the Evaluators tab of a dataset or tracing project and click + Evaluator.
3. In the Configure Evaluator panel, select Decision Model Evaluator under Create from scratch.
4. Name your evaluator.

​

Step 2. Configure the evaluator

Decision model evaluators support OpenAI and TypeSafe decision models with bring-your-own-key (BYOK). You define the state and questions in the evaluator itself. Decision model evaluators cannot load a prompt from the Prompt Hub or use a custom output schema. They also do not support few-shot examples. To reuse a decision model evaluator on another dataset or tracing project, select Attach an existing evaluator in the Configure Evaluator panel.

​

Model

Under Prompt & Model, open Model Configuration and select a provider and model:

• For OpenAI, select OpenAI as the provider, then select a decision model. If you stored your key under a name other than OPENAI_API_KEY, enter that name in API Key Name. Click Apply.
• For TypeSafe, select TypeSafe as the provider, then select a Jev model. If you stored your key under a name other than TYPESAFE_API_KEY, enter that name in API Key Name. Click Apply.
• For an OpenAI-Decisions-compatible endpoint or a TypeSafe-compatible endpoint, select its saved configuration. See Use a compatible endpoint.

For the supported models, see Supported models.

​

Use a compatible endpoint

A compatible endpoint points the evaluator at any server that implements the OpenAI Decisions API or the TypeSafe System One API. Examples include OpenRouter or a decision model you host yourself. An evaluator can use one only after you save it as a model configuration. For the settings, setup steps, and example base URLs, see Connect to a compatible decision model provider.

​

State

The State is the context the decision model evaluates. Map variables from your run or example into it:

• On a dataset, map the input, output, or reference output.
• On a tracing project, map run or thread variables, such as the run’s inputs and outputs.

Unlike a prompt for an LLM judge, the state should not include grading instructions. Put grading criteria in the questions instead. Decision models evaluate text only. If the state includes multimodal content, such as images, the evaluator run fails.

​

Questions

Questions are the scoring criteria your decision model evaluator uses. Each question’s name becomes a feedback key on the evaluated run or thread. To define questions:

1. Under Feedback Configuration, add one question for each criterion you want to evaluate.
2. For each question, enter a Name, select a Type, and write the Instructions. The yes or no question type is labeled Predicate for OpenAI models and OpenAI-Decisions-compatible endpoints, and Noul for TypeSafe models and TypeSafe-compatible endpoints.
3. Define the true and false meanings, options, or levels for the question type.

To edit the questions as JSON, click Advanced. For what each question type returns and how answers map to feedback, see Question types and Map answers to feedback keys.

​

Step 3. Save the evaluator

To save the evaluator, click Create or Save. On a dataset, the evaluator runs on each new experiment. On a tracing project, it evaluates incoming runs or threads that match its filter. For filters, sampling rates, and backfills, see Set up decision model online evaluators.

​

See also

• Decision models: Supported models, question types, and feedback keys.
• Connect to a compatible decision model provider: Base URL format and example endpoints.
• Decision models in the LLM Gateway: Call decision models through the LLM Gateway from your own application.

---