Set up decision model online evaluators

Online evaluations provide real-time feedback on your production traces. An online evaluator can use a decision model as the judge in place of an LLM. To use an LLM as the judge, see Set up LLM-as-a-judge online evaluators.

​

Decision models

Decision models answer structured questions about text and return typed answers, such as a probability, a selected option, or a score, rather than free-form text. For how they work, see Decision models in the LLM Gateway. When you use a decision model as the judge, LangSmith records the answer to each question you define under its own feedback key. You do not need to write an output schema or parse free-text reasoning.

​

Supported models

LangSmith supports OpenAI and TypeSafe decision models for evaluators, on both tracing projects and datasets. To call a decision model through another provider or on your own server, use an OpenAI-Decisions-compatible or TypeSafe-compatible endpoint. See Connect to a compatible decision model provider.

​

Question types

Each question you add to a decision model evaluator has one of three types. For full definitions, see the TypeSafe primitives documentation or the OpenAI Decisions guide.

• Predicate or Noul: A continuous boolean. OpenAI models and OpenAI-Decisions-compatible endpoints label this type Predicate, and TypeSafe models and TypeSafe-compatible endpoints label it Noul. Returns the probability, from 0 to 1, that a statement is true. Values near 1 mean yes, values near 0 mean no, and values near 0.5 mean the model is uncertain. Use it for yes or no checks, such as whether the output contains personally identifiable information. Phrase the instructions as a yes or no question, and optionally describe what true and false mean.
• Choice: Picks one option from a fixed, unordered list, such as classifying whether the user is asking a question, making a request, or neither (other). Give each option a name and, unless the name is self-explanatory, a description.
• Score: Rates the state along ordered levels, such as how frustrated the user is, from calm to very angry. Define the levels in order, starting at level 0. The score can fall between two levels.

The OpenAI Decisions API has no field for the true and false descriptions of a predicate. For OpenAI models and OpenAI-Decisions-compatible endpoints, LangSmith appends them to the question’s instructions. LangSmith enforces the following limits on decision model questions:

​

Map answers to feedback keys

Each question’s name becomes a feedback key on the evaluated run or thread. LangSmith converts each typed answer to feedback as follows: Feedback stores only the score or value. The model’s full answer, which includes probabilities and confidence for choice and score questions, appears in the evaluator’s trace. LangSmith traces evaluators by default. If tracing is turned off for the evaluator, no trace is recorded. To open the trace on a tracing project, go to the Evaluators tab, click the evaluator, and click the Evaluator traces tab. In an experiment, hover over the score and click Evaluator trace. Because each answer is standard feedback, you can filter, chart, and alert on it, and trigger automations from it, as you would with any other evaluator’s feedback. Question names follow the same limits as other feedback keys and must be unique within the evaluator.

​

Add a decision model online evaluator

Decision model evaluators require a LangSmith Plus or Enterprise plan. They are not available on BYOC deployments. To add a decision model online evaluator:

1. In the LangSmith UI, navigate to the Tracing page and select a tracing project.
2. Click the Evaluators tab.
3. Click + Evaluator to open the Configure Evaluator panel.
4. Under Create from scratch, select Decision Model Evaluator, then name your evaluator.
5. Select a decision model, map run or thread variables into the State, and add questions. For each setting, see How to define a decision model evaluator.
6. (Optional) Apply a filter, set a sampling rate, or apply the rule to past runs or threads. These settings work the same way as for LLM-as-a-judge online evaluators.
7. To save the evaluator, click Create or Save.

The evaluator checks each new run or thread that matches its filter. Each question’s answer appears under its own feedback key.

​

See also

• How to define a decision model evaluator: Configure the model, state, and questions.
• Connect to a compatible decision model provider: Base URL format and example endpoints.
• Decision models in the LLM Gateway: Call decision models through the LLM Gateway from your own application.

---