Set up decision model online evaluators

Online evaluations provide real-time feedback on your production traces. An online evaluator can use a decision model, such as SemIf or Jev, as the judge in place of an LLM. To use an LLM as the judge, see Set up LLM-as-a-judge online evaluators.

​

Decision models

Decision models answer structured questions about text and return typed answers, such as a probability, a selected option, or a score, rather than free-form text. For how they work, see Decision models in the LLM Gateway. When you use a decision model as the judge, LangSmith records the answer to each question you define under its own feedback key. You do not need to write an output schema or parse free-text reasoning.

​

Supported models

LangSmith supports two decision models for evaluators, on both tracing projects and datasets. Both are configured and behave the same way. They differ only in provider and setup. To call a decision model through another provider or on your own server, use a TypeSafe-compatible endpoint.

​

Question types

Each question you add to a decision model evaluator has one of three types. For full definitions, see the TypeSafe primitives documentation.

• Noul: A continuous boolean. Returns the probability, from 0 to 1, that a statement is true. Values near 1 mean yes, values near 0 mean no, and values near 0.5 mean the model is uncertain. Use it for yes or no checks, such as whether the output contains personally identifiable information. Phrase the instructions as a yes or no question, and optionally describe what true and false mean.
• Choice: Picks one option from a fixed, unordered list, such as classifying whether the user is asking a question, making a request, or neither (other). Give each option a name and, unless the name is self-explanatory, a description.
• Score: Rates the state along ordered levels, such as how frustrated the user is, from calm to very angry. Define the levels in order, starting at level 0. The score can fall between two levels.

Each model enforces its own limits on questions. A SemIf evaluator accepts up to 32 questions.

​

Map answers to feedback keys

Each question’s name becomes a feedback key on the evaluated run or thread. LangSmith converts each typed answer to feedback as follows: LangSmith also stores the model’s full answer on each feedback entry, under typesafe in the feedback source metadata. For choice and score questions, the answer includes probabilities and confidence. Because each answer is standard feedback, you can filter, chart, and alert on it, and trigger automations from it, as you would with any other evaluator’s feedback. Question names follow the same limits as other feedback keys, and each name must be unique within the evaluator.

​

Add a decision model online evaluator

To add a decision model online evaluator:

1. In the LangSmith UI, navigate to the Tracing page and select a tracing project.
2. Click the Evaluators tab.
3. Click + Evaluator to open the Configure Evaluator panel.
4. Under Create from scratch, select LLM-as-a-Judge Evaluator, then name your evaluator.
5. Select a decision model, map run or thread variables into the State, and add questions. For each setting, see How to define a decision model evaluator.
6. (Optional) Apply a filter, set a sampling rate, or apply the rule to past runs or threads. These settings work the same way as for LLM-as-a-judge online evaluators.
7. To save the evaluator, click Create or Save.

The evaluator checks each new run or thread that matches its filter. Each question’s answer appears under its own feedback key.

​

See also

• How to define a decision model evaluator: Configure the model, state, and questions.
• Connect to a TypeSafe-compatible model provider: Base URL format and example endpoints.
• Decision models in the LLM Gateway: Call SemIf and Jev directly from code.

---