> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.
# How to define a decision model evaluator
> Set up a LangSmith evaluator that uses SemIf or Jev as the judge, with noul, choice, and score questions that each return typed feedback.
A decision model evaluator uses a [decision model](/langsmith/online-evaluations-decision-models#decision-models), such as SemIf or Jev, as the judge. Each question you define returns a typed answer that LangSmith records under its own feedback key.
This guide shows you how to define a decision model evaluator in the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-decision-model-evaluator). You can use it on a dataset to [automatically run evaluations on experiments](/langsmith/bind-evaluator-to-dataset), or on a tracing project as an [online evaluator](/langsmith/online-evaluations-decision-models). To use an LLM as the judge instead, see [How to define an LLM-as-a-judge evaluator](/langsmith/llm-as-judge).
You can create decision model evaluators only in the LangSmith UI. The LangSmith SDKs do not support creating them yet.
SemIf is enabled for US organizations on Free, Developer, and Plus plans.
## Step 1. Create the evaluator
1. (Jev only) Add a TypeSafe API key. Generate an API key from your TypeSafe account. In LangSmith, go to **Settings > Integrations > Provider secrets** and click **+ Secret**. Select **TypeSafe** as the provider and paste your key. LangSmith stores it as the workspace secret `TYPESAFE_API_KEY`. To store it under a different name, select **Custom** and enter your own secret name. For more information, see [Add provider secrets](/langsmith/llm-gateway-admin-setup#1-add-provider-secrets).
2. In the LangSmith UI, open the **Evaluators** tab of a dataset or tracing project and click **+ Evaluator**.
3. In the **Configure Evaluator** panel, select **LLM-as-a-Judge Evaluator** under **Create from scratch**.
4. Name your evaluator.
## Step 2. Configure the evaluator
Decision model evaluators support TypeSafe models, such as Jev, with bring-your-own-key (BYOK), and SemIf through the [LLM Gateway](/langsmith/llm-gateway). You define the state and questions in the evaluator itself. Decision model evaluators cannot load a prompt from the [Prompt Hub](/langsmith/prompt-context-hub#prompts) or use a custom output schema. They also do not support [few-shot examples](/langsmith/create-few-shot-evaluators). To reuse a decision model evaluator on another dataset or tracing project, select **Attach an existing evaluator** in the **Configure Evaluator** panel.
### Model
Under **Prompt & Model**, open **Model Configuration** and select a provider and model:
* For SemIf, select **LangSmith Gateway** as the provider, then select a SemIf model. Click **Apply**.
* For Jev, select **TypeSafe** as the provider, then select a Jev model. If you stored your key under a name other than `TYPESAFE_API_KEY`, enter that name in **API Key Name**. Click **Apply**.
* For a TypeSafe-compatible endpoint, select its saved configuration. See [Use a TypeSafe-compatible endpoint](#use-a-typesafe-compatible-endpoint).
For a comparison of the supported models, see [Supported models](/langsmith/online-evaluations-decision-models#supported-models).
#### Use a TypeSafe-compatible endpoint
A TypeSafe-compatible endpoint points the evaluator at any server that implements the [TypeSafe System One API](https://docs.typesafe.ai). Examples include OpenRouter or a decision model you host yourself. An evaluator can use one only after you save it as a model configuration. For the settings, setup steps, and example base URLs, see [Connect to a TypeSafe-compatible model provider](/langsmith/typesafe-compatible-model).
### State
The **State** is the context the decision model evaluates. Map variables from your run or example into it:
* On a dataset, map the input, output, or reference output.
* On a tracing project, map run or thread variables, such as the run's inputs and outputs.
Unlike a prompt for an LLM judge, the state should not include grading instructions. Put grading criteria in the questions instead.
### Questions
Questions are the scoring criteria your decision model evaluator uses. Each question's name becomes a feedback key on the evaluated run or thread. To define questions:
1. Under **Feedback Configuration**, add one question for each criterion you want to evaluate.
2. For each question, enter a **Name**, select a **Type**, and write the **Instructions**.
3. Define the true and false meanings, options, or levels for the question type.
To edit the questions as JSON, click **Advanced**.
For what each question type returns and how answers map to feedback, see [Question types](/langsmith/online-evaluations-decision-models#question-types) and [Map answers to feedback keys](/langsmith/online-evaluations-decision-models#map-answers-to-feedback-keys).
## Step 3. Save the evaluator
To save the evaluator, click **Create** or **Save**.
On a dataset, the evaluator runs on each new experiment. On a tracing project, it evaluates incoming runs or threads that match its filter. For filters, sampling rates, and backfills, see [Set up decision model online evaluators](/langsmith/online-evaluations-decision-models).
## See also
* [Decision models](/langsmith/online-evaluations-decision-models#decision-models): Supported models, question types, and feedback keys.
* [Connect to a TypeSafe-compatible model provider](/langsmith/typesafe-compatible-model): Base URL format and example endpoints.
* [Decision models in the LLM Gateway](/langsmith/llm-gateway-decision-models): Call SemIf and Jev directly from code.
***
[Connect these docs](/use-these-docs) to your agent of choice via MCP for real-time answers.
[Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/decision-model-evaluator.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).