Use the LangSmith SDK to create and manage evaluators programmatically. Evaluators created through the SDK are workspace-level resources that appear in the Evaluators table in the LangSmith UI, the same as evaluators created in the UI. You can attach them to datasets to run offline evaluations and to tracing projects to run online evaluations. Use the SDK to automate evaluator management and integrate evaluation into your existing workflows.
Prerequisites
Managing evaluators through the SDK requires:
- Python:
langsmith>=0.8.17 (PyPI)
- TypeScript:
langsmith>=0.7.10 (npm)
The examples on this page initialize the client with no arguments, so it reads the LANGSMITH_API_KEY and LANGSMITH_ENDPOINT environment variables. Configure your API key through environment variables rather than hardcoding it.
In the following examples, replace placeholders such as <evaluator-uuid> with the corresponding information from LangSmith.
Create an evaluator
Code evaluator
A code evaluator scores each run or example with a function that you define.
from langsmith import Client
client = Client()
created = client.online_evaluators.create(
name="Correctness evaluator",
type="code",
code_evaluator={
"code": "def perform_eval(run, example):\n return {'score': 1}",
"language": "python",
},
)
evaluator_id = created.evaluator.id
print("Created evaluator:", evaluator_id)
LLM-as-a-judge evaluator
An LLM-as-a-judge evaluator references a prompt from the prompt hub and maps your run or example fields to the prompt variables.
The prompt must be a structured prompt (type StructuredPrompt). A StructuredPrompt combines a prompt template with an output schema, ensuring the model returns data in a defined structure.
created = client.online_evaluators.create(
name="LLM judge",
type="llm",
llm_evaluator={
"prompt_repo_handle": "<prompt-repo-handle>",
"commit_hash_or_tag": "<commit-hash-or-tag>",
"variable_mapping": {
"input": "inputs.question",
"output": "outputs.answer",
"reference": "reference.answer",
},
},
)
evaluator_id = created.evaluator.id
prompt_repo_handle is the prompt’s internal repository name, not its display title or URL. To find it, list your workspace prompts and read the repo_handle field, or retrieve a specific prompt by its identifier in LangSmith. The identifier of a prompt can be in the format:
promptName (for private prompts), for example my-prompt.owner/promptName (for public prompts), for example langchain-ai/correctness.
# List workspace prompts and read each repo handle
for prompt in client.list_prompts(limit=10).repos:
print("prompt-repo-handle:", prompt.repo_handle) # value to use for prompt_repo_handle
print("prompt-full-name:", prompt.full_name) # display name
print("description:", prompt.description)
# Or retrieve a specific prompt by identifier
prompt = client.get_prompt("<prompt-identifier>")
print("prompt-repo-handle:", prompt.repo_handle)
Retrieve an evaluator
Fetch a single evaluator by its ID to read its configuration, including its name, type, feedback keys, and run rules.
evaluator = client.online_evaluators.retrieve(evaluator_id)
print(evaluator.name)
print(evaluator.type)
print(evaluator.feedback_keys)
print(evaluator.run_rules)
Update an evaluator
Pass the field that matches the evaluator type: code_evaluator for a code evaluator or llm_evaluator for an LLM-as-a-judge evaluator. update changes only the fields you pass.
# Update a code evaluator
code_evaluator_id = "<code-evaluator-uuid>"
updated = client.online_evaluators.update(
code_evaluator_id,
name="Updated correctness evaluator",
code_evaluator={
"code": "def perform_eval(run, example):\n return {'score': 0.8}",
"language": "python",
},
)
print(updated.evaluator.name if updated.evaluator else None)
# Update the name and prompt of an LLM-as-a-judge evaluator
llm_evaluator_id = "<llm-evaluator-uuid>"
client.online_evaluators.update(
llm_evaluator_id,
name="Updated LLM judge",
llm_evaluator={
"prompt_repo_handle": "<prompt-repo-handle>",
"commit_hash_or_tag": "<commit-hash-or-tag>",
},
)
variable_mapping: Maps run or example fields to the judge prompt variables. It applies when the evaluator runs.use_corrections_dataset and num_few_shot_examples: Enable few-shot learning from human score corrections. They apply only when the evaluator is attached to a project or dataset and corrections have been submitted.
These settings take effect on the next evaluation run, not when you call update.
llm_evaluator_id = "<llm-evaluator-uuid>"
client.online_evaluators.update(
llm_evaluator_id,
llm_evaluator={
"prompt_repo_handle": "<prompt-repo-handle>",
"commit_hash_or_tag": "<commit-hash-or-tag>",
"variable_mapping": {
"input": "inputs.question",
"output": "outputs.answer",
},
"use_corrections_dataset": True,
"num_few_shot_examples": 3,
},
)
List evaluators
Filter by name, type, feedback key, attached resource, or tag value, and sort or paginate the results. list() auto-paginates through every match when you iterate the returned object directly. limit sets the per-request page size (1 to 100), not the total number of results. sort_by is optional, accepts created_at or updated_at, and defaults to created_at.
# Read a single page of results
page = client.online_evaluators.list(
name_contains="correctness",
type="code",
limit=10,
)
for evaluator in page.evaluators:
print(evaluator.id, evaluator.name, evaluator.type)
# Collect every match into a list
evaluators = list(
client.online_evaluators.list(feedback_key="correctness", limit=20)
)
# Filter, sort, and paginate
evaluators = client.online_evaluators.list(
feedback_key="correctness",
name_contains="judge",
resource_id=["<project-or-dataset-uuid>"],
tag_value_id=["<tag-value-uuid>"],
type="llm",
sort_by="updated_at", # "created_at" (default) or "updated_at"
sort_by_desc=False,
limit=20,
offset=0,
)
Track evaluator spend
Retrieve estimated USD spend and trace counts for your evaluators:
period_start: A date-only ISO string, such as 2026-06-29. Passing a datetime returns a 400 error.
- Window:
period_start starts a fixed 7-day window that includes period_start and the six days after it. The window is half-open, [period_start, period_start + 7 days), so period_end (period_start plus 7 days) is excluded. For example, a period_start of 2026-06-29 covers 2026-06-29 through 2026-07-05, and 2026-07-06 is excluded.
type: Scopes results to a single evaluator type, llm or code. Omit it to include all types.- Empty result: If no spend is recorded for the window, the returned
groups list is empty.
Pass exactly one of group_by, evaluator_id, session_id (the LangSmith tracing project UUID), or dataset_id.
evaluator_uuid = "<evaluator-uuid>"
start_date = "<period-start-date>" # for example, "2026-06-29"
# Spend for a single evaluator
spend = client.online_evaluators.spend(
period_start=start_date,
evaluator_id=evaluator_uuid,
)
for group in spend.groups or []:
print(group.evaluator_name, group.total_spend_usd, group.total_trace_count)
# Group spend by evaluator
spend_by_evaluator = client.online_evaluators.spend(
period_start=start_date,
group_by="evaluator",
type="llm",
)
print("Group by evaluator")
for group in spend_by_evaluator.groups or []:
print(group.evaluator_name, group.total_spend_usd, group.total_trace_count)
# Group spend by resource
spend_by_resource = client.online_evaluators.spend(
period_start=start_date,
group_by="resource",
type="llm",
)
print("Group by resource")
for group in spend_by_resource.groups or []:
print(group.session_name, group.dataset_name, group.total_spend_usd, group.total_trace_count)
# Group spend by run_rule
spend_by_run_rule = client.online_evaluators.spend(
period_start=start_date,
group_by="run_rule",
type="llm",
)
print("Group by run_rule")
for group in spend_by_run_rule.groups or []:
print(group.run_rule_name, group.total_spend_usd, group.total_trace_count)
Delete an evaluator
You cannot delete an evaluator while it is attached to a tracing project or dataset. Set delete_run_rules to true to delete the run rules that reference the evaluator before deleting the evaluator.
evaluator_id = "<evaluator-uuid>"
client.online_evaluators.delete(
evaluator_id,
delete_run_rules=True, # run rules referencing the evaluator are deleted first
)
