> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage feedback & annotation queues programmatically

Use the LangSmith SDK to manage feedback configurations and [annotation queue](/langsmith/evaluation-concepts#human) rubrics programmatically. Define reusable feedback schemas at the organization level (like accuracy scores or pass/fail judgments), then assign them to specific queues with custom instructions. This enables version control, automation across projects, and consistency—particularly useful for CI/CD pipelines or replicating evaluation setups across environments.

<Callout icon="code">
  This guide uses the Python and TypeScript SDKs. For installation and setup, refer to the [Python SDK documentation](https://reference.langchain.com/python/langsmith) and [TypeScript SDK documentation](https://reference.langchain.com/javascript/modules/langsmith.html).
</Callout>

## Feedback layers

LangSmith uses a three-layer architecture for structured human feedback:

1. **Feedback configs**: Organization-wide definitions of feedback keys that establish the schema for evaluation metrics. For example, you might define "accuracy" as a continuous 0–1 score or "correctness" as a pass/fail categorical choice. These configs are reusable across all annotation queues in your organization.
2. **Annotation queue rubric items**: Queue-specific assignments that determine which feedback configs annotators must fill out when reviewing [runs](/langsmith/observability-concepts#runs) in a particular queue. Each rubric item can include custom descriptions, guidance for specific score values, and whether the feedback is required or optional.
3. **Feedback**: Individual scores and values that annotators submit on specific [runs](/langsmith/observability-concepts#runs). This is the actual evaluation data collected using the schemas you've defined. Learn more about [feedback in LangSmith](/langsmith/observability-concepts#feedback).

## Feedback configs

### Create a feedback config

Feedback configs define the schema for a feedback key—whether it's a continuous score, a categorical choice, or freeform text. A unique key identifies each config within your organization and specifies how annotators can submit feedback for that metric.

<Note>
  Calling [`create_feedback_config`](https://reference.langchain.com/python/langsmith/client/Client/create_feedback_config) with an identical config that already exists returns the existing config. If a different config already exists for the same key, the system raises a 400 error.
</Note>

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langsmith import Client

  client = Client()

  # Continuous score
  client.create_feedback_config(
      "accuracy",
      feedback_config={
          "type": "continuous",
          "min": 0,
          "max": 1,
      },
      is_lower_score_better=False,
  )

  # Categorical
  client.create_feedback_config(
      "correctness",
      feedback_config={
          "type": "categorical",
          "categories": [
              {"value": 1, "label": "Pass"},
              {"value": 0, "label": "Fail"},
          ],
      },
  )

  # Freeform text
  client.create_feedback_config(
      "notes",
      feedback_config={"type": "freeform"},
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { Client } from "langsmith";

  const client = new Client();

  // Continuous score
  await client.createFeedbackConfig({
    feedbackKey: "accuracy",
    feedbackConfig: { type: "continuous", min: 0, max: 1 },
    isLowerScoreBetter: false,
  });

  // Categorical
  await client.createFeedbackConfig({
    feedbackKey: "correctness",
    feedbackConfig: {
      type: "categorical",
      categories: [
        { value: 1, label: "Pass" },
        { value: 0, label: "Fail" },
      ],
    },
  });

  // Freeform text
  await client.createFeedbackConfig({
    feedbackKey: "notes",
    feedbackConfig: { type: "freeform" },
  });
  ```
</CodeGroup>

* **Continuous** (`"accuracy"`): Defines a numeric scale from 0 to 1. The `is_lower_score_better` parameter indicates whether lower values represent better performance. Use continuous configs for rating scales or percentage-based metrics.
* **Categorical** (`"correctness"`): Provides predefined options with associated values. Each category requires a `value` (used for scoring and analytics) and a `label` (shown to annotators). Use categorical configs for binary choices or multi-class classifications.
* **Freeform** (`"notes"`): Allows open-ended text input with no predefined structure. Use freeform configs for qualitative observations or explanations.

### List feedback configs

Retrieve feedback configs to see what evaluation criteria are available in your organization with [`list_feedback_configs`](https://reference.langchain.com/python/langsmith/client/Client/list_feedback_configs). You can list all configs or filter by specific keys. Each returned config object includes the key, type, configuration details (like `min`/`max` or `categories`), and metadata like `is_lower_score_better`:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  # List all configs
  for config in client.list_feedback_configs():
      print(f"{config.feedback_key}: {config.feedback_config}")

  # Filter by specific keys
  for config in client.list_feedback_configs(
      feedback_key=["accuracy", "correctness"]
  ):
      print(config.feedback_key)
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  // List all configs
  for await (const config of client.listFeedbackConfigs()) {
    console.log(`${config.feedback_key}: ${JSON.stringify(config.feedback_config)}`);
  }

  // Filter by specific keys
  for await (const config of client.listFeedbackConfigs({
    feedbackKeys: ["accuracy", "correctness"],
  })) {
    console.log(config.feedback_key);
  }
  ```
</CodeGroup>

### Update a feedback config

Modify an existing feedback config with [`update_feedback_config`](https://reference.langchain.com/python/langsmith/client/Client/update_feedback_config) by updating specific fields. The method only changes the fields you provide—the rest remain unchanged. This is a partial update that preserves other configuration settings:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  client.update_feedback_config(
      "accuracy",
      is_lower_score_better=True,
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  await client.updateFeedbackConfig("accuracy", {
    isLowerScoreBetter: true,
  });
  ```
</CodeGroup>

### Delete a feedback config

Remove a feedback config from your organization with [`delete_feedback_config`](https://reference.langchain.com/python/langsmith/client/Client/delete_feedback_config). This performs a soft delete, which marks the config as deleted but doesn't permanently remove it from the system. You can recreate a config with the same key later if needed:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  client.delete_feedback_config("accuracy")
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  await client.deleteFeedbackConfig("accuracy");
  ```
</CodeGroup>

## Annotation queue rubric items

Rubric items assign feedback configs to a specific annotation queue. They control which feedback forms annotators see when reviewing [runs](/langsmith/observability-concepts#runs) in that queue, and whether each form is required or optional.

### Create a queue with rubric items

Create an annotation queue with [`create_annotation_queue`](https://reference.langchain.com/python/langsmith/client/Client/create_annotation_queue) and assign feedback configs to it through rubric items. Each rubric item references a feedback config by its key and customizes how it appears to annotators in this specific queue.

The example creates a queue with three rubric items. The queue-level `rubric_instructions` provides general guidance shown at the top of the annotation interface:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  queue = client.create_annotation_queue(
      name="QA Review Queue",
      description="Review LLM outputs for accuracy and correctness",
      rubric_instructions="Score each response. Add notes for anything unusual.",
      rubric_items=[
          {
              "feedback_key": "accuracy",
              "description": "How accurate is the response?",
              "score_descriptions": {
                  "0": "Completely wrong",
                  "1": "Perfectly accurate",
              },
              "is_required": True,
          },
          {
              "feedback_key": "correctness",
              "description": "Did the response pass or fail?",
              "value_descriptions": {
                  "Pass": "Factually correct",
                  "Fail": "Contains errors",
              },
              "is_required": True,
          },
          {
              "feedback_key": "notes",
              "description": "Any additional observations",
              "is_required": False,
          },
      ],
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  const queue = await client.createAnnotationQueue({
    name: "QA Review Queue",
    description: "Review LLM outputs for accuracy and correctness",
    rubricInstructions: "Score each response. Add notes for anything unusual.",
    rubricItems: [
      {
        feedback_key: "accuracy",
        description: "How accurate is the response?",
        score_descriptions: { "0": "Completely wrong", "1": "Perfectly accurate" },
        is_required: true,
      },
      {
        feedback_key: "correctness",
        description: "Did the response pass or fail?",
        value_descriptions: { Pass: "Factually correct", Fail: "Contains errors" },
        is_required: true,
      },
      {
        feedback_key: "notes",
        description: "Any additional observations",
        is_required: false,
      },
    ],
  });
  ```
</CodeGroup>

* `feedback_key`: The key of an existing feedback config (create this first).
* `description`: Queue-specific guidance for annotators about this metric.
* `score_descriptions` / `value_descriptions`: Optional labels that explain what specific values mean (use `score_descriptions` for continuous configs, `value_descriptions` for categorical).
* `is_required`: Whether annotators must complete this feedback before submitting.

### Update rubric items on an existing queue

Modify the rubric items assigned to an annotation queue with [`update_annotation_queue`](https://reference.langchain.com/python/langsmith/client/Client/update_annotation_queue). This operation replaces the entire rubric items list, so you must include all items you want to keep—the operation removes any items you don't include.

You'll need the queue ID, which you get when you create the queue or by listing queues:

<Note>
  Updating rubric items replaces the full list. Include all items you want to keep.
</Note>

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  client.update_annotation_queue(
      queue.id,
      rubric_items=[
          {"feedback_key": "accuracy", "is_required": True},
          {"feedback_key": "correctness", "is_required": True},
          {
              "feedback_key": "tone",
              "description": "Is the tone appropriate?",
              "is_required": False,
          },
      ],
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  await client.updateAnnotationQueue(queue.id, {
    rubricItems: [
      { feedback_key: "accuracy", is_required: true },
      { feedback_key: "correctness", is_required: true },
      { feedback_key: "tone", description: "Is the tone appropriate?", is_required: false },
    ],
  });
  ```
</CodeGroup>

## Feedback config types (detailed)

### Continuous

Continuous configs define numeric rating scales with minimum and maximum values. Annotators can select any value within the range, making this ideal for scoring dimensions like accuracy, quality, or relevance on a numeric scale:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  # Simple continuous score
  client.create_feedback_config(
      "accuracy",
      feedback_config={
          "type": "continuous",
          "min": 0,
          "max": 1,
      },
  )

  # Continuous with labeled points on the scale
  client.create_feedback_config(
      "quality",
      feedback_config={
          "type": "continuous",
          "min": 1,
          "max": 5,
          "categories": [
              {"value": 1, "label": "Poor"},
              {"value": 3, "label": "Average"},
              {"value": 5, "label": "Excellent"},
          ],
      },
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  await client.createFeedbackConfig({
    feedbackKey: "accuracy",
    feedbackConfig: { type: "continuous", min: 0, max: 1 },
  });

  await client.createFeedbackConfig({
    feedbackKey: "quality",
    feedbackConfig: {
      type: "continuous",
      min: 1,
      max: 5,
      categories: [
        { value: 1, label: "Poor" },
        { value: 3, label: "Average" },
        { value: 5, label: "Excellent" },
      ],
    },
  });
  ```
</CodeGroup>

The first example shows a 0–1 scale without labels. The second example demonstrates adding `categories` with labeled anchor points on the scale (like "Poor", "Average", "Excellent") to help annotators understand what different values represent. These labels are optional but can improve consistency in how annotators interpret the scale.

### Categorical

Categorical configs provide a discrete set of predefined options for annotators to choose from. Each category must have a `value` (a numeric identifier used for scoring and analytics) and a `label` (the text shown to annotators). You must define at least 2 categories.

Use categorical configs for binary decisions (pass/fail, correct/incorrect), multi-class classifications (sentiment, topic categories), or any evaluation with a fixed set of discrete options. Do not set `min` or `max` for categorical configs:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  # Binary pass/fail
  client.create_feedback_config(
      "correctness",
      feedback_config={
          "type": "categorical",
          "categories": [
              {"value": 1, "label": "Pass"},
              {"value": 0, "label": "Fail"},
          ],
      },
  )

  # Multi-class
  client.create_feedback_config(
      "sentiment",
      feedback_config={
          "type": "categorical",
          "categories": [
              {"value": 0, "label": "Negative"},
              {"value": 1, "label": "Neutral"},
              {"value": 2, "label": "Positive"},
          ],
      },
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  await client.createFeedbackConfig({
    feedbackKey: "correctness",
    feedbackConfig: {
      type: "categorical",
      categories: [
        { value: 1, label: "Pass" },
        { value: 0, label: "Fail" },
      ],
    },
  });

  await client.createFeedbackConfig({
    feedbackKey: "sentiment",
    feedbackConfig: {
      type: "categorical",
      categories: [
        { value: 0, label: "Negative" },
        { value: 1, label: "Neutral" },
        { value: 2, label: "Positive" },
      ],
    },
  });
  ```
</CodeGroup>

The first example shows a binary pass/fail config. The second example demonstrates a multi-class config for sentiment with three options. The numeric values allow you to compute aggregate scores even for categorical feedback.

### Freeform

Freeform configs allow annotators to provide open-ended text feedback without any predefined structure or constraints. This type has no `min`, `max`, or `categories` fields—annotators can enter any text they want.

Freeform feedback is valuable for capturing nuanced insights but is harder to aggregate and analyze compared to structured feedback types:

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  client.create_feedback_config(
      "notes",
      feedback_config={"type": "freeform"},
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  await client.createFeedbackConfig({
    feedbackKey: "notes",
    feedbackConfig: { type: "freeform" },
  });
  ```
</CodeGroup>

## Validation rules

| Type          | min/max         | categories                      | Constraints                                         |
| ------------- | --------------- | ------------------------------- | --------------------------------------------------- |
| `continuous`  | Optional        | Optional (labeled scale points) | `min < max`; category values within \[`min`, `max`] |
| `categorical` | Must not be set | Required, min 2                 | Unique values and labels                            |
| `freeform`    | Must not be set | Must not be set                 | N/A                                                 |

## Reference

### Feedback config types

| Type          | Fields                                | Description                       |
| ------------- | ------------------------------------- | --------------------------------- |
| `continuous`  | `min`, `max`                          | Numeric score within a range      |
| `categorical` | categories (list of `{value, label}`) | Selection from predefined options |
| `freeform`    | None                                  | Free-text input                   |

### Rubric item fields

| Field                | Type                     | Description                                                                      |
| -------------------- | ------------------------ | -------------------------------------------------------------------------------- |
| `feedback_key`       | `string`                 | Required. Must match an existing feedback config key.                            |
| `description`        | `string`                 | Shows annotators guidance for this item.                                         |
| `score_descriptions` | `Record<string, string>` | Labels for specific score values (continuous).                                   |
| `value_descriptions` | `Record<string, string>` | Labels for specific category values (categorical).                               |
| `is_required`        | `boolean`                | Whether annotators must complete this item before submitting. Defaults to false. |

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/annotation-queues-sdk.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>