How to collect user feedback for Agent Server runs

​

How it works

1. Create a run and include feedback_keys in the request body. For example, when calling POST /threads/{thread_id}/runs/stream, set feedback_keys in the request body to:
   Copy

   ["user_liked", "user_disliked"]
   
2. The feedback object from the response contains a pre-signed URL for each key. For example, the feedback object is:
   Copy

   {
       "user_liked": "https://api.smith.langchain.com/api/v1/feedback/tokens/ef19fedf-dcac-4cbb-a59c-00661efd6425",
       "user_disliked": "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5"
   }
   
3. Request the returned URL (e.g. POST /api/v1/feedback/tokens/{token_id}) to associate the feedback key with the trace generated from the Agent Server run. For more details, refer to the LangSmith API reference.
4. LangSmith associates the submitted feedback with the run using the selected feedback key (e.g. user_liked or user_disliked).

​

Call the streaming run API with feedback_keys

from langgraph_sdk import get_client

client = get_client(url="<DEPLOYMENT_URL>", api_key="<API_KEY>")

thread = await client.threads.create()
thread_id = thread["thread_id"]

feedback_urls = {}

async for event in client.runs.stream(
    thread_id,
    "agent",
    input={
        "messages": [
            {"role": "user", "content": "Tell me a joke about databases."}
        ]
    },
    stream_mode="updates",
    feedback_keys=["user_liked", "user_disliked"],
):
    if event.event == "feedback":
        # Example: {"user_liked": ".../feedback/tokens/<id>", "user_disliked": "..."}
        feedback_urls = event.data
        print("Feedback URLs:", feedback_urls)
    elif event.event == "updates":
        print(event.data)

import { Client } from "@langchain/langgraph-sdk";

const client = new Client({ apiUrl: "<DEPLOYMENT_URL>", apiKey: "<API_KEY>" });

const thread = await client.threads.create();
const threadId = thread.thread_id;

let feedbackUrls = {};

const streamResponse = client.runs.stream(threadId, "agent", {
  input: {
    messages: [{ role: "user", content: "Tell me a joke about databases." }],
  },
  streamMode: "updates",
  feedbackKeys: ["user_liked", "user_disliked"],
});

for await (const event of streamResponse) {
  if (event.event === "feedback") {
    // Example: { user_liked: ".../feedback/tokens/<id>", user_disliked: "..." }
    feedbackUrls = event.data;
    console.log("Feedback URLs:", feedbackUrls);
  } else if (event.event === "updates") {
    console.log(event.data);
  }
}

curl --request POST \
  --url "<DEPLOYMENT_URL>/threads/<THREAD_ID>/runs/stream" \
  --header "Content-Type: application/json" \
  --header "x-api-key: <API_KEY>" \
  --data '{
    "assistant_id": "agent",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "Tell me a joke about databases."
        }
      ]
    },
    "stream_mode": "updates",
    "feedback_keys": ["user_liked", "user_disliked"]
  }'

​

Handle the streamed feedback event

event: feedback
data: {"user_liked":"https://api.smith.langchain.com/api/v1/feedback/tokens/ef19fedf-dcac-4cbb-a59c-00661efd6425", "user_disliked": "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5"}

​

Submit feedback with the generated URL

curl --request POST \
  --url "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5" \
  --header "Content-Type: application/json" \
  --data '{
    "score": 1,
    "value": 0,
    "comment": "I didn't like this joke because it didn't make me laugh.",
    "correction": {},
    "metadata": {}
  }'

curl --request GET \
  --url "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5?score=1&value=0&comment=I%20didn%27t%20like%20this%20joke%20because%20it%20didn%27t%20make%20me%20laugh.&correction=%7B%7D"

​

Optimize feedback data model

• Use key="user_score" with score=1 for user_liked
• Use key="user_score" with score=-1 for user_disliked

​

Productionize in a client UI

1. Create the run from your backend or frontend.
2. Capture the feedback object and store the returned URLs.
3. Render feedback controls such as thumbs up/down buttons and feedback forms.
4. On feedback submission, POST or GET a feedback URL based on the user’s feedback intent.
5. Optionally disable the feedback controls after submission and show confirmation to the user.