Skip to main content
While you can manually sift through and process production logs from your LLM application, it often becomes difficult as your application scales to more users. LangSmith provides Automations that allow you to trigger certain actions on your trace data. You can define an automations by a filter, sampling rate, and action. Automation rules can trigger actions such as: adding traces to a dataset, adding to an annotation queue, triggering a webhook (e.g., for remote evaluations) or extending data retention. Some examples of automations you can set up:
  • Send all traces with negative feedback to an annotation queue for human review.
  • Send 10% of all traces to an annotation queue for human review to spot check for issues.
  • Upgrade all traces with errors for extended data retention.
To configure online evaluations, visit the online evaluations page.
An automation rule can upgrade matching traces to extended data retention when retention extension is enabled for that rule. This upgrade impacts trace pricing, but ensures that traces meeting your automation criteria (typically those most valuable for analysis) are preserved for investigation. Each action type has its own default, refer to the action-level retention settings for details. For the full retention model, see data retention auto-upgrades.

​
How automation rules execute

Each automation rule runs on an independent polling schedule. If you have multiple rules on the same project, a webhook rule may process a run before an evaluator rule has scored it, or vice versa. Within a single rule, if multiple actions are configured, they execute in this order:
  1. Add to annotation queue.
  2. Add to dataset.
  3. Trigger webhook.
  4. Run online evaluator.
  5. Run custom code evaluator.
  6. Trigger alert.
If your workflow requires data produced by one rule to be present when another fires—for example, you want a webhook to include an evaluation score—use a filter on the downstream rule to create that dependency explicitly. For an example, refer to Ensuring evaluations complete before the webhook fires.

​
View automation rules

In the UI, navigate to the Tracing in the sidebar and select a tracing project. To view existing automation rules for that tracing project, click on the Automations tab.

​
Create a rule

  1. In the UI, navigate to the Tracing in the sidebar and select a tracing project. Click on + New in the top right corner of the tracing project page, then click on New Automation.
  2. Name your rule.
  3. Create a filter. Automation rule filters work the same way as filters applied to traces in the project. For more information on filters, you can refer to Filter traces.
  4. Configure a Sampling Rate to control the percentage of filtered runs that trigger the automation action. You can specify a sampling rate between 0 and 1 for automations. This will control the percent of the filtered runs that are sent to an automation action. For example, if you set the sampling rate to 0.5, then 50% of the traces that pass the filter will be sent to the action.
  5. (Optional) Apply rule to past runs by toggling the Apply to past runs and entering a Backfill from date. This is only possible upon rule creation.
    The backfill is processed as a background job, so you will not see the results immediately. In order to track progress of the backfill, you can view logs for your automations.
  6. Select an action to trigger when the rule is applied. There are four actions you can take with an automation rule:
    • Add to dataset: Add the inputs and outputs of the trace to a dataset.
    • Add to annotation queue: Add the matching run/trace to an annotation queue as a run item. Automation rules do not enqueue entire threads as thread items; to review a full conversation, add the thread from the UI (Assign runs and threads).
    • Trigger webhook: Trigger a webhook with the trace data.
    • Extend data retention: Extends the data retention period on matching traces that use base retention (refer to the data retention docs for more details).
      Each action has an independent Extend data retention toggle that controls whether matching traces are upgraded to extended retention:
      • Add to dataset: opt-in (default: off). Enable the toggle to upgrade matching traces.
      • Add to annotation queue: opt-out (default: on). Disable the toggle to skip upgrading matching traces.
      • Trigger webhook: opt-in (default: off). Enable the toggle to upgrade matching traces.
      • Extend data retention action and online/code evaluators: unchanged; always upgrade matching traces.
      The retention toggle for each action is an admin-only control, gated by the rules:configure-retention permission. Non-admin workspace members see the toggles as disabled and cannot change them, but can still create and edit rules without affecting retention settings. For the full retention model, refer to data retention auto-upgrades.

​
View logs for your automations

Logs allow you to gain confidence that your rules are working as expected. You can view logs for your automations by navigating to the Automations tab within a tracing project and clicking the Logs button for the rule you created. The logs tab allows you to:
  • View all runs processed by a given rule for the time period selected.
  • If a particular rule execution has triggered an error, you can view the error message by hovering over the error icon.
  • You can monitor the progress of a backfill job by filtering to the rule’s creation timestamp. This is because the backfill starts from when the rule was created.
  • Inspect the run that the automation rule applied to using the View run button. For rules that add runs as examples to datasets, you can view the example produced.
Connect these docs to Claude, VSCode, and more via MCP for real-time answers.
Edit this page on GitHub or file an issue.