> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Validate Rule

> Validate a rule by executing it with test data without creating a saved rule.

This endpoint allows testing LLM-as-judge evaluators before saving them. It accepts
a rule configuration (same as rule creation) and test data, executes the evaluator,
and returns the evaluation results in the same format as batch_invoke_evaluator.

Only LLM-as-judge rules (evaluators) are supported. Code evaluators are not allowed.

The evaluator execution traces are written to the database (in the "evaluators"
project), which allows users to see the evaluator execution history.



## OpenAPI

````yaml /langsmith/langsmith-platform-openapi.json post /api/v1/runs/rules/validate
openapi: 3.1.0
info:
  title: LangSmith
  description: >+
    The LangSmith API is used to programmatically create and manage LangSmith
    resources.


    ## Host

    https://api.smith.langchain.com


    ## Authentication

    To authenticate with the LangSmith API, set the `X-Api-Key` header

    to a valid [LangSmith API
    key](https://docs.langchain.com/langsmith/create-account-api-key#create-an-api-key).

  version: 0.1.0
servers:
  - url: /
security: []
tags:
  - name: run
    x-group: Tracing
  - name: runs
    x-group: Tracing
  - name: sessions
    x-group: Tracing
  - name: tracer-sessions
    x-group: Tracing
  - name: datasets
    x-group: Datasets
  - name: examples
    x-group: Datasets
  - name: evaluators
    x-group: Evaluation
  - name: experiment-view-overrides
    x-group: Evaluation
  - name: experiments
    x-group: Evaluation
  - name: annotation-queues
    x-group: Feedback & Annotation
  - name: annotation_queues
    x-group: Feedback & Annotation
  - name: feedback
    x-group: Feedback & Annotation
  - name: feedback-configs
    x-group: Feedback & Annotation
  - name: alert_rules
    x-group: Monitoring
  - name: bulk-exports
    x-group: Monitoring
  - name: charts
    x-group: Monitoring
  - name: commits
    x-group: Prompts & Playground
  - name: directories
    x-group: Prompts & Playground
  - name: hub_environments
    x-group: Prompts & Playground
  - name: playground-settings
    x-group: Prompts & Playground
  - name: prompt-webhooks
    x-group: Prompts & Playground
  - name: prompts
    x-group: Prompts & Playground
  - name: tag-transitions
    x-group: Prompts & Playground
  - name: comments
    x-group: Prompt Hub
  - name: likes
    x-group: Prompt Hub
  - name: optimization-jobs
    x-group: Prompt Hub
  - name: ownerships
    x-group: Prompt Hub
  - name: repos
    x-group: Prompt Hub
  - name: settings
    x-group: Prompt Hub
  - name: tags
    x-group: Prompt Hub
  - name: integrations
    x-group: Integrations & Tools
  - name: mcp
    x-group: Integrations & Tools
  - name: mcp_vendors
    x-group: Integrations & Tools
  - name: oauth
    x-group: Integrations & Tools
  - name: tools
    x-group: Integrations & Tools
  - name: gateway-policies
    x-group: LLM Gateway
  - name: sandboxes
    x-group: Sandboxes
  - name: issues
    x-group: Issues
  - name: issues-agent
    x-group: Issues
  - name: Organizations
    x-group: Administration
  - name: SCIM Tokens
    x-group: Administration
  - name: TTL Settings
    x-group: Administration
  - name: access_policies
    x-group: Administration
  - name: api-key
    x-group: Administration
  - name: audit-logs
    x-group: Administration
  - name: auth
    x-group: Administration
  - name: aws_marketplace
    x-group: Administration
  - name: data_planes
    x-group: Administration
  - name: me
    x-group: Administration
  - name: orgs
    x-group: Administration
  - name: service-accounts
    x-group: Administration
  - name: tenant
    x-group: Administration
  - name: ttl-settings
    x-group: Administration
  - name: usage-limits
    x-group: Administration
  - name: workspaces
    x-group: Administration
  - name: ace
    x-group: System
  - name: backfills
    x-group: System
  - name: features
    x-group: System
  - name: info
    x-group: System
  - name: model-price-map
    x-group: System
  - name: public
    x-group: System
  - name: threads
    x-group: System
  - name: agents
    x-hidden: true
  - name: beacon
    x-hidden: true
  - name: engine-trial
  - name: fleet
  - name: fleet auth
    x-hidden: true
  - name: fleet credentials
    x-hidden: true
  - name: fleet github-app
    x-hidden: true
  - name: fleet integrations
    x-hidden: true
  - name: fleet mcp
    x-hidden: true
  - name: fleet threads
    x-hidden: true
  - name: fleet trigger-templates
    x-hidden: true
  - name: fleet triggers
    x-hidden: true
  - name: fleet usage
    x-hidden: true
  - name: fleet_webhooks
    x-hidden: true
  - name: health
  - name: internal
    x-hidden: true
  - name: metrics
    x-hidden: true
  - name: nps
    x-hidden: true
  - name: sandboxes-internal
    x-hidden: true
  - name: skills
    x-hidden: true
paths:
  /api/v1/runs/rules/validate:
    post:
      tags:
        - run
      summary: Validate Rule
      description: >-
        Validate a rule by executing it with test data without creating a saved
        rule.


        This endpoint allows testing LLM-as-judge evaluators before saving them.
        It accepts

        a rule configuration (same as rule creation) and test data, executes the
        evaluator,

        and returns the evaluation results in the same format as
        batch_invoke_evaluator.


        Only LLM-as-judge rules (evaluators) are supported. Code evaluators are
        not allowed.


        The evaluator execution traces are written to the database (in the
        "evaluators"

        project), which allows users to see the evaluator execution history.
      operationId: validate_rule_api_v1_runs_rules_validate_post
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/RunRulesValidateSchema'
        required: true
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                items:
                  additionalProperties: true
                  type: object
                type: array
                title: Response Validate Rule Api V1 Runs Rules Validate Post
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      security:
        - API Key: []
        - Tenant ID: []
        - Bearer Auth: []
components:
  schemas:
    RunRulesValidateSchema:
      properties:
        display_name:
          type: string
          title: Display Name
        session_id:
          anyOf:
            - type: string
              format: uuid
            - type: 'null'
          title: Session Id
        is_enabled:
          type: boolean
          title: Is Enabled
          default: true
        dataset_id:
          anyOf:
            - type: string
              format: uuid
            - type: 'null'
          title: Dataset Id
        sampling_rate:
          type: number
          title: Sampling Rate
        filter:
          anyOf:
            - type: string
            - type: 'null'
          title: Filter
        trace_filter:
          anyOf:
            - type: string
            - type: 'null'
          title: Trace Filter
        tree_filter:
          anyOf:
            - type: string
            - type: 'null'
          title: Tree Filter
        backfill_from:
          anyOf:
            - type: string
              format: date-time
            - type: 'null'
          title: Backfill From
        use_corrections_dataset:
          type: boolean
          title: Use Corrections Dataset
          default: false
        num_few_shot_examples:
          anyOf:
            - type: integer
            - type: 'null'
          title: Num Few Shot Examples
        extend_only:
          type: boolean
          title: Extend Only
          default: false
        transient:
          type: boolean
          title: Transient
          default: false
        add_to_annotation_queue_id:
          anyOf:
            - type: string
              format: uuid
            - type: 'null'
          title: Add To Annotation Queue Id
        add_to_dataset_id:
          anyOf:
            - type: string
              format: uuid
            - type: 'null'
          title: Add To Dataset Id
        add_to_dataset_prefer_correction:
          type: boolean
          title: Add To Dataset Prefer Correction
          default: false
        evaluators:
          anyOf:
            - items:
                $ref: '#/components/schemas/EvaluatorTopLevel'
              type: array
            - type: 'null'
          title: Evaluators
        code_evaluators:
          anyOf:
            - items:
                $ref: '#/components/schemas/CodeEvaluatorTopLevel'
              type: array
            - type: 'null'
          title: Code Evaluators
        evaluator_id:
          anyOf:
            - type: string
              format: uuid
            - type: 'null'
          title: Evaluator Id
        alerts:
          anyOf:
            - items:
                $ref: '#/components/schemas/RunRulesPagerdutyAlertSchema'
              type: array
            - type: 'null'
          title: Alerts
        webhooks:
          anyOf:
            - items:
                $ref: '#/components/schemas/RunRulesWebhookSchema'
              type: array
            - type: 'null'
          title: Webhooks
        evaluator_version:
          anyOf:
            - type: integer
            - type: 'null'
          title: Evaluator Version
        create_alignment_queue:
          type: boolean
          title: Create Alignment Queue
          default: false
        include_extended_stats:
          type: boolean
          title: Include Extended Stats
          default: false
        group_by:
          anyOf:
            - type: string
              const: thread_id
            - type: 'null'
          title: Group By
        spend_limit:
          anyOf:
            - $ref: '#/components/schemas/RunRuleSpendLimitSchema-Input'
            - type: 'null'
        tracer_session_issue_id:
          anyOf:
            - type: string
              format: uuid
            - type: 'null'
          title: Tracer Session Issue Id
        test_inputs:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Test Inputs
        test_outputs:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Test Outputs
        test_reference_outputs:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Test Reference Outputs
        test_attachments:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Test Attachments
      type: object
      required:
        - display_name
        - sampling_rate
      title: RunRulesValidateSchema
      description: |-
        Schema for validating rules without creating them.

        Extends RunRulesCreateSchema with test data fields for validation.
        Only LLM-as-judge rules (evaluators) are supported, not code_evaluators.
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    EvaluatorTopLevel:
      properties:
        structured:
          $ref: '#/components/schemas/EvaluatorStructuredOutput'
      type: object
      required:
        - structured
      title: EvaluatorTopLevel
    CodeEvaluatorTopLevel:
      properties:
        code:
          type: string
          title: Code
        language:
          anyOf:
            - $ref: '#/components/schemas/CodeEvaluatorLanguage'
            - type: 'null'
          default: python
      type: object
      required:
        - code
      title: CodeEvaluatorTopLevel
    RunRulesPagerdutyAlertSchema:
      properties:
        type:
          anyOf:
            - $ref: '#/components/schemas/RunRulesAlertType'
            - type: 'null'
          default: pagerduty
        routing_key:
          type: string
          title: Routing Key
        summary:
          anyOf:
            - type: string
            - type: 'null'
          title: Summary
        severity:
          anyOf:
            - $ref: '#/components/schemas/PagerdutySeverity'
            - type: 'null'
          default: warning
      type: object
      required:
        - routing_key
      title: RunRulesPagerdutyAlertSchema
    RunRulesWebhookSchema:
      properties:
        url:
          type: string
          title: Url
        headers:
          anyOf:
            - type: object
              additionalProperties:
                type: string
            - type: 'null'
          title: Headers
      type: object
      required:
        - url
      title: RunRulesWebhookSchema
    RunRuleSpendLimitSchema-Input:
      properties:
        limit_usd:
          anyOf:
            - type: number
              exclusiveMinimum: 0
            - type: string
          title: Limit Usd
        window:
          $ref: '#/components/schemas/RunRuleSpendLimitWindow'
      type: object
      required:
        - limit_usd
        - window
      title: RunRuleSpendLimitSchema
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
    EvaluatorStructuredOutput:
      properties:
        hub_ref:
          anyOf:
            - type: string
            - type: 'null'
          title: Hub Ref
        prompt:
          anyOf:
            - items:
                prefixItems:
                  - type: string
                  - type: string
                type: array
                maxItems: 2
                minItems: 2
              type: array
            - type: 'null'
          title: Prompt
        template_format:
          anyOf:
            - type: string
            - type: 'null'
          title: Template Format
        schema:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Schema
        variable_mapping:
          anyOf:
            - type: object
              additionalProperties:
                type: string
            - type: 'null'
          title: Variable Mapping
        model:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Model
      type: object
      title: EvaluatorStructuredOutput
      description: Evaluator structured output schema.
    CodeEvaluatorLanguage:
      type: string
      enum:
        - python
        - javascript
      title: CodeEvaluatorLanguage
    RunRulesAlertType:
      type: string
      enum:
        - pagerduty
      title: RunRulesAlertType
      description: Enum for alert types.
    PagerdutySeverity:
      type: string
      enum:
        - critical
        - warning
        - error
        - info
      title: PagerdutySeverity
      description: Enum for severity.
    RunRuleSpendLimitWindow:
      type: string
      enum:
        - weekly
      title: RunRuleSpendLimitWindow
  securitySchemes:
    API Key:
      type: apiKey
      in: header
      name: X-API-Key
    Tenant ID:
      type: apiKey
      in: header
      name: X-Tenant-Id
    Bearer Auth:
      type: http
      description: >-
        Bearer tokens are used to authenticate from the UI. Must also specify
        x-tenant-id or x-organization-id (for org scoped apis).
      scheme: bearer

````