Skip to main content
Audit logs are available on Enterprise plans.
Organization Admins and Organization Operators can forward LangSmith audit logs to a security information and event management (SIEM) system. A scheduled script pulls events from the audit logs API and posts them to the SIEM’s ingest endpoint. Your security team can then search LangSmith activity, such as agent access token requests, next to logs from your other systems. This page uses Splunk as the example. The same approach works with any SIEM that accepts JSON events over HTTP.

Prerequisites

Meet the audit logs prerequisites. On a self-hosted instance, enable audit logs first. You also need:
  • A personal access token or service key with the Organization Admin or Organization Operator role.
  • Your organization ID.
  • An HTTP endpoint and token for your SIEM.
  • Python 3.10 or later, to run the example script.

Pull events on a schedule

LangSmith does not push audit logs to other systems. Instead, a script you run on a schedule pulls each new window of events. Events follow the OCSF API Activity schema, and the full LangSmith record is under unmapped.original_audit_log. When you write the script:
  • Lag the window: LangSmith writes audit events in batches, so an event can appear about a minute after the request. End each window 2 minutes before the current time.
  • Keep windows back to back: Use the same length for the schedule and the window. For example, run every 5 minutes over a 5-minute window.
  • Follow the cursor: Each response returns at most 100 events. Pass the returned cursor to get the next page, until a response has no cursor.
  • Deduplicate on metadata.uid: start_time and end_time are both inclusive, so an event on a window boundary can appear in two windows. metadata.uid is unique for each event.
  • Retry failed windows: A failed run skips its window. A scheduler with a dead-letter queue, such as AWS Lambda with EventBridge Scheduler, records which windows to retry.
The Splunk example follows all of these except retries. It removes duplicates in each search, with dedup metadata.uid. To send events to another SIEM, replace its send_to_splunk function.

Find agent access token requests

An agent access token is a short-lived token that identifies a deployed agent to other services. Each time LangSmith issues one, it records an event with the operation mint_agent_access_token:
  • The event belongs to the agent’s organization and workspace. This includes Fleet agents, whose tokens LangSmith requests on your behalf.
  • A failed request does not record an event.
  • The token itself is never recorded.
The token’s details are under unmapped.original_audit_log.enrichments.agent_access_token. Each field uses the name of the matching claim in the token. To match an event to a token, use its jti or sub. The event’s actor depends on how the deployment proved its identity, shown in ls_proof:
  • deployment_key: actor.user.credential_uid is the key’s ID.
  • sandbox_callback: A sandbox requested the token for a user, and unmapped.original_audit_log.ls_user_id is that user.
  • pod_token: Both fields are empty. This applies to your own deployments that use a pod token and to agents LangSmith runs for you, such as Fleet agents.
resources lists the agent’s UUID and the deployment ID. To find one agent’s tokens through the API, pass the agent’s UUID as resource_ids. Tokens for AWS, Azure, and Google Cloud record an event too. To match one to your cloud provider’s logs, use sub, aud, and the time.

Forward events to Splunk

This example sends events to the Splunk HTTP Event Collector (HEC), then searches them for agent access token requests.

Create an HEC token

Create an HEC token in Splunk, following Splunk’s guide to setting up HEC in Splunk Web. Give the token a recognizable name, such as langsmith-audit-logs, and note the index it writes to. The searches on this page use main. Copy the token value for the forwarder. On Splunk Enterprise, the event endpoint is https://<splunk-host>:8088/services/collector/event, where 8088 is the default HEC port.

Run the forwarder

Save the following script as splunk_forwarder.py. Each run reads one window of events and posts them to HEC in a single request.
Set four environment variables, then run the script:
LANGSMITH_API_URL defaults to the GCP US API URL. For another region, set it to that region’s API URL from Regions. On a self-hosted instance, set it to your instance’s base URL, without /api/v1. To run the script every 5 minutes, add a cron entry:
Cron does not load your shell profile, so set the environment variables in the crontab or in a wrapper script.

Search for agent access token requests

Each search starts by dropping duplicate events and shortening the token’s long field paths to token.*:
Then add one of these:

See also