Documentation Index
Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
Use this file to discover all available pages before exploring further.
Tavily is a search engine built specifically for AI agents (LLMs), delivering real-time, accurate, and factual results at speed. Tavily offers a Crawl endpoint that performs a structured web traversal from a starting URL, with built-in content extraction and intelligent discovery.
Overview
Integration details
| Returns artifact | Native async | Return data | Pricing |
|---|
| ❌ | ✅ | base URL, page URL, raw content, images, favicon | 1,000 free credits / month |
Setup
The integration lives in the @langchain/tavily package, which you can install as shown below:
npm install @langchain/tavily @langchain/core
Credentials
Set up a Tavily API key and set it as an environment variable named TAVILY_API_KEY.
process.env.TAVILY_API_KEY = "YOUR_API_KEY"
process.env.LANGSMITH_TRACING="true"
process.env.LANGSMITH_API_KEY="your-api-key"
Instantiation
The tool accepts the following parameters during instantiation:
maxDepth (optional, number): Maximum number of hops from the starting URL. Default is 3.maxBreadth (optional, number): Maximum number of pages to crawl per level. Default is 50.limit (optional, number): Maximum total number of pages to crawl. Default is 100.instructions (optional, string): Natural-language instructions that guide the crawler.selectPaths (optional, string[]): Only crawl URLs containing these path regexes.selectDomains (optional, string[]): Only crawl these domain regexes.excludePaths (optional, string[]): Skip URLs containing these path regexes.excludeDomains (optional, string[]): Skip URLs from these domain regexes.allowExternal (optional, boolean): Allow the crawler to follow external links.extractDepth (optional, string): "basic" or "advanced" extraction depth for each crawled page. Default is "basic".format (optional, string): "markdown" or "text" content format. Default is "markdown".includeImages (optional, boolean): Include images discovered on each page. Default is false.
For a comprehensive overview of the available parameters, refer to the Tavily Crawl API documentation.
import { TavilyCrawl } from "@langchain/tavily";
const tool = new TavilyCrawl({
maxDepth: 3,
maxBreadth: 50,
// limit: 100,
// extractDepth: "basic",
// format: "markdown",
// includeImages: false,
// allowExternal: false,
});
Invocation
The Tavily crawl tool accepts the following arguments during invocation:
url (required): The base URL to start crawling from.- The following arguments can also be set during invocation:
instructions, selectPaths, selectDomains, excludePaths, excludeDomains, allowExternal.
NOTE: The optional arguments are available for agents to dynamically set. If you set an argument during instantiation and then invoke the tool with a different value, the tool will use the value you passed during invocation.
await tool.invoke({ url: "https://docs.tavily.com" });
{
"base_url": "https://docs.tavily.com",
"results": [
{
"url": "https://docs.tavily.com/",
"raw_content": "\n\n\n##### Welcome\n\n## Documentation Index\n\nFetch the complete documentation index at: <https://docs.tavily.com/llms.txt>\n\nUse this file to discover all available pages before exploring further.\n\n# Build with Tavily\n\nYour journey to state-of-the-art web search starts right here.\n\n## Python SDK\n\n`pip install tavily-python`\n\n## JavaScript SDK\n\n`npm i @tavily/core`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.search(\"Who is Leo Messi?\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.extract(\"https://en.wikipedia.org/wiki/Artificial_intelligence\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.crawl(\"https://docs.tavily.com\", instructions=\"Find all pages on the Python SDK\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.map(\"https://docs.tavily.com\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.research(\"What are the latest developments in AI?\")\n\nprint(response)`\n\n### Developer Resources\n\n## API Credits Overview\n\n## Rate Limits\n\n## Playground",
"favicon": "https://docs.tavily.com/mintlify-assets/_mintlify/favicons/tavilyai/SXaxSfweEU3ftIlh/_generated/favicon/apple-touch-icon.png"
},
{
"url": "https://docs.tavily.com/documentation/about",
"raw_content": "Performance\n\nCommitted to speed and efficiency, our API guarantees real-time and trusted information. Our team works hard to improve Tavily’s performance over time.\n\nIntegration-friendly\n\nWe appreciate the essence of adaptability. That’s why integrating our API with your existing setup is a breeze. You can choose our Python library, JavaScript package or a simple API call. You can also use Tavily through any of our supported partners such as LangChain and LlamaIndex.\n\nTransparent & Informative\n\nOur detailed documentation ensures you’re never left in the dark. From setup basics to nuanced features, we’ve got you covered.\n\n## \n\nHow does the Search API work? [...] With LLM hallucinations, it’s crucial to optimize for RAG with the right context and information. This is where Tavily comes in, delivering accurate and precise information for your RAG applications.\n\n## \n\nGetting started\n\nSign up for Tavily to get your API key. You get 1,000 free API Credits every month. No credit card required.\n\n## Get your free API key You get 1,000 free API Credits every month. No credit card required.\n\nHead to our API Playground to familiarize yourself with our API.To get started with Tavily’s APIs and SDKs using code, head to our Quickstart Guide and follow the steps.\n\nGot questions? Stumbled upon an issue? Simply intrigued? Don’t hesitate! Our support team is always on standby, eager to assist. Join us, dive deep, and redefine your search experience! Contact us! [...] # About - Tavily Docs\n\nSkip to main content\n\nTavily Docs home pageImage 1: light logoImage 2: dark logo\n\nSearch...\n\nCtrl K Ask AI\n\n Support\n Get an API key\n Get an API key\n\nSearch...\n\nNavigation\n\nGetting Started\n\nAbout\n\nHomeIntroductionAPI & SDKsEcosystemExamplesChangelogHelp\n\n API Playground\n Community\n Blog\n\n##### Getting Started\n\n About\n Quickstart\n Credits & Pricing\n Rate Limits\n\n##### FAQ\n\n Frequently Asked Questions\n\nOn this page\n\n Who are we?\n What is the Tavily Search Engine?\n Why choose Tavily?\n How does the Search API work?\n Getting started\n\nGetting Started\n\n# About\n\nCopy page\n\nWelcome to Tavily!\n\nCopy page\n\nLooking for a step-by-step tutorial to get started in under 5 minutes? Head to our Quickstart guide and start coding!\n\n## \n\nWho are we?",
"favicon": "https://docs.tavily.com/mintlify-assets/_mintlify/favicons/tavilyai/SXaxSfweEU3ftIlh/_generated/favicon/apple-touch-icon.png"
},
{
"url": "https://docs.tavily.com/documentation/api-reference/introduction",
"raw_content": "# Introduction - Tavily Docs\n\nSkip to main content\n\nTavily Docs home pageImage 1: light logoImage 2: dark logo\n\nSearch...\n\n⌘K Ask AI\n\n Support\n Get an API key\n Get an API key\n\nSearch...\n\nNavigation\n\nAPI Reference\n\nIntroduction\n\nHomeIntroductionAPI & SDKsEcosystemExamplesChangelogHelp\n\n API Playground\n Community\n Blog\n\n##### API Reference\n\n Introduction\n POST Search\n POST Extract\n POST Crawl\n POST Map\n Research \n GET Usage\n\n##### Enterprise API Reference\n\n API Key Generator \n\n##### Python SDK\n\n Quickstart\n SDK Reference\n\n##### JavaScript SDK\n\n Quickstart\n SDK Reference\n\n##### Best Practices\n\n Search\n Extract\n Crawl\n Research\n API Key Management\n\nOn this page\n\n Base URL\n Authentication\n Endpoints\n Project Tracking\n\nAPI Reference [...] When using the SDKs, you can specify a project using the `project_id` parameter when instantiating the client, or by setting the `TAVILY_PROJECT` environment variable.\n\nTavily Search Next\n\n⌘I\n\nTavily Docs home pageImage 3: light logoImage 4: dark logo\n\nlinkedinxgithubyoutube\n\nResources\n\nAboutTavily Search CrawlerTavily Status\n\nLegal\n\nSecurity & CompliancePrivacy PolicyWebsite Terms of UsePlatform Terms of UseCookie Notice\n\nlinkedinxgithubyoutube\n\nPowered by This documentation is built and hosted on Mintlify, a developer documentation platform\n\nAssistant\n\nResponses are generated using AI and may contain mistakes. [...] API Reference\n\n# Introduction\n\nCopy page\n\nEasily integrate our APIs with your services.\n\nCopy page\n\n> ## Documentation Index\n> \n> \n> Fetch the complete documentation index at: \n> \n> \n> Use this file to discover all available pages before exploring further.\n\n## \n\nBase URL\n\nThe base URL for all requests to the Tavily API is:\n\n```\n\n```\n\n## \n\nAuthentication\n\nAll Tavily endpoints are authenticated using API keys. Get your free API key.\n\n```\ncurl -X POST \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer tvly-YOUR_API_KEY\" \\\n -d '{\"query\": \"Who is Leo Messi?\"}'\n```\n\n## \n\nEndpoints\n\n`/search`Tavily’s powerful web search API.\n\n`/extract`Tavily’s powerful content extraction API.\n\n`/crawl` , `/map`Tavily’s intelligent sitegraph navigation and extraction tools.",
"favicon": "https://docs.tavily.com/mintlify-assets/_mintlify/favicons/tavilyai/SXaxSfweEU3ftIlh/_generated/favicon/apple-touch-icon.png"
},
{
"url": "https://github.com/tavily-ai",
"raw_content": "JavaScript1.8k239 \n2. tavily-pythontavily-python Public \nThe Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. E…\n\nPython1.2k148 \n3. skillsskills Public \nPython23321 \n4. meeting-prep-agentmeeting-prep-agent Public \nPrepare for meetings with company and attendee research notes, integrated with Google Calendar through MCP\n\nPython19334 \n5. tavily-jstavily-js Public \nOfficial TypeScript/JavaScript library for Tavily.\n\nTypeScript839 \n6. tavily-chattavily-chat Public \nConversational agent that fuses chat data with live web results through Tavily search, extract, and crawl.\n\nTypeScript7216 \n\n### Repositories\n\nLoading\n\nType\n\nSelect type [...] ### Repositories\n\nLoading\n\nType\n\nSelect type\n\nAll Public Sources Forks Archived Mirrors Templates \n\nLanguage\n\nSelect language\n\nAll JavaScript Jupyter Notebook Python TypeScript \n\nSort\n\nSelect order\n\nLast updated Name Stars \n\nShowing 10 of 33 repositories [...] langchain-tavily Public ### Uh oh! There was an error while loading. [Please reload this page.]( past year of commit activity Python22MIT 1511 Updated Apr 16, 2026 \n langchainjs Public Forked from langchain-ai/langchainjs\n🦜🔗 Build context-aware reasoning applications ### Uh oh! There was an error while loading. [Please reload this page.]( past year of commit activity TypeScript 0 MIT 3,20800 Updated Apr 16, 2026",
"favicon": "https://github.githubassets.com/favicons/favicon.svg"
}
],
"response_time": 13.45
}
ToolCall, in which case a ToolMessage will be returned:
// This is usually generated by a model, but we'll create a tool call directly for demo purposes.
const modelGeneratedToolCall = {
args: {
url: "https://docs.tavily.com",
instructions: "Find API reference pages",
},
id: "1",
name: tool.name,
type: "tool_call",
};
const toolMsg = await tool.invoke(modelGeneratedToolCall);
console.log(toolMsg.content.slice(0, 400));
Use within an agent
We can use the crawl tool directly with a LangChain agent by passing it to createAgent. The agent can dynamically set url, instructions, and the path/domain filters as part of its tool call.
// @lc-docs-hide-cell
import { ChatOpenAI } from "@langchain/openai";
const llm = new ChatOpenAI({
model: "gpt-5.4",
temperature: 0,
});
import { TavilyCrawl } from "@langchain/tavily";
import { createAgent } from "langchain";
const tavilyCrawlTool = new TavilyCrawl({
maxDepth: 1,
maxBreadth: 10,
limit: 10,
});
const agent = createAgent({
model: llm,
tools: [tavilyCrawlTool],
});
const userInput = "Crawl https://docs.tavily.com and find the API reference for the search endpoint.";
const events = await agent.stream(
{ messages: [["human", userInput]] },
{ streamMode: "values" },
);
for await (const event of events) {
const lastMsg = event.messages[event.messages.length - 1];
if (lastMsg.tool_calls?.length) {
console.dir(lastMsg.tool_calls, { depth: null });
} else if (lastMsg.content) {
console.log(lastMsg.content);
}
}
API reference
For detailed documentation of all Tavily Crawl API features and configurations head to the API reference: docs.tavily.com/documentation/api-reference/endpoint/crawl
