Overview
Spider allows you to use high performance proxies to prevent detection, caches AI actions, webhooks for crawling status, scheduled crawls etc… This guide shows how to crawl/scrape a website using Spider and loading the LLM-ready documents withSpiderLoader
in LanghChain.
Setup
Get your own Spider API key on spider.cloud.Usage
Here’s an example of how to use theSpiderLoader
:
Spider offers two scraping modes scrape
and crawl
. Scrape only gets the content of the url provided while crawl gets the content of the url provided and crawls deeper following subpages.
npm
Additional Parameters
See the Spider documentation for all the availableparams
.