AgentQL’s document loader provides structured data extraction from any web page using an AgentQL query. AgentQL can be used across multiple languages and web pages without breaking over time and change.

​
Overview

AgentQLLoader requires the following two parameters: Setting the following parameters are optional:
  • api_key: Your AgentQL API key from dev.agentql.com. Optional.
  • timeout: The number of seconds to wait for a request before timing out. Defaults to 900.
  • is_stealth_mode_enabled: Whether to enable experimental anti-bot evasion strategies. This feature may not work for all websites at all times. Data extraction may take longer to complete with this mode enabled. Defaults to False.
  • wait_for: The number of seconds to wait for the page to load before extracting data. Defaults to 0.
  • is_scroll_to_bottom_enabled: Whether to scroll to bottom of the page before extracting data. Defaults to False.
  • mode: "standard" uses deep data analysis, while "fast" trades some depth of analysis for speed and is adequate for most usecases. Learn more about the modes in this guide. Defaults to "fast".
  • is_screenshot_enabled: Whether to take a screenshot before extracting data. Returned in ‘metadata’ as a Base64 string. Defaults to False.
AgentQLLoader is implemented with AgentQL’s REST API

​
Integration details

ClassPackageLocalSerializableJS support
AgentQLLoaderlangchain-agentql✅❌❌

​
Loader features

SourceDocument Lazy LoadingNative Async Support
AgentQLLoader✅❌

​
Setup

To use the AgentQL Document Loader, you will need to configure the AGENTQL_API_KEY environment variable, or use the api_key parameter. You can acquire an API key from our Dev Portal.

​
Installation

Install langchain-agentql. 
%pip install -qU langchain-agentql

​
Set Credentials

import os

os.environ["AGENTQL_API_KEY"] = "YOUR_AGENTQL_API_KEY"

​
Initialization

Next instantiate your model object: 
from langchain_agentql.document_loaders import AgentQLLoader

loader = AgentQLLoader(
    url="https://www.agentql.com/blog",
    query="""
    {
        posts[] {
            title
            url
            date
            author
        }
    }
    """,
    is_scroll_to_bottom_enabled=True,
)

​
Load

docs = loader.load()
docs[0]

Document(metadata={'request_id': 'bdb9dbe7-8a7f-427f-bc16-839ccc02cae6', 'generated_query': None, 'screenshot': None}, page_content="{'posts': [{'title': 'Launch Week Recap—make the web AI-ready', 'url': 'https://www.agentql.com/blog/2024-launch-week-recap', 'date': 'Nov 18, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Accurate data extraction from PDFs and images with AgentQL', 'url': 'https://www.agentql.com/blog/accurate-data-extraction-pdfs-images', 'date': 'Feb 1, 2025', 'author': 'Rachel-Lee Nabors'}, {'title': 'Introducing Scheduled Scraping Workflows', 'url': 'https://www.agentql.com/blog/scheduling', 'date': 'Dec 2, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Updates to Our Pricing Model', 'url': 'https://www.agentql.com/blog/2024-pricing-update', 'date': 'Nov 19, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Get data from any page: AgentQL’s REST API Endpoint—Launch week day 5', 'url': 'https://www.agentql.com/blog/data-rest-api', 'date': 'Nov 15, 2024', 'author': 'Rachel-Lee Nabors'}]}")

print(docs[0].metadata)

{'request_id': 'bdb9dbe7-8a7f-427f-bc16-839ccc02cae6', 'generated_query': None, 'screenshot': None}

​
Lazy Load

AgentQLLoader currently only loads one Document at a time. Therefore, load() and lazy_load() behave the same: 
pages = [doc for doc in loader.lazy_load()]
pages

[Document(metadata={'request_id': '06273abd-b2ef-4e15-b0ec-901cba7b4825', 'generated_query': None, 'screenshot': None}, page_content="{'posts': [{'title': 'Launch Week Recap—make the web AI-ready', 'url': 'https://www.agentql.com/blog/2024-launch-week-recap', 'date': 'Nov 18, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Accurate data extraction from PDFs and images with AgentQL', 'url': 'https://www.agentql.com/blog/accurate-data-extraction-pdfs-images', 'date': 'Feb 1, 2025', 'author': 'Rachel-Lee Nabors'}, {'title': 'Introducing Scheduled Scraping Workflows', 'url': 'https://www.agentql.com/blog/scheduling', 'date': 'Dec 2, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Updates to Our Pricing Model', 'url': 'https://www.agentql.com/blog/2024-pricing-update', 'date': 'Nov 19, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Get data from any page: AgentQL’s REST API Endpoint—Launch week day 5', 'url': 'https://www.agentql.com/blog/data-rest-api', 'date': 'Nov 15, 2024', 'author': 'Rachel-Lee Nabors'}]}")]

​
API reference

For more information on how to use this integration, please refer to the git repo or the langchain integration documentation