Nimble Extract

import getpass
import os

if not os.environ.get("NIMBLE_API_KEY"):
    os.environ["NIMBLE_API_KEY"] = getpass.getpass("Nimble API key:\n")

from langchain_nimble import NimbleExtractRetriever

# Basic retriever - requires URLs to extract
retriever = NimbleExtractRetriever()

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

# Create a RAG prompt
prompt = ChatPromptTemplate.from_template(
    """Analyze the extracted content from the provided URLs.
Answer the question based only on the extracted content.
If you cannot answer based on the content, say so.

Content: {content}

Question: {question}

Answer:"""
)

llm = ChatOpenAI(model="gpt-4o-mini")

# Configure retriever for content extraction
retriever = NimbleExtractRetriever(
    parsing_type="markdown",
    wait=3000
)


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


# Example: Extract and analyze content from LangChain documentation
urls = [
    "https://python.langchain.com/docs/concepts/retrievers/",
    "https://python.langchain.com/docs/concepts/tools/",
    "https://python.langchain.com/docs/tutorials/agents/"
]

# Create a custom runnable that passes URLs to retriever
def get_docs(question):
    # In a real scenario, URLs might be determined by previous steps
    return retriever.invoke(urls)


# Build the RAG chain
chain = (
    {
        "content": lambda _: get_docs(_) | format_docs,
        "question": RunnablePassthrough()
    }
    | prompt
    | llm
    | StrOutputParser()
)

# Ask a question about the extracted content
response = chain.invoke("What are the key differences between retrievers and tools in LangChain?")
print(response)

Based on the extracted LangChain documentation, here are the key differences:

**Retrievers:**
- Interface for document retrieval based on unstructured queries
- Primary use case is RAG (Retrieval Augmented Generation)
- Returns documents from various sources like vector stores
- Focuses on semantic search and information retrieval
- Core component for question-answering systems

**Tools:**
- Interface for agents to interact with external systems
- Enables actions beyond text generation (API calls, calculations, web search)
- Used by agents to extend capabilities dynamically
- Supports both synchronous and asynchronous execution
- Can be chained together for complex workflows

**Agents:**
- High-level orchestrators that use tools to accomplish tasks
- Make decisions about which tools to use and when
- Can combine multiple tools to solve complex problems
- Tutorial shows how to build agent workflows with tool integration

The documentation emphasizes that retrievers are specialized for information retrieval, while tools provide broader action capabilities for agents.

from langchain_nimble import NimbleExtractRetriever

# Retriever optimized for JavaScript-heavy documentation sites
retriever = NimbleExtractRetriever(
    parsing_type="markdown",
    driver="vx10",  # Use comprehensive driver for complex SPAs
    wait=5000,  # Wait up to 5 seconds for full page render
    render=True,  # Enable JavaScript rendering
    locale="en-US",
    country="US"
)

# Extract content from specific LangChain documentation pages
docs = retriever.invoke([
    "https://python.langchain.com/docs/concepts/chat_models/",
    "https://python.langchain.com/docs/concepts/prompts/"
])