Skip to main content
Petals runs 100B+ language models at home, BitTorrent-style. This notebook goes over how to use LangChain with Petals.

​
Install petals

The petals package is required to use the Petals API. Install petals using pip3 install petals. For Apple Silicon(M1/M2) users please follow this guide https://github.com/bigscience-workshop/petals/issues/147#issuecomment-1365379642 to install petals 
!pip3 install petals

​
Imports

import os

from langchain_classic.chains import LLMChain
from langchain_community.llms import Petals
from langchain_core.prompts import PromptTemplate

​
Set the environment API Key

Make sure to get your API key from Huggingface. 
from getpass import getpass

HUGGINGFACE_API_KEY = getpass()

 ········

os.environ["HUGGINGFACE_API_KEY"] = HUGGINGFACE_API_KEY

​
Create the petals instance

You can specify different parameters such as the model name, max new tokens, temperature, etc. 
# this can take several minutes to download big files!

llm = Petals(model_name="bigscience/bloom-petals")

Downloading:   1%|▏                        | 40.8M/7.19G [00:24<15:44, 7.57MB/s]

​
Create a Prompt template

We will create a prompt template for Question and Answer. 
template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate.from_template(template)

​
Initiate the LLMChain

llm_chain = LLMChain(prompt=prompt, llm=llm)

​
Run the LLMChain

Provide a question and run the LLMChain. 
question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"

llm_chain.run(question)
Edit this page on GitHub or file an issue.
Connect these docs to Claude, VSCode, and more via MCP for real-time answers.