Splitting code

@[RecursiveCharacterTextSplitter] includes pre-built lists of separators that are useful for splitting text in a specific programming language. Supported languages are kept in the SupportedTextSplitterLanguages type. They include:

"cpp",
"go",
"java",
"js",
"php",
"proto",
"python",
"rst",
"ruby",
"rust",
"scala",
"swift",
"markdown",
"latex",
"html",
"sol",

To view the list of separators for a given language, pass a value from this enum into

RecursiveCharacterTextSplitter.getSeparatorsForLanguage()

To instantiate a splitter that is tailored for a specific language, pass a value from the enum into

RecursiveCharacterTextSplitter.fromLanguage()

Below we demonstrate examples for the various languages.

npm install @langchain/textsplitters

Python

Here’s an example using the python text splitter:

const pythonSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "python",
    { chunkSize: 50, chunkOverlap: 0 }
);
const pythonDocs = pythonSplitter.createDocuments([{ pageContent: PYTHON_CODE }]);
console.log(pythonDocs);

[
    Document { metadata: {}, pageContent: 'def hello_world():\n    print("Hello, World!")' },
    Document { metadata: {}, pageContent: '# Call the function\nhello_world()' }
]

JS

Here’s an example using the JS text splitter:

const JS_CODE = `
function helloWorld() {
  console.log("Hello, World!");
}

// Call the function
helloWorld();
`;

const jsSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "js",
    { chunkSize: 60, chunkOverlap: 0 }
);
const jsDocs = jsSplitter.createDocuments([{ pageContent: JS_CODE }]);
console.log(jsDocs);

[
    Document { metadata: {}, pageContent: 'function helloWorld() {\n  console.log("Hello, World!");\n}' },
    Document { metadata: {}, pageContent: '// Call the function\nhelloWorld()' }
]

TS

Here’s an example using the typescript text splitter:

const TS_CODE = `
function helloWorld(): void {
  console.log("Hello, World!");
}

// Call the function
helloWorld();
`;

const tsSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "ts",
    { chunkSize: 60, chunkOverlap: 0 }
);
const tsDocs = tsSplitter.createDocuments([{ pageContent: TS_CODE }]);
console.log(tsDocs);

[
    Document { metadata: {}, pageContent: 'function helloWorld(): void {' },
    Document { metadata: {}, pageContent: 'console.log("Hello, World!");\n}' },
    Document { metadata: {}, pageContent: '// Call the function\nhelloWorld()' }
]

Markdown

Here’s an example using the Markdown text splitter:

const markdownText = `
# 🦜️🔗 LangChain

⚡ Building applications with LLMs through composability ⚡

## What is LangChain?

# Hopefully this code block isn't split
LangChain is a framework for...

As an open-source project in a rapidly developing field, we are extremely open to contributions.
`;

const mdSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "markdown",
    { chunkSize: 60, chunkOverlap: 0 }
);
const mdDocs = mdSplitter.createDocuments([{ pageContent: markdownText }]);
console.log(mdDocs);

[
    Document { metadata: {}, pageContent: '# 🦜️🔗 LangChain' },
    Document { metadata: {}, pageContent: '⚡ Building applications with LLMs through composability ⚡' },
    Document { metadata: {}, pageContent: '## What is LangChain?' },
    Document { metadata: {}, pageContent: "# Hopefully this code block isn't split" },
    Document { metadata: {}, pageContent: 'LangChain is a framework for...' },
    Document { metadata: {}, pageContent: 'As an open-source project in a rapidly developing field, we' },
    Document { metadata: {}, pageContent: 'are extremely open to contributions.' }
]

Latex

Here’s an example on Latex text:

const latexText = `
\\documentclass{article}

\\begin{document}

\\maketitle

\\section{Introduction}
Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.

\\subsection{History of LLMs}
The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.

\\subsection{Applications of LLMs}
LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.

\\end{document}
`;

const latexSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "latex",
    { chunkSize: 60, chunkOverlap: 0 }
);
const latexDocs = latexSplitter.createDocuments([{ pageContent: latexText }]);
console.log(latexDocs);

[
    Document { metadata: {}, pageContent: '\\documentclass{article}\n\n\\begin{document}\n\n\\maketitle' },
    Document { metadata: {}, pageContent: '\\section{Introduction}' },
    Document { metadata: {}, pageContent: 'Large language models (LLMs) are a type of machine learning' },
    Document { metadata: {}, pageContent: 'model that can be trained on vast amounts of text data to' },
    Document { metadata: {}, pageContent: 'generate human-like language. In recent years, LLMs have' },
    Document { metadata: {}, pageContent: 'made significant advances in a variety of natural language' },
    Document { metadata: {}, pageContent: 'processing tasks, including language translation, text' },
    Document { metadata: {}, pageContent: 'generation, and sentiment analysis.' },
    Document { metadata: {}, pageContent: '\\subsection{History of LLMs}' },
    Document { metadata: {}, pageContent: 'The earliest LLMs were developed in the 1980s and 1990s,' },
    Document { metadata: {}, pageContent: 'but they were limited by the amount of data that could be' },
    Document { metadata: {}, pageContent: 'processed and the computational power available at the' },
    Document { metadata: {}, pageContent: 'time. In the past decade, however, advances in hardware and' },
    Document { metadata: {}, pageContent: 'software have made it possible to train LLMs on massive' },
    Document { metadata: {}, pageContent: 'datasets, leading to significant improvements in' },
    Document { metadata: {}, pageContent: 'performance.' },
    Document { metadata: {}, pageContent: '\\subsection{Applications of LLMs}' },
    Document { metadata: {}, pageContent: 'LLMs have many applications in industry, including' },
    Document { metadata: {}, pageContent: 'chatbots, content creation, and virtual assistants. They' },
    Document { metadata: {}, pageContent: 'can also be used in academia for research in linguistics,' },
    Document { metadata: {}, pageContent: 'psychology, and computational linguistics.' },
    Document { metadata: {}, pageContent: '\\end{document}' }
]

HTML

Here’s an example using an HTML text splitter:

const htmlText = `
<!DOCTYPE html>
<html>
    <head>
        <title>🦜️🔗 LangChain</title>
        <style>
            body {
                font-family: Arial, sans-serif;
            }
            h1 {
                color: darkblue;
            }
        </style>
    </head>
    <body>
        <div>
            <h1>🦜️🔗 LangChain</h1>
            <p>⚡ Building applications with LLMs through composability ⚡</p>
        </div>
        <div>
            As an open-source project in a rapidly developing field, we are extremely open to contributions.
        </div>
    </body>
</html>
`;

const htmlSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "html",
    { chunkSize: 60, chunkOverlap: 0 }
);
const htmlDocs = htmlSplitter.createDocuments([{ pageContent: htmlText }]);
console.log(htmlDocs);

[
    Document { metadata: {}, pageContent: '<!DOCTYPE html>\n<html>' },
    Document { metadata: {}, pageContent: '<head>\n        <title>🦜️🔗 LangChain</title>' },
    Document { metadata: {}, pageContent: '<style>\n            body {\n                font-family: Aria' },
    Document { metadata: {}, pageContent: 'l, sans-serif;\n            }\n            h1 {' },
    Document { metadata: {}, pageContent: 'color: darkblue;\n            }\n        </style>\n    </head' },
    Document { metadata: {}, pageContent: '>' },
    Document { metadata: {}, pageContent: '<body>' },
    Document { metadata: {}, pageContent: '<div>\n            <h1>🦜️🔗 LangChain</h1>' },
    Document { metadata: {}, pageContent: '<p>⚡ Building applications with LLMs through composability ⚡' },
    Document { metadata: {}, pageContent: '</p>\n        </div>' },
    Document { metadata: {}, pageContent: '<div>\n            As an open-source project in a rapidly dev' },
    Document { metadata: {}, pageContent: 'eloping field, we are extremely open to contributions.' },
    Document { metadata: {}, pageContent: '</div>\n    </body>\n</html>' }
]

Solidity

Here’s an example using the Solidity text splitter:

const SOL_CODE = `
pragma solidity ^0.8.20;
contract HelloWorld {
   function add(uint a, uint b) pure public returns(uint) {
       return a + b;
   }
}
`;

const solSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "sol",
    { chunkSize: 128, chunkOverlap: 0 }
);
const solDocs = solSplitter.createDocuments([{ pageContent: SOL_CODE }]);
console.log(solDocs);

[
    Document { metadata: {}, pageContent: 'pragma solidity ^0.8.20;' },
    Document { metadata: {}, pageContent: 'contract HelloWorld {\n   function add(uint a, uint b) pure public returns(uint) {\n       return a + b;\n   }\n}' }
]

C#

Here’s an example using the C# text splitter:

const C_CODE = `
using System;
class Program
{
    static void Main()
    {
        int age = 30; // Change the age value as needed

        // Categorize the age without any console output
        if (age < 18)
        {
            // Age is under 18
        }
        else if (age >= 18 && age < 65)
        {
            // Age is an adult
        }
        else
        {
            // Age is a senior citizen
        }
    }
}
`;

const csharpSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "csharp",
    { chunkSize: 128, chunkOverlap: 0 }
);
const csharpDocs = csharpSplitter.createDocuments([{ pageContent: C_CODE }]);
console.log(csharpDocs);

[
    Document { metadata: {}, pageContent: 'using System;' },
    Document { metadata: {}, pageContent: 'class Program\n{\n    static void Main()\n    {\n        int age = 30; // Change the age value as needed' },
    Document { metadata: {}, pageContent: '// Categorize the age without any console output\n        if (age < 18)\n        {\n            // Age is under 18' },
    Document { metadata: {}, pageContent: '}\n        else if (age >= 18 && age < 65)\n        {\n            // Age is an adult\n        }\n        else\n        {' },
    Document { metadata: {}, pageContent: '// Age is a senior citizen\n        }\n    }\n}' }
]

Haskell

Here’s an example using the Haskell text splitter:

const HASKELL_CODE = `
main :: IO ()
main = do
    putStrLn "Hello, World!"
-- Some sample functions
add :: Int -> Int -> Int
add x y = x + y
`;

const haskellSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "haskell",
    { chunkSize: 50, chunkOverlap: 0 }
);
const haskellDocs = haskellSplitter.createDocuments([{ pageContent: HASKELL_CODE }]);
console.log(haskellDocs);

[
    Document { metadata: {}, pageContent: 'main :: IO ()' },
    Document { metadata: {}, pageContent: 'main = do\n    putStrLn "Hello, World!"\n-- Some' },
    Document { metadata: {}, pageContent: 'sample functions\nadd :: Int -> Int -> Int\nadd x y' },
    Document { metadata: {}, pageContent: '= x + y' }
]

PHP

Here’s an example using the PHP text splitter:

const PHP_CODE = `<?php
namespace foo;
class Hello {
    public function __construct() { }
}
function hello() {
    echo "Hello World!";
}
interface Human {
    public function breath();
}
trait Foo { }
enum Color
{
    case Red;
    case Blue;
}`;

const phpSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "php",
    { chunkSize: 50, chunkOverlap: 0 }
);
const phpDocs = phpSplitter.createDocuments([{ pageContent: PHP_CODE }]);
console.log(phpDocs);

[
    Document { metadata: {}, pageContent: '<?php\nnamespace foo;' },
    Document { metadata: {}, pageContent: 'class Hello {' },
    Document { metadata: {}, pageContent: 'public function __construct() { }\n}' },
    Document { metadata: {}, pageContent: 'function hello() {\n    echo "Hello World!";\n}' },
    Document { metadata: {}, pageContent: 'interface Human {\n    public function breath();\n}' },
    Document { metadata: {}, pageContent: 'trait Foo { }\nenum Color\n{\n    case Red;' },
    Document { metadata: {}, pageContent: 'case Blue;\n}' }
]

PowerShell

Here’s an example using the PowerShell text splitter:

const POWERSHELL_CODE = `
$directoryPath = Get-Location

$items = Get-ChildItem -Path $directoryPath

$files = $items | Where-Object { -not $_.PSIsContainer }

$sortedFiles = $files | Sort-Object LastWriteTime

foreach ($file in $sortedFiles) {
    Write-Output ("Name: " + $file.Name + " | Last Write Time: " + $file.LastWriteTime)
}
`;

const powershellSplitter = RecursiveCharacterTextSplitter.fromLanguage(
    "powershell",
    { chunkSize: 100, chunkOverlap: 0 }
);
const powershellDocs = powershellSplitter.createDocuments([{ pageContent: POWERSHELL_CODE }]);
console.log(powershellDocs);

[
    Document { metadata: {}, pageContent: '$directoryPath = Get-Location\n\n$items = Get-ChildItem -Path $directoryPath' },
    Document { metadata: {}, pageContent: '$files = $items | Where-Object { -not $_.PSIsContainer }' },
    Document { metadata: {}, pageContent: '$sortedFiles = $files | Sort-Object LastWriteTime' },
    Document { metadata: {}, pageContent: 'foreach ($file in $sortedFiles) {' },
    Document { metadata: {}, pageContent: 'Write-Output ("Name: " + $file.Name + " | Last Write Time: " + $file.LastWriteTime)\n}' }
]

Visual Basic 6

const VISUALBASIC6_CODE = `Option Explicit

Public Sub HelloWorld()
    MsgBox "Hello, World!"
End Sub

Private Function Add(a As Integer, b As Integer) As Integer
    Add = a + b
End Function
`;

const visualbasic6Splitter = RecursiveCharacterTextSplitter.fromLanguage(
    "visualbasic6",
    { chunkSize: 128, chunkOverlap: 0 }
);
const visualbasic6Docs = visualbasic6Splitter.createDocuments([{ pageContent: VISUALBASIC6_CODE }]);
console.log(visualbasic6Docs);

[
    Document { metadata: {}, pageContent: 'Option Explicit' },
    Document { metadata: {}, pageContent: 'Public Sub HelloWorld()\n    MsgBox "Hello, World!"\nEnd Sub' },
    Document { metadata: {}, pageContent: 'Private Function Add(a As Integer, b As Integer) As Integer\n    Add = a + b\nEnd Function' }
]

Edit the source of this page on GitHub

Contribute

Integrations

Python

JS

TS

Markdown

Latex

HTML

Solidity

C#

Haskell

PHP

PowerShell

Visual Basic 6

Contribute

Integrations

​Python

​JS

​TS

​Markdown

​Latex

​HTML

​Solidity

​C#

​Haskell

​PHP

​PowerShell

​Visual Basic 6

Python

JS

TS

Markdown

Latex

HTML

Solidity

C#

Haskell

PHP

PowerShell

Visual Basic 6