license: apache-2.0
configs:

• config_name: recipes
  data_files: recipes/*.json
• config_name: schemas
  data_files: "*.schema.json"

This repository contains all the recipes that you can use with Chonkie to manage various documents, languages, and more.

Usage

To use the recipes, you need to install chonkie with the hub feature, with the following command:

pip install "chonkie[hub]"

This would enable Hubie which is used internally to get the recipes from this repository. So, you can do things like use the from_recipe method to create a RecursiveChunker from a recipe.

from chonkie import RecursiveChunker

chunker = RecursiveChunker.from_recipe("markdown")
with open("example.md", "r") as f:
    text = f.read()
chunks = chunker.chunk(text)
for chunk in chunks:
    print(chunk)

The above code would read the example.md file and chunk it into smaller chunks using the markdown recipe.

Recipes

The recipes are stored in the recipes folder. Each recipe is a JSON file that contains the rules for the recipe.

{
    "name": "markdown",
    "description": "Recipe for markdown documents in English",
    "language": "en",
    "metadata": {
        "version": "0.1.0",
        "author": "Chonkie Team"
    },
    "recipe": {
        "delimiters": [".", "!", "?", "\n"],
        "include_delim": "prev",
        "recursive_rules": [
            {
                "delimiters": ["######", "#####", "####", "###", "##", "#"],
                "include_delim": "next",
                "whitespace": false
            },
            {
                "delimiters": ["\n\n", "\n\r"],
                "include_delim": "next",
                "whitespace": false
            },
            ...
        ]
    }
}

The full JSON schema for the recipes is available here. You can use this schema to validate the recipes before using them.

Contributing

If you want to contribute to this repository, you can do so by creating a pull request. Please follow the guidelines for the recipes and the schema.

License

This project is licensed under the Apache 2.0 License.