Recursive Splitter Processor

The Recursive Splitter processor takes text document as input and splits the text progressively into smaller chunks using a list of separators.

This approach allows handling various levels of granularity, ensuring the resulting chunks fit within size constraints while preserving context.

Below are the configuration details of the processor.

Drop Non-Chunked Columns

All columns except for the chunked ones will be dropped from the output.

Select the column containing the text you want to split.

Specify the name of the column where the split text chunks will be stored.

Type

Choose whether to define a separator as a string or regex.

Value

Enter your custom separator value.

👉

In Recursive Splitter processor, you can pass multiple separators.

Check the box if you want to include the separators in the output chunks.

Size

Set the maximum number of characters for each chunk.

Overlap

Define the number of characters to repeat between consecutive chunks.

👉

Up to five records will be shown at design time. All records will be available at runtime.

option to add further configurations in key-value pair is available.

If you have any feedback on Gathr documentation, please email us!