findTextChunkContext

Find text chunk context

Since R2026a

Syntax

idxContext = findTextChunkContext(chunkTable,idx)

idxContext = findTextChunkContext(chunkTable,Name=Value)

Description

Use findTextChunkContext to identify context text chunks surrounding target text chunks.

The surrounding text chunks can provide useful context for the target chunks. For example, in retrieval-augmented generation (RAG), after you have retrieved the relevant text chunks, you can provide additional context to a large language model by adding text chunks surrounding the retrieved chunks.

idxContext = findTextChunkContext(chunkTable,idx) finds text chunks surrounding the target text chunks at chunkTable(idx,:).

idxContext = findTextChunkContext(chunkTable,Name=Value) specifies additional options using one or more name-value arguments.

Examples

Find Text Chunk Context

Load the example data. The file sonnets.txt contains Shakespeare's sonnets in plain text. Extract the text from sonnets.txt using the extractFileText function.

str = extractFileText("sonnets.txt");

Split str into text chunks using the splitTextChunks function. Specify the target length as 100.

chunkTable = splitTextChunks(str,TargetLength=100);
summary(chunkTable)

chunkTable: 1277×1 table

Variables:

    Text: string

Statistics for applicable variables:

            NumMissing

    Text        0     

Specify the text chunk index.

idx = 10;

Specify a target length of 300 characters. The findTextChunkContext function returns the indices of the target text chunk and surrounding context chunks, such that the total length of the target chunk plus the context chunks is 300 characters or less.

findTextChunkContext(chunkTable,idx,TargetLength=300)

ans = 1×4

     8     9    10    11

Input Arguments

chunkTable — Input table of text chunks

table

Input table of text chunks. chunkTable must contain a variable named Text, specified as a string scalar that contains the text chunks.

Create a table of text chunks from a document or table of documents by using the splitTextChunks, splitHTMLSections, splitMarkdownSections, or splitMarkdownSections function.

idx — Index of text chunk

positive integer | array of positive integers

Index of the text chunk, or indices of the text chunks, for which to find context, specified as a positive integer or array of positive integers.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: findTextChunkContext(chunkTable,idx,TargetLength=100) sets the total target length of the output text chunks to 100.

TargetLength — Total target length

4000 (default) | positive integer

Total target length of output text chunks, specified as a positive integer that represents the number of characters.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64
Complex Number Support: Yes

Levels — Section levels

string array | character vector | cell array of character vectors

Section levels within which to find text chunk context, specified as a string array, character vector, or cell array of character vectors.

If you specify this argument, then the function only identifies text chunks as contextual if they are part of the same section, that is, if they have all of the same entries as the target text chunk in the columns specified by Levels.

If you do not specify this argument, then the function uses every table variable except for Text as section levels. For example, if a table has variables Text, H1, and H2, then by default, the function uses Levels = ["H1","H2"].

If chunkTable does not have a variable specified by Levels, then findTextChunkContext ignores that variable. For example, if you specify Labels = ["H1","H7"] and there is no column header named "H7", then the function uses Levels = "H1".

Example: ["H1","H2"]

Data Types: char | string | cell

Output Arguments

idxContext — Indices of context text chunks

positive integer | array of positive integers

Indices of context text chunks and the target chunk or chunks, returned as a positive integer or as an array of positive integers.

More About

Text Chunks

Many analysis tools, including large language models (LLMs), perform better on small chunks of text than on large documents. Text Analytics Toolbox™ includes a range of functions that allow you to split large documents into semantically meaningful chunks.

Basic Workflow

The splitTextChunks function splits a document recursively into text chunks of a given target length. The function first splits a document into paragraphs. If any of the paragraphs are longer than the target length, then the function splits those paragraphs into sentences, and so on.

chunks = splitTextChunks(str);

Advanced Workflow

1. Split your document into sections and preserve the section metadata using one of these functions:

   splitHTMLSections	Split an HTML-formatted document into HTML sections according to the section tags <h1>...</h1>, <h2>...</h2>, …, <h6>...</h6>.
   splitMarkdownSections	Split a Markdown-formatted document into Markdown sections, for example according to ATX section tags # , ## , …, ###### .
   splitCustomSections	Split a document into custom sections according to custom section delimiters.

2. Split your documents or your chunks recursively into paragraphs, sentences, and tokens using the splitTextChunks function.

3. To avoid redundancy, join similar adjacent chunks using the joinSimilarTextChunks function.

4. Add overlap between adjacent text chunks using the addTextChunkOverlap function. Adding text chunk overlap avoids changing the meaning of sentences by splitting at inopportune points, for example, splitting the sentence "I would never say I love cats" into "I would never say" and "I love cats." Adding overlap in this example results in the two chunks "I would never say I love" and "never say I love cats." You can also add surrounding text to individual chunks as context by using the findTextChunkContext function.

For an example showing the advanced workflow, see Split Document Into Semantically Meaningful Text Chunks.

Retrieval-Augmented Generation (RAG)

RAG combines the text generation capabilities of large language models (LLMs) with reliable information contained in a set of source documents. First, retrieve documents relevant to the user prompt from the set of source documents. Then, append the relevant document to the prompt and use the LLM to generate a response.

1. To improve the quality of the generated output, split large documents into smaller, semantically meaningful chunks.

2. Use information retrieval to identify the text chunks that are relevant to the query. For more information, see Information Retrieval with Document Embeddings.

3. Create a prompt based on the most relevant chunks. To provide the LLM with additional context, you can add text from adjacent prompts within the same section by using the findTextChunkContext function, or you can you can add overlap between text chunks before information retrieval by using the addTextChunkOverlap function. Create a Markdown-formatted string from the text chunks using the formatTextChunks function. For an example, see Create Large Language Model (LLM) Prompt from Text Chunk.

4. Generate an answer using an LLM. To connect to large language model APIs using MATLAB, use the Large Language Models (LLMs) with MATLAB add-on.

Version History

Introduced in R2026a