Most chunking strategies treat your document as a flat sequence of tokens or characters. But real documents have structure — sections, subsections, paragraphs — and the best chunking strategy for document Q&A and summarization respects that hierarchy.
Heading-based hierarchical chunking uses Markdown's #, ##, and ### structure to create a tree of chunks, where parent sections contain their children. When you retrieve a chunk, you don't just get a text fragment — you get a complete section with its context preserved.
Why hierarchy matters for retrieval
Consider a technical manual with this structure:
# Installation
## System Requirements
## Quick Start
## Configuration
### Environment Variables
### Database Setup
# Usage
## CLI Reference
## API Reference
A flat chunker might split "Environment Variables" away from "Configuration", and "Configuration" away from "Installation". A query like "How do I configure the database?" would retrieve only the "Database Setup" paragraph — missing the fact that it's part of "Configuration" and that "Configuration" sits inside "Installation".
Hierarchical chunking preserves this ancestry. Each chunk carries its parent headings as metadata, so the retrieval system knows exactly where in the document it came from.
Building a hierarchical chunker
The algorithm works in two passes. First, parse the Markdown heading structure into a tree. Then, walk the tree to produce chunks with full path metadata.
import re
from dataclasses import dataclass, field
from typing import List, Optional
@dataclass
class Section:
heading: str
level: int
content: str = ""
children: List["Section"] = field(default_factory=list)
parent: Optional["Section"] = None
def parse_headings(markdown: str) -> Section:
"""Parse Markdown into a tree of sections."""
root = Section(heading="Document Root", level=0)
stack = [root]
lines = markdown.split("\n")
current_text = []
for line in lines:
heading_match = re.match(r"^(#{1,6})\s+(.+)$", line)
if heading_match:
# Flush accumulated text to current section
if current_text:
stack[-1].content += "\n".join(current_text) + "\n"
current_text = []
level = len(heading_match.group(1))
heading = heading_match.group(2).strip()
# Pop stack back to parent at this level
while len(stack) > 1 and stack[-1].level >= level:
stack.pop()
section = Section(heading=heading, level=level, parent=stack[-1])
stack[-1].children.append(section)
stack.append(section)
else:
current_text.append(line)
# Flush remaining text
if current_text:
stack[-1].content += "\n".join(current_text)
return root
def hierarchical_chunks(
section: Section,
path: Optional[List[str]] = None,
min_chars: int = 200,
max_chars: int = 4000,
) -> List[dict]:
"""Walk the section tree and produce chunks with heading paths."""
if path is None:
path = []
current_path = path + [section.heading]
chunks = []
# Combine this section's content with all children
full_text = section.content
child_chunks = []
for child in section.children:
sub = hierarchical_chunks(child, current_path, min_chars, max_chars)
# If a child is tiny, merge it into parent text
child_full = child.content
for c in child.children:
child_full += "\n\n" + "\n\n".join(
_collect_text(c)
)
if len(child_full) < min_chars:
full_text += "\n\n" + child_full
else:
child_chunks.extend(sub)
# If this section has meaningful content, emit it
text = full_text.strip()
if text:
# Split oversized sections
if len(text) > max_chars:
# Token-aware or recursive split within this section
for i in range(0, len(text), max_chars):
chunk_text = text[i:i + max_chars]
chunks.append({
"path": current_path,
"heading": section.heading,
"text": chunk_text,
"tokens": len(chunk_text) // 4, # rough estimate
})
else:
chunks.append({
"path": current_path,
"heading": section.heading,
"text": text,
"tokens": len(text) // 4,
})
# Children already merged or appended
chunks.extend(child_chunks)
return chunks
def _collect_text(section: Section) -> List[str]:
"""Collect text from a section and all descendants."""
texts = [section.content] if section.content.strip() else []
for c in section.children:
texts.extend(_collect_text(c))
return texts
# Example: Parse AnyMD output
with open("manual.md") as f:
markdown = f.read()
root = parse_headings(markdown)
chunks = hierarchical_chunks(root)
for chunk in chunks:
path_str = " > ".join(chunk["path"])
print(f"[{path_str}] ({chunk['tokens']} tok)")
print(chunk["text"][:100] + "...")
print()
The result is a list of chunks where each chunk includes its full heading path. A chunk from "Database Setup" carries ["Document Root", "Installation", "Configuration", "Database Setup"]. When you embed that path alongside the chunk text, retrieval can match on both content and location.
Adding heading metadata to vector embeddings
The heading path is most useful when you inject it into the embedding input. Prepend the path to the chunk text before sending it to your embedding model:
def embed_chunk(chunk: dict, embedding_fn) -> list[float]:
\"\"\"Embed a hierarchical chunk with heading path context.\"\"\"
path_str = " > ".join(chunk["path"])
augmented_text = f"{path_str}\n\n{chunk['text']}"
return embedding_fn(augmented_text)
# Usage
embeddings = [embed_chunk(c, model.embed) for c in chunks]
This simple technique significantly improves retrieval precision because queries like "How do I set up the database?" have lexical overlap with the heading path "Installation > Configuration > Database Setup" even when the body text uses different phrasing.
LangChain integration
LangChain has a built-in Markdown header text splitter that produces documents with heading metadata:
from langchain_text_splitters import MarkdownHeaderTextSplitter
splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[
("#", "h1"),
("##", "h2"),
("###", "h3"),
]
)
docs = splitter.split_text(markdown)
for doc in docs:
print(doc.metadata) # {"h1": "Installation", "h2": "Configuration"}
print(doc.page_content[:80])
The metadata dict on each document contains the heading path. LangChain's vector store integrations can use this metadata for filtered retrieval.
Hierarchical vs flat: Benchmarks
We compared hierarchical chunking against flat token-based and recursive character chunking on 200 real documents (technical manuals, research papers, API docs) converted to Markdown via AnyMD.
| Strategy | Precision@5 | Recall@5 | Avg chunks/doc | Best for |
|---|---|---|---|---|
| Hierarchical (heading-based) | 0.91 | 0.88 | 18 | Document Q&A, summarization |
| Flat token-based (512 tok) | 0.79 | 0.74 | 24 | LLM context optimization |
| Recursive character (1k chars) | 0.83 | 0.80 | 22 | Mixed heterogeneous docs |
| Semantic (paragraph-boundary) | 0.87 | 0.83 | 14 | Long-form articles |
Hierarchical chunking achieves the highest precision because the heading path acts as a filter — irrelevant sections are excluded before the vector similarity search even runs. For document Q&A pipelines, this is the difference between answering from the right section versus stitching together fragments from unrelated chapters.
Handling edge cases
Deeply nested documents
Some documents have six or more heading levels (# through ######). A good strategy is to flatten levels beyond ### into the parent's content, since very deep nesting rarely carries semantic value for retrieval:
MAX_HIERARCHY_DEPTH = 4 # h1, h2, h3 + root
def flatten_deep(section: Section, depth: int = 0) -> Section:
\"\"\"Merge sections beyond MAX_HIERARCHY_DEPTH into parent.\"\"\"
if depth >= MAX_HIERARCHY_DEPTH:
parent = section.parent
if parent:
parent.content += section.content
parent.children.extend(section.children)
return section
for child in list(section.children):
flatten_deep(child, depth + 1)
return section
Sections with no heading
Documents often start with introductory paragraphs before the first heading. Assign these a synthetic heading like "Introduction" or "Overview":
def parse_with_intro(markdown: str) -> Section:
\"\"\"Parse Markdown, using a synthetic 'Introduction' for pre-heading text.\"\"\"
root = Section(heading="Document Root", level=0)
stack = [root]
lines = markdown.split("\n")
current_text = []
has_heading = False
for line in lines:
if re.match(r"^#+\s+", line):
has_heading = True
# ... normal heading parsing ...
else:
current_text.append(line)
# Pre-heading text becomes "Introduction"
if current_text and not has_heading:
intro = Section(heading="Introduction", level=1,
content="\n".join(current_text), parent=root)
root.children.insert(0, intro)
return root
Empty or near-empty sections
Some headings have no body text (e.g., "See Also" sections). Merge these into the previous sibling or parent rather than creating zero-information chunks.
Integration with LlamaIndex
LlamaIndex's MarkdownNodeParser produces hierarchical nodes out of the box:
from llama_index.core.node_parser import MarkdownNodeParser
from llama_index.core import Document
parser = MarkdownNodeParser()
nodes = parser.get_nodes_from_documents([
Document(text=markdown)
])
for node in nodes:
print(node.metadata) # Heading path metadata included
Each node carries a metadata dict with the heading hierarchy, which LlamaIndex's index structures can use for auto-filtering during retrieval.
When not to use hierarchical chunking
Hierarchical chunking isn't always the right choice:
- Noise-heavy documents — OCR'd PDFs with broken heading detection produce a messy tree. Clean them with AnyMD first.
- Shallow documents — A document with one heading and ten paragraphs is better served by recursive or token-aware chunking.
- Code-heavy docs — API references with dozens of small functions under one heading benefit from function-level (code-block-aware) splitting.
- Real-time streaming — Building the tree requires the full document, so it's not suitable for incremental/streaming pipelines.
Putting it all together: A production pipeline
import anymd # hypothetical AnyMD Python SDK
from langchain_text_splitters import MarkdownHeaderTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
# Step 1: Convert any document to Markdown
response = anymd.convert("manual.docx")
markdown = response.markdown
# Step 2: Hierarchical chunk with heading metadata
splitter = MarkdownHeaderTextSplitter([
("#", "h1"), ("##", "h2"), ("###", "h3"),
])
docs = splitter.split_text(markdown)
# Step 3: Embed and index
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings)
# Step 4: Retrieve with hierarchy awareness
query = "How do I configure database settings?"
results = vectorstore.similarity_search(
query,
k=5,
# Optional: filter by section
filter={"h1": "Installation"},
)
for doc in results:
print(doc.metadata)
print(doc.page_content[:200])
print("---")
Next steps
Heading-based hierarchical chunking is the best strategy for most document Q&A and summarization workloads. For a deeper comparison of all chunking strategies, see semantic chunking from Markdown for RAG and fixed-size vs variable-size chunking benchmarks.
AnyMD gives you clean, structured Markdown from any document format in under a second — the perfect starting point for any chunking strategy.
Get your free API key and start building your RAG pipeline today.