If you're building a RAG pipeline, the quality of your retrieval depends almost entirely on how you chunk your documents. Poor chunking means relevant context gets split across two vectors — and your LLM never sees the full picture.
Markdown preserves document structure — headings, lists, tables — which makes it the ideal input for semantic chunking strategies. After converting with AnyMD, you're not working with raw text from a PDF parser; you're working with structured documents where every heading, list item, and table cell has meaningful boundaries.
This post compares three chunking strategies on Markdown output from AnyMD: heading-based, token-based, and recursive character chunking.
Why Markdown matters for chunking
Raw PDF text extraction loses everything that makes a document a document. Headings become plain paragraphs. List numbering collapses. Tables become jagged rows of unaligned text. When you feed that into a chunking algorithm, the chunks have no semantic anchors — they're just word-count windows over a linear stream of text.
Markdown from AnyMD preserves:
- Heading hierarchy —
#,##,###map directly to document sections - List structure — ordered and unordered lists with correct nesting
- Table headers — GFM tables with column alignment
- Code blocks — fenced blocks with language tags
- Block quotes — semantic boundaries for cited content
Each of these is a semantic boundary — a natural place to split your chunks so that each chunk contains a complete thought.
Strategy 1: Heading-based chunking
Heading-based chunking splits a Markdown document at heading boundaries. Each # or ## starts a new chunk. This is the simplest and often most effective strategy for long-form documents.
import re
from pathlib import Path
def heading_chunks(markdown: str, max_chars: int = 2000) -> list[dict]:
"""Split Markdown into chunks by heading boundaries."""
# Split on any heading (# through ######)
parts = re.split(r'(^#+\s+.*$)', markdown, flags=re.MULTILINE)
chunks = []
current = {"heading": "Introduction", "content": ""}
for part in parts:
if re.match(r'^#+\s+', part):
# Save previous chunk
if current["content"]:
chunks.append(current)
current = {"heading": part.strip("# ").strip(), "content": part + "\n"}
else:
current["content"] += part
if current["content"]:
chunks.append(current)
# Merge small chunks under a parent heading
merged = []
for c in chunks:
if merged and len(c["content"]) + len(merged[-1]["content"]) < max_chars:
merged[-1]["content"] += "\n" + c["content"]
else:
merged.append(c)
return merged
# Apply to AnyMD output
md_text = Path("report.md").read_text()
chunks = heading_chunks(md_text)
for i, c in enumerate(chunks):
print(f"Chunk {i+1}: {c['heading']} ({len(c['content'])} chars)")The result is a list of chunks where each chunk maps to a document section. For retrieval, this means a query about "installation" retrieves the installation section — not a fragment that happens to contain the word "installation" somewhere in the middle.
Pros and cons
| Aspect | Rating | Notes |
|---|---|---|
| Retrieval precision | High | Chunks align with document structure |
| Implementation complexity | Low | ~15 lines of Python |
| Chunk size control | Medium | Depends on section length |
| Context coherence | Excellent | Each chunk is a complete section |
| Best for | — | Technical docs, reports, manuals |
Strategy 2: Token-aware chunking
Token-aware chunking accounts for your model's context window. Instead of counting characters, you count tokens using the same tokenizer your embedding or LLM model uses.
import tiktoken
from typing import List
def token_chunks(
markdown: str,
model: str = "text-embedding-3-small",
max_tokens: int = 512,
overlap_tokens: int = 64,
) -> List[str]:
enc = tiktoken.encoding_for_model(model)
tokens = enc.encode(markdown)
chunks = []
start = 0
while start < len(tokens):
end = start + max_tokens
chunk_tokens = tokens[start:end]
chunks.append(enc.decode(chunk_tokens))
start += max_tokens - overlap_tokens
return chunks
# Tokenize AnyMD output
md_text = """## Installation
To install AnyMD, send the file to the API endpoint.
The response is clean GitHub-Flavored Markdown."""
chunks = token_chunks(md_text, max_tokens=128)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {len(chunk)} chars")Token-aware chunking gives you precise control over chunk size — every chunk fits inside your model's context window. The overlap parameter (default 64 tokens) ensures that no concept is split across a chunk boundary.
Pros and cons
| Aspect | Rating | Notes |
|---|---|---|
| Retrieval precision | Medium | Can break mid-section |
| Implementation complexity | Medium | Needs tokenizer library |
| Chunk size control | Excellent | Guaranteed fit in context window |
| Context coherence | Medium | Overlap helps but not perfect |
| Best for | — | LLM context optimization |
Strategy 3: Recursive character chunking
LangChain-style recursive chunking tries to split on natural boundaries (paragraphs, sentences, words) in order of preference, falling back to smaller separators only when a chunk exceeds the size limit.
def recursive_chunk(
text: str,
max_chars: int = 1000,
overlap: int = 100,
) -> list[str]:
separators = ["\n\n", "\n", ". ", " ", ""]
def _chunk(text: str, sep_idx: int = 0) -> list[str]:
if len(text) <= max_chars or sep_idx >= len(separators):
return [text] if text.strip() else []
sep = separators[sep_idx]
parts = text.split(sep)
chunks = []
current = ""
for part in parts:
candidate = current + (sep if current else "") + part
if len(candidate) <= max_chars:
current = candidate
else:
if current:
chunks.append(current)
current = part
if current:
chunks.append(current)
# If splitting on this separator didn't reduce size, try next
if len(chunks) == 1 and len(chunks[0]) > max_chars * 1.1:
return _chunk(text, sep_idx + 1)
return chunks
raw = _chunk(text)
# Add overlap: prepend last `overlap` chars from previous chunk
result = []
prev_tail = ""
for chunk in raw:
if prev_tail:
chunk = prev_tail + chunk
if len(chunk) > max_chars * 1.5:
chunk = chunk[len(prev_tail):] # Remove overlap if too long
else:
result[-1] = result[-1][:-len(prev_tail)] # Trim previous
result.append(chunk)
prev_tail = chunk[-overlap:] if len(chunk) >= overlap else chunk
return result
# Compare on real AnyMD output
md = Path("report.md").read_text()
chunks = recursive_chunk(md)
for i, c in enumerate(chunks):
print(f"Chunk {i+1}: {len(c)} chars")Recursive chunking works well on any text, but it performs best on well-structured Markdown because the \n\n separator cleanly splits paragraphs. On raw PDF text (which often has no paragraph breaks), it falls all the way to word-level splits.
Pros and cons
| Aspect | Rating | Notes |
|---|---|---|
| Retrieval precision | Medium-High | Depends on document quality |
| Implementation complexity | Medium | Recursive logic needs testing |
| Chunk size control | Good | Hard limit per chunk |
| Context coherence | Good | Overlap preserves transitions |
| Best for | — | Mixed document types |
Benchmark: Three strategies on 100 real documents
We tested the three strategies on 100 real documents (PDFs, DOCX, reports, articles) converted to Markdown via AnyMD. The metric is retrieval precision@5 — the fraction of top-5 retrieved chunks that contain the answer.
| Strategy | Precision@5 | Avg chunks/doc | Avg chunk size | Implementation |
|---|---|---|---|---|
| Heading-based | 0.87 | 14 | 1,240 chars | 15 lines |
| Token-aware (512 tok) | 0.79 | 22 | 1,890 chars | 10 lines + tokenizer |
| Recursive character | 0.83 | 18 | 980 chars | 40 lines |
Heading-based chunking wins on precision because chunks align with document semantics. When a user asks "How do I install the CLI?", the answer lives in the "Installation" section — not split across two chunks.
Token-aware chunking is the most consistent for context-window management, making it the right choice when you're feeding chunks directly into an LLM.
Recursive character chunking is the best default for heterogeneous document collections, where section structure varies wildly between documents.
Building a hybrid pipeline
In production, the best approach combines strategies. Here's a pipeline that starts with heading boundaries and falls back to token-aware splitting for oversized sections:
import tiktoken
import re
from pathlib import Path
from typing import List, Dict, Any
def hybrid_chunk(
markdown: str,
section_header: str = "Introduction",
max_tokens: int = 512,
overlap_tokens: int = 48,
model: str = "text-embedding-3-small",
) -> List[Dict[str, Any]]:
enc = tiktoken.encoding_for_model(model)
chunks = []
# Step 1: Split by heading
parts = re.split(r'(^#+\s+.*$)', markdown, flags=re.MULTILINE)
sections = []
current_header = section_header
current_text = ""
for part in parts:
if re.match(r'^#+\s+', part):
if current_text.strip():
sections.append((current_header, current_text.strip()))
current_header = part.strip("# ").strip()
current_text = part
else:
current_text += part
if current_text.strip():
sections.append((current_header, current_text.strip()))
# Step 2: Tokenize each section; split oversized ones
for header, text in sections:
tokens = enc.encode(text)
if len(tokens) <= max_tokens:
chunks.append({"heading": header, "tokens": len(tokens), "text": text})
continue
# Split oversized section with overlap
for i in range(0, len(tokens), max_tokens - overlap_tokens):
chunk_tokens = tokens[i:i + max_tokens]
chunks.append({
"heading": header,
"tokens": len(chunk_tokens),
"text": enc.decode(chunk_tokens),
"sub_chunk": i > 0,
})
return chunks
# Use with AnyMD
md = Path("converted_report.md").read_text()
chunks = hybrid_chunk(md)
print(f"Created {len(chunks)} chunks from {md.count(chr(10)) + 1} lines")
for c in chunks[:5]:
print(f" [{c['heading']}] {c['tokens']} tok")Integration with LangChain and LlamaIndex
Both frameworks support Markdown-aware splitters:
# LangChain
from langchain_text_splitters import MarkdownHeaderTextSplitter
splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[
("#", "h1"),
("##", "h2"),
("###", "h3"),
]
)
chunks = splitter.split_text(markdown)# LlamaIndex
from llama_index.core.node_parser import MarkdownNodeParser
parser = MarkdownNodeParser()
nodes = parser.get_nodes_from_documents([markdown_doc])These splitters work best when the input is well-formed Markdown — which is exactly what AnyMD produces. Feed raw PDF text into them and the heading detection finds nothing; feed AnyMD output and every section boundary is properly tagged.
Where AnyMD fits in the pipeline
AnyMD sits at the front of your ingestion pipeline: raw documents → clean Markdown → chunk → embed → index. By converting to Markdown first, every chunking strategy works better because the structural information the chunker needs (headings, lists, tables) is preserved in the text.
For a complete RAG ingestion pipeline using AnyMD + LangChain, see our RAG pipeline guide.
For a deeper comparison of 10 chunking strategies across 500 real documents, see fixed-size vs variable-size chunking benchmarks.
Pricing
AnyMD converts documents to Markdown under standard page-based pricing — the chunking happens on your side, so you only pay for the conversion:
| Plan | Pages/month | Max file size | Price |
|---|---|---|---|
| Free | 100 | 10 MB | $0 |
| Starter | 500 | 25 MB | $19 |
| Pro | 5,000 | 50 MB | $99 |
| Enterprise | Custom | Custom | Custom |
Get your free API key — 100 pages/month, no credit card required.