AI Text Chunker — Split Text for RAG and Embeddings
What is AI Text Chunker — Split Text for RAG and Embeddings
A text chunker (also called a text splitter) divides long documents into smaller pieces that fit within the context window of LLMs or the input size limits of embedding models. This is a core step in building RAG (Retrieval-Augmented Generation) systems, vector databases, and semantic search pipelines. You can split by estimated token count, character count, or paragraph boundaries — with configurable overlap to preserve context across chunk boundaries.
How to use
- Paste the text you want to split into the input field.
- Choose the split mode: Tokens (best for LLM context), Characters (precise size), or Paragraphs (natural breaks).
- Set the chunk size (e.g. 512 tokens for most embedding models) and optional overlap (e.g. 50 tokens to maintain context).
- Click Split text — each chunk appears below with its token and character count.
- Copy individual chunks with the Copy button next to each one.
When to use it
Loading a 50-page PDF into a vector database for semantic search? Paste the extracted text, set 512 tokens with 50 token overlap, and you get about 80 chunks that keep sentence boundaries intact. The overlap carries enough context across cut points that meaning survives, so you can send the chunks straight to an embedding model and store them for RAG retrieval.
Frequently asked questions
What chunk size should I use for RAG?
For most embedding models (text-embedding-3-small, text-embedding-ada-002, Cohere Embed), 256–512 tokens is the sweet spot. Larger chunks carry more context but may dilute the semantic signal. For LLM context windows, you can go up to 2000–4000 tokens per chunk depending on how many chunks you retrieve at once.
What is chunk overlap and why does it matter?
Overlap is the number of tokens/characters shared between consecutive chunks. Without overlap, a sentence that falls exactly on a chunk boundary gets split across two chunks, losing context. A 10–20% overlap (e.g. 50 tokens overlap for 512-token chunks) ensures continuity between adjacent chunks.
Should I split by tokens or characters?
Splitting by tokens is more accurate for LLM and embedding model limits. Splitting by characters gives deterministic, precise sizes — useful when the downstream system measures bytes or characters, not tokens. Paragraph mode is best for preserving natural document structure.
Does this tool break sentences mid-word?
No. The chunker tries to break at paragraph boundaries first, then sentence-ending punctuation, then word spaces. It only breaks mid-word if no better boundary is found within a 15% lookahead window.
Is there a limit on text length?
No hard limit — the tool runs entirely in your browser. Very large texts (over 1 million characters) may cause your browser to slow down during processing, but there is no server-side restriction.
Can I share this tool with my inputs pre-filled?
Yes. The URL updates automatically as you type. Copy it from the address bar or use the Share button — anyone who opens the link will see your exact inputs ready to go.
Related tools
AI Token Counter — GPT, Claude & Gemini
Count tokens in any text for GPT-4, Claude, and Gemini. Free online AI token counter. See context window usage, word count, and character count instantly.
Atbash Cipher Encoder and Decoder
Encode and decode the Atbash cipher online. Atbash mirrors the alphabet and is its own inverse, so one click scrambles and unscrambles. Free and private.
Background Remover
Remove the background from any image online and get a transparent PNG. Runs in your browser with AI, your photo is never uploaded. Free.