# marcelroed/gigatoken

By Saurabh Kumar · TIL note, Jul 24, 2026 · Tags: search, performance

Source: https://saurabh-kumar.com/til/1784894400/

Embedding pipelines do not feed whole documents straight into the model. They first use the embedding model's tokenizer to measure and split text into token-bounded, overlapping chunks, then embed each chunk and store the vectors. Tokenizer must match the model. Tools like [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken) accelerate this preparation at corpus scale but do not create embeddings themselves.