Tokenization Boundaries Impact AI Model Compression Costs

VERA Brief

AI-generated. Grounded in the article and its cited sources.

New research quantifies the cost of pre-tokenization boundaries in AI language models. Findings suggest these boundaries increase optimal token counts and influence dictionary selection for compression and prediction.

Key facts

• Pre-tokenization boundaries increase the optimal token count by 28.3% to 36.8% on English Wikipedia.
• Byte pair encoding was found to be 2.1% above the constrained lower bound and 10.9% above the unrestricted bound.
• Compression and prediction favor different dictionaries.
• Unrestricted fitting yielded higher mean held-out bits per byte across 12 languages under certain conditions.
• Licensing 10% of the vocabulary budget recovered significant token-count reduction.