Over-Tokenized Transformers: Decoupling Input and Output Vocabularies for Improved Large Language Models

The idea of decoupling input and output vocabularies in language models is a fascinating and potentially transformative approach. Traditionally, both vocabularies are scaled together, leading to computational inefficiencies, especially in terms of expanding the output vocabulary. By independently scaling the input vocabulary (using techniques like large n-gram embeddings) and keeping the output vocabulary more compact, this approach appears to address significant inefficiencies in tokenization. It reduces the computational burden of scaling the output vocabulary, which is crucial for smaller models that may struggle with the complexity of a larger output space.

Efficiency: This decoupling could dramatically improve the efficiency of large language models (LLMs) without requiring proportionally more computational resources. As the study suggests, increasing the input vocabulary size by 128 times results in performance gains that would otherwise require doubling the model size—without additional computational costs. This shift allows models to better generalize and process complex language patterns with fewer resources, making LLMs more cost-effective and accessible.

Scalability: The key to scaling LLMs effectively lies in balancing model size and computational cost. The traditional approach tends to scale both the input and output vocabularies together, leading to prohibitive computational expenses when scaling up. Decoupling them provides a more flexible and scalable solution where only the input vocabulary is scaled as needed, which leads to a more efficient use of resources while still improving model performance.

In the future, this approach could lead to models that are both more efficient and capable of handling more complex tasks without requiring exponentially more computational power. It could also open the door for more specialized tokenization strategies, optimizing the trade-off between input and output vocabularies based on specific use cases, model sizes, or application domains. For instance, smaller models could benefit from a larger input vocabulary for better language representation, while avoiding overfitting from expanded output vocabularies.

Moreover, this decoupling could potentially facilitate the integration of domain-specific tokenization strategies, allowing language models to be fine-tuned more effectively for specialized tasks. By adjusting the input vocabulary independently, models could better capture domain-specific terminology or linguistic patterns without overburdening the model with unnecessary output complexity.

In conclusion, decoupling the input and output vocabularies could lead to more scalable, cost-effective, and powerful language models, offering a more flexible foundation for the future of NLP and large-scale language processing.