Skip to content

Home / Linguistics Research

07/2025 - 12/2025

Tokenization Inequity Research (Linguistics & AI Research)

Researched inequities in LLM tokenization across 50 languages, developing a novel metric that linked token inefficiency to tokenizer design bias rather than inherent linguistic complexity.

First page of the paper "Are Some Languages Inherently Harder to Tokenize? Disentangling Linguistic Structure from Tokenizer Bias in Multilingual NLP" by Anders Christensen, with abstract

Title slide: "Are Certain Languages Harder to Tokenize Than Others?" by Anders Christensen, Seoul International School

Table of award results with entry N-367, a tokenization paper in Korean by an individual entrant, highlighted as a silver award