Jamo Tokenization Translator
Korean morpheme boundaries often fall inside a syllable, where character-level BPE can't cut. This project decomposes each syllable into its jamo (initial, medial, final) before applying BPE — yielding fewer, more morphologically aligned tokens — and measures the effect by training an identical Korean→English Seq2Seq Transformer on each tokenizer.
The morpheme boundary lives inside the syllable
Standard tokenizers — BPE, WordPiece, SentencePiece — treat the character or byte as the smallest unit. Korean breaks that assumption: in 노랫말 (lyrics) the interfix ㅅ sits inside the syllable 랫, and in 먹었어요 the past-tense ㅆ is fused into 었. A character tokenizer can never split those apart.
The hypothesis: decompose each Hangul syllable into its jamo first, then let BPE merge. Sub-syllable structure becomes visible, so the tokenizer can land on morpheme-aligned pieces and represent the language with fewer tokens — especially at small vocabularies and on out-of-domain text.
Two tokenizers, one idea: decompose, then merge
Reversible jamo decomposition
Hangul syllables (U+AC00–U+D7A3) decompose by S = ((init × 21) + med) × 28 + fin + 0xAC00. Using kss's h2hcj / hcj2h, text round-trips losslessly between syllables and compatibility jamo, so any tokenization is fully reversible.
JamoBPE — from scratch
A pure-Python BPE: build word-frequency tuples, count adjacent pair statistics, merge the most frequent pair, repeat until the vocab target or a min-frequency floor. Written for conceptual clarity over the whole train→merge loop.
HFJamoBPE — production
A HuggingFace tokenizers build on the Rust backend with end_of_word_suffix "</w>", trained on millions of CC-100 Korean lines at 32k and 16k vocabularies — the version used for the real experiments.
Lossless decoding
Decoding rejoins tokens, turns </w> back into spaces, and reconstructs Hangul with hcj2h() — so the jamo round-trip is invisible to anything downstream.
An identical Transformer behind each tokenizer
Clear token savings, an honest downstream caveat
Fewer tokens off-domain
On small or domain-mismatched text, jamo uses 30–50% fewer tokens than character BPE — BTS's 봄날 came to 530 tokens vs 953 — with the gap converging to ~1.6% on large general CC-100 text.
Tighter at small vocab
At a 16k vocabulary, jamo averaged ~5.5% fewer tokens and showed lower variance, so the representation is both shorter and more consistent.
Morpheme-aligned splits
Qualitatively, jamo segmentation exposes morpheme boundaries inside syllables (interfix ㅅ, past-tense ㅆ) that character BPE structurally cannot reach.
No free lunch downstream
At this model scale the jamo and character training-loss curves were nearly identical (<1%) — the efficiency gain did not translate into measurably better translation, stated plainly as a limitation.
Tokenize and translate, live
A Streamlit app lets you pick a dataset (small / large / large-16k) and a mode (jamo / full character), type Korean, and see the tokenization and token count side by side — with live Korean→English translation on the large models. It made the comparison between approaches something you can feel rather than just read off a table.
What it's built on
- Tokenization
- jamokssHuggingFace tokenizerstiktoken (baseline)
- Modeling
- PyTorchSeq2Seq TransformerSinusoidal PEGreedy decode
- Data
- CC-100 (ko)AI Hub ko–endatasets
- Demo & infra
- StreamlitPython 3.12bfloat16
Carried end to end — the tokenizer design, the translation model and training, the quantitative and qualitative evaluation, and the live demo — to probe an underexplored direction in Korean tokenization.