Pa'O Tokenizer
Repository: paodigitalhub/pao-tokenizer
A Hugging Face compatible tokenizer designed for Pa'O language text.
PURPOSE
This repository provides a tokenizer for Pa'O language text.
The tokenizer is designed to support:
- Basic Unicode components
- Pa'O combined characters
- Syllable and subword units
- Pa'O words
- Longer Pa'O expressions
- Space-less Pa'O text
- Token -> ID conversion
- ID -> Token conversion
- Hugging Face Transformers
- PreTrainedTokenizerFast
TOKEN HIERARCHY
The vocabulary is organized into several conceptual levels.
- BASIC COMPONENT LEVEL
Examples:
ှ ီ ိ ် ါ ာ ဲ ွ ္ ၁ ၈ ၀ ၅ ံ ၊။ ဆ တ ေ ျ ဖ ထ ခ လ ဘ ယ း ႏ ꩻ
- COMBINED / SYLLABLE LEVEL
Examples:
ကု သီ နေꩻ စိုႏ လမ်း ဆာ ခွ ကꩻ ကား ဒူ
- WORD LEVEL
Examples:
နာꩻ လွေꩻ ကျောင်ꩻ ခွေ သဲင်ꩻ လမ်း ဖိုး အံႏ အဆာႏ
- LONGER EXPRESSION LEVEL
Examples:
ကွပ်ဒုံသား ယံငါႏတဲင် ဖါဖြားပိုလီꩻဖုံႏ ထွဉ်ႏ လိုꩻတန်ဖုံႏတွမ်ႏ
LONGEST-MATCH TOKENIZATION
The tokenizer uses a longest-match strategy.
For example:
Input:
နာꩻလွေꩻကျောင်ꩻ
Tokens:
နာꩻ လွေꩻ ကျောင်ꩻ
IDs:
150 151 152
Therefore:
နာꩻလွေꩻကျောင်ꩻ ↓ [150, 151, 152]
ID -> TOKEN
150 -> နာꩻ 151 -> လွေꩻ 152 -> ကျောင်ꩻ
TOKEN -> ID
နာꩻ -> 150 လွေꩻ -> 151 ကျောင်ꩻ -> 152
PYTHON USAGE
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained( "paodigitalhub/pao-tokenizer", use_fast=True )
text = "နာꩻလွေꩻကျောင်ꩻ"
result = tokenizer( text, add_special_tokens=False )
print(result["input_ids"])
print( tokenizer.convert_ids_to_tokens( result["input_ids"] ) )
EXPECTED OUTPUT
[150, 151, 152]
['နာꩻ', 'လွေꩻ', 'ကျောင်ꩻ']
ID -> TOKEN EXAMPLE
ids = [150, 151, 152]
tokens = tokenizer.convert_ids_to_tokens(ids)
print(tokens)
Expected:
['နာꩻ', 'လွေꩻ', 'ကျောင်ꩻ']
TOKEN -> ID EXAMPLE
tokens = [ "နာꩻ", "လွေꩻ", "ကျောင်ꩻ" ]
ids = tokenizer.convert_tokens_to_ids(tokens)
print(ids)
Expected:
[150, 151, 152]
DECODE EXAMPLE
ids = [150, 151, 152]
text = tokenizer.decode( ids, skip_special_tokens=True, clean_up_tokenization_spaces=False )
print(text)
Expected:
နာꩻလွေꩻကျောင်ꩻ
SPECIAL TOKENS
[PAD] = 0 [UNK] = 1 [CLS] = 2 [SEP] = 3 [MASK] = 4
PROJECT FILES
vocab.json Token vocabulary and token IDs.
tokenizer.json Hugging Face Tokenizers serialization.
tokenizer_config.json Transformers tokenizer configuration.
special_tokens_map.json Special token configuration.
added_tokens.json Added token information.
train_tokenizer.py Builds the tokenizer.
test_tokenizer.py Tests local tokenization.
inspect_vocab.py Displays the vocabulary as ID -> Token.
hf_test.py Tests loading the tokenizer from Hugging Face.
requirements.txt Python dependencies.
README.txt Project documentation.
LICENSE Project license.
BUILD
Install dependencies:
pip install -r requirements.txt
Build the tokenizer:
python train_tokenizer.py
Test the tokenizer:
python test_tokenizer.py
Inspect vocabulary:
python inspect_vocab.py
HUGGING FACE TEST
After uploading the repository to Hugging Face:
python hf_test.py
The tokenizer can also be loaded directly:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained( "paodigitalhub/pao-tokenizer", use_fast=True )
CURRENT VOCABULARY
The current vocabulary is an initial development vocabulary.
It is not intended to represent the complete Pa'O language vocabulary.
The vocabulary should be expanded using verified Pa'O text corpora.
FUTURE VOCABULARY LEVELS
Future versions may include:
- Unicode components
- Consonants
- Vowels
- Diacritics
- Syllables
- Subwords
- Words
- Phrases
- Sentence structures
INTENDED APPLICATIONS
Possible applications include:
- Pa'O NLP
- Pa'O corpus processing
- Pa'O text normalization
- Pa'O language learning tools
- Pa'O search
- Pa'O keyboard systems
- Pa'O speech-to-text preprocessing
- Pa'O text-to-speech preprocessing
- Pa'O language models
- Pa'O language technology
CORPUS DEVELOPMENT
The tokenizer is intended to grow together with a Pa'O text corpus.
Future corpus development may include:
- 5,000 Pa'O sentences
- 10,000+ Pa'O sentences
- 50,000+ Pa'O sentences
- 100,000+ Pa'O sentences
The corpus should contain verified Pa'O text.
IMPORTANT
This repository provides a tokenizer.
It is not a complete Pa'O language model.
The tokenizer does not generate Pa'O text by itself.
Token quality depends on vocabulary coverage and the quality of the source corpus.
LANGUAGE
Pa'O
ISO 639-3:
blk
PROJECT
Pa'O Digital Hub
Hugging Face:
paodigitalhub/pao-tokenizer