--- license: mit language: - dan # ISO 639-3 code or "und" if not identifiable tags: - tokenizer - bpe - flexitok - fineweb2 --- # Byte-Level BPE Tokenizer: dan_Latn (4K) A **Byte-Level BPE** tokenizer trained on **dan_Latn** data from Fineweb-2-HQ. ## Training Details | Parameter | Value | |-----------|-------| | Algorithm | Byte-Level BPE | | Language | `dan_Latn` | | Target Vocab Size | 4,000 | | Final Vocab Size | 4,000 | | Pre-tokenizer | gpt4 | | Number handling | individual | | Contraction handling | True | | Normalizer | NFC | | Special Tokens | ``, ``, ``, `` | | Training Shards | 2 | ## Usage ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_dan_Latn_4000") tokens = tokenizer.encode("Hello, world!") ``` ## Files - `tokenizer.json` — Full HuggingFace tokenizer - `vocab.json` — Vocabulary mapping - `merges.txt` — BPE merge rules ## Sample Encoding | Text | Tokens | Token IDs | |------|--------|-----------| | `Hello, world! 12345 This is a test. こんにちは` | `H, ell, o, ,, Ġw, or, ld, !, Ġ, 12, 3, 45, ĠTh, is, Ġis, Ġa, Ġtest, ., Ġ, ã` | `43, 1250, 82, 15, 772, 217, 2088, 4, 174, 1268, 22, 3888, 1859, 268, 951, 213, 1886, 17, 174, 159` |