|
Download README.md from flexitok/bpe_dan_Latn_4000: direct link, hf CLI and curl.
- Browser
- Download file 1.27 kB
-
https://huggingface.co/flexitok/bpe_dan_Latn_4000/resolve/main/README.md
- Command line
-
hf download hf://flexitok/bpe_dan_Latn_4000/README.md
-
curl -L -o README.md https://huggingface.co/flexitok/bpe_dan_Latn_4000/resolve/main/README.md
1.27 kB
| license: mit | |
| language: | |
| - dan # ISO 639-3 code or "und" if not identifiable | |
| tags: | |
| - tokenizer | |
| - bpe | |
| - flexitok | |
| - fineweb2 | |
| # Byte-Level BPE Tokenizer: dan_Latn (4K) | |
| A **Byte-Level BPE** tokenizer trained on **dan_Latn** data from Fineweb-2-HQ. | |
| ## Training Details | |
| | Parameter | Value | | |
| |-----------|-------| | |
| | Algorithm | Byte-Level BPE | | |
| | Language | `dan_Latn` | | |
| | Target Vocab Size | 4,000 | | |
| | Final Vocab Size | 4,000 | | |
| | Pre-tokenizer | gpt4 | | |
| | Number handling | individual | | |
| | Contraction handling | True | | |
| | Normalizer | NFC | | |
| | Special Tokens | `<s>`, `</s>`, `<pad>`, `<unk>` | | |
| | Training Shards | 2 | | |
| ## Usage | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_dan_Latn_4000") | |
| tokens = tokenizer.encode("Hello, world!") | |
| ``` | |
| ## Files | |
| - `tokenizer.json` — Full HuggingFace tokenizer | |
| - `vocab.json` — Vocabulary mapping | |
| - `merges.txt` — BPE merge rules | |
| ## Sample Encoding | |
| | Text | Tokens | Token IDs | | |
| |------|--------|-----------| | |
| | `Hello, world! 12345 This is a test. こんにちは` | `H, ell, o, ,, Ġw, or, ld, !, Ġ, 12, 3, 45, ĠTh, is, Ġis, Ġa, Ġtest, ., Ġ, ã` | `43, 1250, 82, 15, 772, 217, 2088, 4, 174, 1268, 22, 3888, 1859, 268, 951, 213, 1886, 17, 174, 159` | | |