Feature Extraction
Transformers
Sinhala
Hindi
English
tokenizer
WWHO
SGPE
linguis_trie
token
tokenization
Syllable
remeinium
transformer
linguistics
NLP
sinhala
hindi
english
BPE
GPE
Eval Results (legacy)
Instructions to use Remeinium/WWHO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Remeinium/WWHO with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="Remeinium/WWHO")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Remeinium/WWHO", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -153,18 +153,17 @@ Apache License 2.0 — see [LICENSE](LICENSE).
|
|
| 153 |
|
| 154 |
## Citations
|
| 155 |
|
| 156 |
-
<div style="background: #
|
| 157 |
-
<
|
| 158 |
-
<p style="font-size: 1.1rem; line-height: 1.6;">
|
| 159 |
<strong>Title:</strong> Separate Before You Compress: The WWHO Tokenization Architecture<br>
|
| 160 |
<strong>Author:</strong> Kusal Darshana<br>
|
| 161 |
<strong>Year:</strong> 2026<br>
|
| 162 |
-
<strong>DOI/ArXiv:</strong> <a href="https://arxiv.org/abs/2603.25309">2603.25309</a>
|
| 163 |
</p>
|
| 164 |
|
| 165 |
<details style="margin-top: 1rem;">
|
| 166 |
-
<summary style="cursor: pointer; color: #
|
| 167 |
-
<pre style="background: #2d2d2d; color: #
|
| 168 |
@article{darshana2026separate,
|
| 169 |
title={Separate Before You Compress: The {WWHO} Tokenization Architecture},
|
| 170 |
author={Darshana, Kusal},
|
|
|
|
| 153 |
|
| 154 |
## Citations
|
| 155 |
|
| 156 |
+
<div style="background: #1d1d1d; padding: 1.5rem; border-radius: 8px; border: 1px solid #222; font-family: sans-serif;">
|
| 157 |
+
<p style="font-size: 1.1rem; line-height: 1.6; color:#e0e0e0;">
|
|
|
|
| 158 |
<strong>Title:</strong> Separate Before You Compress: The WWHO Tokenization Architecture<br>
|
| 159 |
<strong>Author:</strong> Kusal Darshana<br>
|
| 160 |
<strong>Year:</strong> 2026<br>
|
| 161 |
+
<strong>DOI/ArXiv:</strong> <a href="https://arxiv.org/abs/2603.25309" style="color: #e2e4bf; text-decoration: underline;">2603.25309</a>
|
| 162 |
</p>
|
| 163 |
|
| 164 |
<details style="margin-top: 1rem;">
|
| 165 |
+
<summary style="cursor: pointer; color: #d78cee; font-weight: bold;">View BibTeX</summary>
|
| 166 |
+
<pre style="background: #2d2d2d; color: #e9e9e9; padding: 1rem; border-radius: 6px; overflow-x: auto; font-size: 0.9rem; margin-top: 0.5rem;">
|
| 167 |
@article{darshana2026separate,
|
| 168 |
title={Separate Before You Compress: The {WWHO} Tokenization Architecture},
|
| 169 |
author={Darshana, Kusal},
|