SabiBERTa-135M

A 135M-parameter encoder for 70 African languages, pretrained from scratch with masked language modelling. Sabi is Nigerian Pidgin for "to know".

Its point is coverage. AfroXLMR covers 17 African languages and AfriBERTa 11; SabiBERTa covers 70, including languages with no prior encoder at all — Dagbani, Mossi, Kamba, Nuer, Sango, Umbundu, Kimbundu, Tiv, Urhobo, Efik.

Pooling: use mean pooling, not CLS

<cls> and <sep> are reserved in the tokenizer but were never trained. MLM pretraining here never inserts them into a sequence — windows are packed documents separated by </s> — so those two embedding rows only ever received the tied-head softmax gradient that every unused row receives. Measured during training: their gradient magnitude sits at 0.74–0.80x that of an ordinary token that never occurs.

This differs from BERT and from ModernBERT itself, both of which wrap every pretraining sequence as [CLS] ... [SEP] and so train those vectors.

Practically: pool by masked mean over the last hidden state, which is what E5, BGE and sentence-transformers do by default anyway. A CLS-pooling recipe copied from a BERT tutorial will start from an effectively random vector and underperform.

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("AletheiaAING/SabiBERTa-135M")
model = AutoModel.from_pretrained("AletheiaAING/SabiBERTa-135M")

def embed(texts):
    b = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    h = model(**b).last_hidden_state                      # (B, T, 768)
    m = b["attention_mask"].unsqueeze(-1).float()         # (B, T, 1)
    return (h * m).sum(1) / m.sum(1).clamp(min=1e-9)      # masked mean

For token-level tasks (NER, POS) this does not apply — use the per-token hidden states directly.

Architecture

ModernBERT (Warner et al., 2024), the current best-practice encoder recipe:

layers / dim / heads	12 / 768 / 12
context	4096
vocabulary	65,536 (+1 <mask>)
parameters	135.8M
positions	RoPE
FFN	GeGLU, pre-norm, no bias
attention	alternating local (128) / global, every 3rd layer global

The small vocabulary is why this is 135M where XLM-R base is 277M at the same depth and width: a 65,536 SabiYarn vocabulary trained on African text costs 50M embedding parameters against XLM-R's 192M for 250k multilingual tokens.

Training data

40.2B tokens drawn from a 100B-token deduplicated corpus (15.7% cross-document duplication removed), allocated by UniMax (Chung et al., ICLR 2023) with a 2-epoch repeat cap, plus a fixed 3B English anchor for cross-lingual transfer and code-switching.

Documents were selected by prior-based filtering (Seo et al., 2025): each document is scored by mean log unigram probability against its own language's prior, and by the standard deviation over equal chunks; those furthest from the language median are dropped, which removes gibberish and boilerplate without a trained classifier.

The corpus is 100% NFC-normalised (verified over ~418k documents).

Training

• Dynamic masking at 30% — a fresh mask every time a window is seen, per RoBERTa's ablation, at ModernBERT's rate rather than BERT's 15%.
• Cross-document attention masking — windows pack multiple documents (mean ~246 tokens), so without this a masked token would be predicted partly from an unrelated neighbouring article.
• Muon (Jordan et al. 2024, Moonlight scaling) on the hidden matrices, AdamW on embeddings, head and 1-D parameters.
• WSD schedule — constant LR after warmup, decayed over the final 10%, so the token budget stays a decision rather than a commitment.

Best checkpoint: step 76500, loss EMA 1.4818, 40.16B tokens seen.

Downstream evaluation

Fine-tuned per language under identical settings for every model (10 epochs, lr 5e-5, batch 32, max_len 256). Mean pooling is used for the sequence tasks, since <cls> was never trained.

Pretrained weights verified unchanged across the whole run (sha256 7abaa66d66fd6207…).

afrisenti — weighted_f1, mean over 12 languages

model	weighted_f1	accuracy	macro F1	weighted F1
SabiBERTa-135M (this model)	0.5896	0.5880	0.5373	0.5896
afro-xlmr-base	0.6567	0.6552	0.6085	0.6567
afriberta_large	0.5956	0.5942	0.5419	0.5956
serengeti-E250	0.6459	0.6414	0.6014	0.6459

masakhapos — accuracy, mean over 20 languages

model	accuracy	accuracy	macro F1	weighted F1
SabiBERTa-135M (this model)	0.8661	0.8661	—	—
afro-xlmr-base	0.8920	0.8920	—	—
afriberta_large	0.8771	0.8771	—	—
serengeti-E250	0.8869	0.8869	—	—

masakhanews — weighted_f1, mean over 16 languages

model	weighted_f1	accuracy	macro F1	weighted F1
SabiBERTa-135M (this model)	0.8727	0.8718	0.7390	0.8727
afro-xlmr-base	0.8900	0.8890	0.7749	0.8900
afriberta_large	0.8622	0.8605	0.6965	0.8622
serengeti-E250	0.8754	0.8750	0.7532	0.8754

sib200 — weighted_f1, mean over 60 languages

model	weighted_f1	accuracy	macro F1	weighted F1
SabiBERTa-135M (this model)	0.6382	0.6440	0.6146	0.6382
afro-xlmr-base	0.6286	0.6368	0.6008	0.6286
afriberta_large	0.5214	0.5300	0.4859	0.5214
serengeti-E250	0.6765	0.6822	0.6544	0.6765

masakhaner2 — span_f1, mean over 20 languages

model	span_f1	accuracy	macro F1	weighted F1
SabiBERTa-135M (this model)	0.7853	0.9719	—	—
afro-xlmr-base	—	—	—	OutOfMemoryError: CUDA out of memory. Tr

afrisenti per language

model	_skipped	amh	arq	ary	hau	ibo	kin	pcm	por	swa	tso	twi	yor
SabiBERTa-135M (this model)	—	0.490	0.565	0.463	0.693	0.685	0.646	0.637	0.617	0.567	0.453	0.630	0.630
afro-xlmr-base	—	0.677	0.644	0.548	0.740	0.764	0.655	0.670	0.655	0.598	0.583	0.640	0.708
afriberta_large	—	0.534	0.443	0.413	0.747	0.754	0.613	0.634	0.552	0.596	0.502	0.623	0.734
serengeti-E250	—	0.623	0.616	0.516	0.773	0.779	0.645	0.657	0.630	0.605	0.540	0.635	0.734

masakhapos per language

model	bam	bbj	ewe	fon	hau	ibo	kin	lug	luo	mos	nya	pcm	sna	swa	tsn	twi	wol	xho	yor	zul
SabiBERTa-135M (this model)	0.880	0.767	0.869	0.882	0.896	0.779	0.973	0.891	0.846	0.879	0.795	0.855	0.874	0.912	0.878	0.810	0.912	0.849	0.912	0.862
afro-xlmr-base	0.900	0.848	0.881	0.902	0.928	0.792	0.982	0.906	0.877	0.906	0.822	0.909	0.886	0.932	0.896	0.840	0.924	0.871	0.946	0.891
afriberta_large	0.887	0.798	0.872	0.882	0.928	0.790	0.977	0.890	0.861	0.894	0.794	0.895	0.877	0.929	0.885	0.826	0.915	0.840	0.943	0.859
serengeti-E250	0.900	0.837	0.883	0.881	0.930	0.786	0.979	0.910	0.880	0.896	0.817	0.902	0.885	0.929	0.894	0.837	0.919	0.854	0.935	0.884

masakhanews per language

model	amh	eng	fra	hau	ibo	lin	lug	orm	pcm	run	sna	som	swa	tir	xho	yor
SabiBERTa-135M (this model)	0.908	0.887	0.844	0.866	0.885	0.879	0.893	0.865	0.974	0.882	0.925	0.736	0.830	0.791	0.900	0.897
afro-xlmr-base	0.926	0.908	0.875	0.880	0.883	0.916	0.900	0.865	0.979	0.912	0.924	0.776	0.838	0.820	0.940	0.896
afriberta_large	0.900	0.883	0.773	0.886	0.851	0.871	0.832	0.868	0.960	0.900	0.885	0.765	0.838	0.820	0.854	0.909
serengeti-E250	0.917	0.890	0.843	0.873	0.867	0.867	0.863	0.889	0.977	0.879	0.901	0.734	0.843	0.840	0.915	0.911

sib200 per language

model	aeb_Arab	afr_Latn	aka_Latn	amh_Ethi	arb_Arab	arb_Latn	ary_Arab	arz_Arab	bam_Latn	bem_Latn	cjk_Latn	dik_Latn	dyu_Latn	eng_Latn	ewe_Latn	fon_Latn	fra_Latn	fuv_Latn	gaz_Latn	hau_Latn	ibo_Latn	kab_Latn	kam_Latn	kbp_Latn	kea_Latn	kik_Latn	kin_Latn	kmb_Latn	knc_Arab	knc_Latn	kon_Latn	lin_Latn	lua_Latn	lug_Latn	luo_Latn	mos_Latn	nso_Latn	nus_Latn	nya_Latn	plt_Latn	run_Latn	sag_Latn	sna_Latn	som_Latn	sot_Latn	ssw_Latn	swh_Latn	taq_Latn	taq_Tfng	tir_Ethi	tsn_Latn	tso_Latn	tum_Latn	twi_Latn	tzm_Tfng	umb_Latn	wol_Latn	xho_Latn	yor_Latn	zul_Latn
SabiBERTa-135M (this model)	0.378	0.790	0.729	0.779	0.416	0.539	0.456	0.446	0.689	0.696	0.508	0.615	0.532	0.816	0.683	0.658	0.734	0.609	0.673	0.760	0.794	0.497	0.600	0.572	0.647	0.723	0.782	0.594	0.345	0.530	0.669	0.677	0.619	0.642	0.607	0.653	0.750	0.586	0.739	0.589	0.743	0.598	0.712	0.700	0.760	0.704	0.789	0.502	0.253	0.751	0.722	0.694	0.733	0.712	0.273	0.610	0.679	0.776	0.708	0.751
afro-xlmr-base	0.820	0.837	0.590	0.765	0.862	0.526	0.810	0.847	0.579	0.607	0.505	0.587	0.560	0.880	0.566	0.515	0.837	0.573	0.588	0.727	0.720	0.436	0.568	0.524	0.661	0.625	0.709	0.580	0.376	0.566	0.619	0.642	0.599	0.550	0.625	0.557	0.664	0.466	0.692	0.776	0.704	0.598	0.674	0.717	0.653	0.624	0.756	0.565	0.290	0.669	0.654	0.593	0.691	0.611	0.210	0.544	0.577	0.697	0.617	0.736
afriberta_large	0.181	0.547	0.552	0.729	0.215	0.501	0.195	0.200	0.547	0.535	0.441	0.552	0.485	0.783	0.509	0.519	0.586	0.531	0.673	0.704	0.711	0.448	0.532	0.506	0.509	0.554	0.704	0.470	0.282	0.550	0.590	0.537	0.542	0.533	0.517	0.562	0.523	0.420	0.577	0.534	0.658	0.514	0.538	0.696	0.564	0.479	0.775	0.547	0.168	0.667	0.518	0.506	0.590	0.574	0.148	0.469	0.542	0.567	0.656	0.517
serengeti-E250	0.769	0.809	0.691	0.806	0.843	0.562	0.756	0.824	0.626	0.681	0.555	0.567	0.543	0.834	0.652	0.613	0.837	0.661	0.746	0.742	0.767	0.509	0.630	0.593	0.753	0.705	0.748	0.577	0.390	0.597	0.751	0.759	0.694	0.627	0.691	0.613	0.712	0.579	0.739	0.784	0.788	0.641	0.689	0.760	0.772	0.654	0.840	0.559	0.344	0.748	0.697	0.740	0.740	0.687	0.253	0.582	0.619	0.705	0.680	0.754

masakhaner2 per language

model	bam	bbj	ewe	fon	hau	ibo	kin	lug	luo	mos	nya	pcm	sna	swa	tsn	twi	wol	xho	yor	zul
SabiBERTa-135M (this model)	0.721	0.634	0.822	0.766	0.749	0.827	0.815	0.823	0.729	0.626	0.885	0.790	0.889	0.900	0.780	0.703	0.712	0.855	0.830	0.849

Intended use and limits

Designed as a base encoder for fine-tuning: classification, retrieval, token labelling, and as a backbone for data-quality filtering across African languages.

It will be weak on the smallest languages. The repeat cap means languages with very little text contribute a tiny share — ijo has ~4k tokens available, kln ~250, gaa ~44k. Those are present so the model does not treat them as noise, not because it has learned them. Do not rely on it for quality judgements in those languages; there is not enough text in existence to support one.

Untoned Yoruba and Igbo are both well represented (18% of Yoruba documents and 52% of Igbo carry no tone marks), which is deliberate — that is what users type — but it means the model does not assume diacritics are present.

Downstream results

Five benchmarks, four models, one harness. masakhaner2 is fine-tuned per language (the protocol the baseline papers headline); the other four use the multilingual protocol -- one model over every language at once, as in MasakhaNER 2.0's Table 5 and AfriSenti's Task B, and as this encoder would actually be deployed.

task	metric	this model	afro-xlmr-base	afriberta-large	serengeti-E250
masakhaner2 (per-lang, 20)	span F1	0.7882	—	83.0 published	—
masakhapos (ml, 20)	accuracy	0.8661	0.8920	0.8771	0.8869
masakhanews (ml, 16)	weighted F1	0.8727	0.8900	0.8622	0.8754
afrisenti (ml, 12)	weighted F1	0.5896	0.6567	0.5956	0.6459
sib200 (ml, 60)	weighted F1	0.6382	0.6286	0.5214	—

Baseline figures quoted as published are not recomputed: this harness reproduced AfroXLMR-base's published per-language MasakhaNER 2.0 scores to a mean absolute difference of 0.57 F1 over 9 languages, and AfriBERTa-large to within 0.4. Single- language run-to-run variance is about 1.1 F1 (Bambara measured 79.9 and 81.0 on two runs of one configuration) and the papers average 5 runs where these are single runs, so read differences under ~1.5 F1 on one language as noise.

Where the deficit comes from: coverage or capacity?

Splitting each task's per-language results by whether the baseline pretrained on that language separates the two explanations. Capacity dilution predicts a roughly uniform deficit; a coverage advantage should appear precisely where a baseline has no exposure.

sib200, against AfriBERTa-large (11 pretraining languages):

	n	this model	afriberta	Δ	wins
covered by AfriBERTa	8	0.7523	0.6906	+6.17	8/8
not covered	52	0.6206	0.4953	+12.53	50/52

The advantage doubles on languages AfriBERTa never saw -- 6.4 F1 wider. Pretraining on 69 languages buys exactly what it should.

sib200, against AfroXLMR-base (20 MAFT languages):

	n	this model	afro-xlmr	Δ	wins
covered by AfroXLMR	15	0.7563	0.7310	+2.54	11/15
not covered	45	0.5988	0.5944	+0.44	33/45

This runs the other way, and the likely reason is that AfroXLMR inherits XLM-R's 100 languages, so "outside the MAFT set" does not mean unseen.

masakhapos, against AfroXLMR-base: −2.52 on covered languages, −2.67 on uncovered, 0/20 wins. The deficit is flat, which is the signature capacity dilution predicts.

So the two effects are both real, on different tasks: a breadth advantage on topic classification across 60 languages, and a uniform ~2.5-point shortfall on POS tagging that coverage does not explain.

Capacity dilution across 69 languages

For the uniform component, the literature gives the mechanism. XLM-R's authors report that "the per-language capacity decreases as we increase the number of languages", that XNLI accuracy falls from 71.8% to 67.7% going from 7 languages to 100, and that "the issue is even more prominent when the capacity of the model is small"; they widened the hidden size 768 → 960 for 30 languages and found 1152 still insufficient for 100. This model sits at 69 languages, d=768, 86M non-embedding parameters, against AfriBERTa's 11 languages, d=768, 72M.

Three alternatives are ruled out by measurement rather than argument:

• Data volume. Hausa is the weakest NER language (−12.7 against published) and is rank 5 in this corpus at 1.27B tokens -- more than Zulu, the strongest (+2.0).
• Corpus quality. The Hausa corpus is cleaner than the Zulu one on every axis measured: 0.6% duplicate 7-grams against Zulu's 10.1%, fewer digits, no detectable language contamination, 87% human-sourced.
• Tokenisation. Fertility over ~92k words of real text is 1.841 tokens/word against AfriBERTa's 2.096 and afro-xlmr's 2.269 -- this tokenizer is the efficient one, on a 65,536 vocabulary against their 70k and 250k.

Two findings remain unexplained. The Hausa corpus uses the hooked consonants ɓ ɗ ƙ at 34.65 per 1,000 words while MasakhaNER 2.0's Hausa is fully ASCII-ised (0.00) -- the only language with that divergence, and the only one with a double-digit gap. But the mechanism does not follow: this tokenizer handles the ASCII-ised benchmark text at 1.237 tokens/word, better than its own corpus, so there is no fragmentation penalty. Separately, Hausa is the only language that got worse with 5× the epochs (75.1 → 72.6), suggesting overfitting there specifically.

No ablation has been run: testing capacity properly means pretraining a wider model or a narrower language set.

Known coverage gap

bbj (Ghomálá') is one of MasakhaNER 2.0's 20 languages and is absent from this corpus entirely, so its 63.8 reflects zero pretraining exposure. The commercially-licensed Ghomálá' text that exists amounts to roughly 88k words (~225k tokens) across stfotso/french-ghomala-bandjoun, masakhane/AfriADR and FineWeb-2 — 0.0006% of this corpus, which would leave it unlearned even if added. masakhane/mafand and aiana94/polynews carry Ghomálá' but are CC-BY-NC and excluded under this project's commercial-use requirement.

Citation

Uses UniMax (Chung et al., ICLR 2023), ModernBERT (Warner et al., 2024), prior-based filtering (Seo et al., 2025), Muon (Jordan et al., 2024), RoBERTa's dynamic masking (Liu et al., 2019).