Add stage-1 model card with training details and results
Browse files
README.md
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: unknown
|
| 3 |
+
base_model: jhu-clsp/mmbert-base
|
| 4 |
+
library_name: transformers
|
| 5 |
+
tags:
|
| 6 |
+
- modernbert
|
| 7 |
+
- mmbert
|
| 8 |
+
- sentiment-analysis
|
| 9 |
+
- twitter
|
| 10 |
+
- multilingual
|
| 11 |
+
datasets:
|
| 12 |
+
- cardiffnlp/tweet_eval
|
| 13 |
+
- cardiffnlp/super_tweeteval
|
| 14 |
+
- cardiffnlp/tweet_sentiment_multilingual
|
| 15 |
+
metrics:
|
| 16 |
+
- f1
|
| 17 |
+
- accuracy
|
| 18 |
+
---
|
| 19 |
+
# mmBERT-base — sentiment task adaptation (stage 1)
|
| 20 |
+
|
| 21 |
+
Full fine-tune of [`jhu-clsp/mmbert-base`](https://huggingface.co/jhu-clsp/mmbert-base) (multilingual ModernBERT, 22 layers, 256K-token Gemma-2 tokenizer) for **3-class tweet sentiment** (`negative` / `neutral` / `positive`), mirroring a tweet task-adapted checkpoint before per-language LoRA specialization (stage 2: [`mmbert-sentiment-lora`](https://huggingface.co/Ido-shraga/mmbert-sentiment-lora)).
|
| 22 |
+
|
| 23 |
+
## Training data (combined, ~87K train / 8.6K val)
|
| 24 |
+
|
| 25 |
+
| Dataset | Config | Train rows | Label handling |
|
| 26 |
+
|---|---|---:|---|
|
| 27 |
+
| `cardiffnlp/tweet_eval` | `sentiment` | 45,615 | 3-class as-is |
|
| 28 |
+
| `cardiffnlp/super_tweeteval` | `tweet_sentiment` | 26,632 | 5-point ABSA scale folded: {strongly negative, negative}→negative, {negative or neutral}→neutral, {positive, strongly positive}→positive |
|
| 29 |
+
| `cardiffnlp/tweet_sentiment_multilingual` | `all` | 14,712 | 3-class as-is (8 languages) |
|
| 30 |
+
|
| 31 |
+
## Hyperparameters
|
| 32 |
+
|
| 33 |
+
- 2 epochs (5,436 steps), batch 32, max length 128, bf16
|
| 34 |
+
- lr 3e-5, linear schedule, warmup 543 steps, weight decay 0.01
|
| 35 |
+
- Attention: PyTorch SDPA (flash-attention kernels) on A10G
|
| 36 |
+
- Best checkpoint selected by combined-validation macro-F1
|
| 37 |
+
|
| 38 |
+
## Results (best checkpoint)
|
| 39 |
+
|
| 40 |
+
| Split | macro-F1 | Accuracy |
|
| 41 |
+
|---|---:|---:|
|
| 42 |
+
| Combined validation | **0.7005** | 0.7044 |
|
| 43 |
+
| tweet_eval sentiment test | 0.7089 | 0.7116 |
|
| 44 |
+
| super_tweeteval tweet_sentiment test (folded to 3-class) | 0.6596 | 0.6682 |
|
| 45 |
+
| tweet_sentiment_multilingual test | 0.6908 | 0.6902 |
|
| 46 |
+
|
| 47 |
+
Note: the super_tweeteval test score is on labels folded to 3 classes and is **not** comparable to the official SuperTweetEval leaderboard metric (`1 − MAE^M` on the 5-point scale).
|
| 48 |
+
|
| 49 |
+
## Usage
|
| 50 |
+
|
| 51 |
+
```python
|
| 52 |
+
from transformers import pipeline
|
| 53 |
+
|
| 54 |
+
clf = pipeline("text-classification", model="Ido-shraga/mmbert-base-sentiment-adapted")
|
| 55 |
+
clf("Absolutely loving the new update, great work!")
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
## References
|
| 59 |
+
|
| 60 |
+
- mmBERT: arXiv 2509.06888
|
| 61 |
+
- TweetEval: arXiv 2010.12421 · SuperTweetEval: arXiv 2310.14757
|