# Reproduce RoBERTa Using Huggingface Transformers

**URL:** <https://discuss.huggingface.co/t/reproduce-roberta-using-huggingface-transformers/48531>\
**Category:** 🤗Transformers\
**Created:** [July 28, 2023, 3:30pm UTC](https://discuss.huggingface.co/t/reproduce-roberta-using-huggingface-transformers/48531 "2023-07-28T15:30:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JackBAI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jackbai/32/17827_2.png) [@JackBAI](https://discuss.huggingface.co/u/JackBAI)\
**Post date:** [July 28, 2023, 3:30pm UTC](https://discuss.huggingface.co/t/reproduce-roberta-using-huggingface-transformers/48531/1 "2023-07-28T15:30:01Z")

</div>

I’m trying to reproduce RoBERTa pre-training results using `run_mlm.py` provided by huggingface `transformers`. However, I’m confused how exactly should the script be called. Here is my script:

```bash
python transformers/examples/pytorch/language-modeling/run_mlm.py \
    --config_name roberta-base \
    --tokenizer_name roberta-base \
    --dataset_name wikitext, bookcorpus, ccnews, openwebtext, stories \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir ./crate/ckpt \
    --overwrite_output_dir

```

Is there anyone already successfully reproduced RoBERTa by this means? Looking forward to any possible help!