# Necessary resources for training a (small/tiny) LM from scratch?

**URL:** <https://discuss.huggingface.co/t/necessary-resources-for-training-a-small-tiny-lm-from-scratch/8139>\
**Category:** Beginners\
**Created:** [July 7, 2021, 12:21pm UTC](https://discuss.huggingface.co/t/necessary-resources-for-training-a-small-tiny-lm-from-scratch/8139 "2021-07-07T12:21:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![brijow](https://avatars.discourse-cdn.com/v4/letter/b/cab0a1/32.png) [@brijow](https://discuss.huggingface.co/u/brijow)\
**Post date:** [July 7, 2021, 12:21pm UTC](https://discuss.huggingface.co/t/necessary-resources-for-training-a-small-tiny-lm-from-scratch/8139/1 "2021-07-07T12:21:51Z")

</div>

This is a follow up question in regards to this hugging face blog post on training a LM (and a tokenizer) from scratch :

> **[How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

I’d like to try this approach, but I’m wondering about cost and how much data I really need to train a (small) LM from scratch on my domain-specific dataset, and what LM maybe is a decent starting point for POC? I’m quite new to the field and haven’t read many papers on this subject as of yet, so I was hoping someone might be able to provide some ballpark estimates about computing resources required for training some small LM(s) from scratch? I’d like to obtain a domain-specific LM to serve as a backbone for various downstream NLP tasks on my domain-specific text data. I have been experimenting with the fine-tuning LM approach, (i.e. fine-tuning BERT based models on MLM before performing task-specific fine-tuning), but I’m curious about the training from scratch option if I can get a rough idea the required compute resources / cost.

Thanks very much in advance for any help / tips on unpacking this question.
