# Gradient accumulation: should I duplicate data? **URL:** \ **Category:** 🤗Transformers\ **Created:** [January 13, 2021, 10:18am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172 "2021-01-13T10:18:49Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\ **Post date:** [January 13, 2021, 10:18am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/1 "2021-01-13T10:18:50Z")
Hello! I am using gradient accumulation to simulate bigger batches when fine-tuning. However, I remember to have seen some [notebooks in the documentation](https://huggingface.co/transformers/notebooks.html) where they would make N copies of the data when N is the number of gradient accumulation steps. I do not understand why this should be done. Is this good practice? Why? Thank you 🙂 ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [January 13, 2021, 11:05am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/2 "2021-01-13T11:05:16Z")
Could you link to the exact notebook where you have seen this? ---
**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\ **Post date:** [January 13, 2021, 11:25am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/3 "2021-01-13T11:25:42Z")
Hey @BramVanroy, thank you for your reply. I have found the notebook, sorry for not being very precise. It’s [Reformer - Pushing the Limits of Language Modeling](https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb). Around box 7 it says: > We then expand the same sample to 8 training samples so that we can accumulate gradients during training. In the code ``` # duplicate data 8 times to have have 8 examples in dataset for key in input_ids_dict.keys(): input_ids_dict[key] = [8 * [x] for x in input_ids_dict[key]][0] ``` And gradient accumulation steps is actually 4, not 8 as I would expect. With batch size 1. ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [January 13, 2021, 12:29pm UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/4 "2021-01-13T12:29:34Z")
Hm, good question. Tagging @patrickvonplaten who created that notebook. ---
**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\ **Post date:** [January 18, 2021, 8:02am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/5 "2021-01-18T08:02:55Z")
Yes, would be helpful to have an update from you @patrickvonplaten ---
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [January 19, 2021, 3:38pm UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/6 "2021-01-19T15:38:32Z")
Ideally gradient accumulation has nothing to do with data . It’s basically , in storage memory of few epochs and then do gradient update, which will have an effect of larger batch size. ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [February 1, 2021, 6:47am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/7 "2021-02-01T06:47:02Z")
Yeah this might have been a bit unprecise in the notebook. So the reason I’m expanding the training data from 1 to 8 samples is a super edge-case. Since Reformer processes the **whole** train dataset in 1 batch, there is only one data sample in the whole dataset. Then if one uses gradient\_accumulation (which as pointed out correctly has nothing to do with data replication) there is a bug if the data set is of size 1 because the training script rightfully expects the dataset to at least have \> 1 training samples when gradient\_accumulation is used. So my solution of expanding the dataset is more of a hack than the recommended way of doing it (actually one should never copy samples from the dataset). I doubt there is any real application of having a dataset of batch 1 =\> this notebook was more of a show-off that Reformer can process the whole dataset in 1 batch, so not super relevant for real scenarios. ---
**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\ **Post date:** [February 1, 2021, 8:33am UTC](https://discuss.huggingface.co/t/gradient-accumulation-should-i-duplicate-data/3172/8 "2021-02-01T08:33:11Z")
This is clear now, thank you!