# FP16 doesn't reduce Trainer Training time **URL:** \ **Category:** Amazon SageMaker\ **Created:** [July 20, 2021, 1:12pm UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517 "2021-07-20T13:12:56Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [July 20, 2021, 1:12pm UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/1 "2021-07-20T13:12:56Z")
Hi, I’m using this SageMaker HF sample [notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/master/sagemaker/01_getting_started_pytorch/sagemaker-notebook.ipynb) adjusted with train\_batch\_size = 128, tested on both 1 p3.16xlarge and 1 p4d.24xlarge. For each instance I’m doing a job with fp16=True and a job without the flag. GPU usage is a erratic (sawtooth oscillating between 50 and 750%) **The impact of fp16=True is only a 1% training time reduction, on each instance**. Is it because: 1. Not specifying fp16 in the trainer already uses fp16? (seems to be [false](https://huggingface.co/transformers/main_classes/trainer.html) by default though) 2. There is a lot of CPU work & I/O in that demo that will not leverage float16? 3. Transformers don’t benefit from fp16 training? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [July 20, 2021, 1:29pm UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/2 "2021-07-20T13:29:36Z")
Hey @OlivierCR, can you please share all of your `hyperparameters` you used? `tran_batch_size = 128` seems pretty high to me to work. Are you using `distributed` training? The training time reduction should be way higher more around 20-40%. When using the example have you adjusted `train.py` to accept fp16 as hyperparameter or have you defined it directly in the script? ---
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [July 21, 2021, 7:46am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/3 "2021-07-21T07:46:16Z")
batch was 32 on 1 V100 so it’s ok to scale 4x on 4\*V100 right? my understanding is that `tran_batch_size` is the cluster-level batch, not the per-GPU batch right? here is my config. it ran fine both on p3.16x and p4d.24xl, and GPU mem is not 800% ```auto from sagemaker.huggingface import HuggingFace # hyperparameters, which are passed into the training job hyperparameters={'epochs': 3, 'train_batch_size': 128, 'model_name':'distilbert-base-uncased', 'fp16': True} huggingface_estimator = HuggingFace(entry_point='train.py', source_dir='./scripts', instance_type='ml.p4d.24xlarge', instance_count=1, role=role, transformers_version='4.6', pytorch_version='1.7', py_version='py36', hyperparameters = hyperparameters) ``` `train_batch_size = 256` also works fine, it is as fast as 128 on p3.16x and 7% faster than 128 on p4d.24xlarge (both still fp16) ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [July 21, 2021, 11:21am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/4 "2021-07-21T11:21:50Z")
No the `train_batch_size` is the batch\_size per device. See [notebooks/train.py at 6b8367544a374daf187854c6ba2274640fa0d7b4 · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/6b8367544a374daf187854c6ba2274640fa0d7b4/sagemaker/01_getting_started_pytorch/scripts/train.py#L64) or in the documentation here [Trainer — transformers 4.7.0 documentation](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments) This speaking `train_batch_size = 256` would have resulted in 256\*8 = 2048 which is not possible. Have you modified `train.py` otherwise `fp16:True` will not be used. Since it is not parsed here [notebooks/train.py at 6b8367544a374daf187854c6ba2274640fa0d7b4 · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/6b8367544a374daf187854c6ba2274640fa0d7b4/sagemaker/01_getting_started_pytorch/scripts/train.py#L16) ---
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [July 21, 2021, 1:21pm UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/5 "2021-07-21T13:21:36Z")
Well running train\_batch\_size = 256 on 1\*p3.16xlarge for me worked ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [July 22, 2021, 8:50am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/6 "2021-07-22T08:50:50Z")
Could you share the exact code/notebook you have used? I would like to reproduce it. ---
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [July 27, 2021, 1:58pm UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/7 "2021-07-27T13:58:25Z")
> **[notebooks/sagemaker/01\_getting\_started\_pytorch at master · huggingface/notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker/01_getting_started_pytorch)** > > master/sagemaker/01\_getting\_started\_pytorch ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [July 28, 2021, 6:47am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/8 "2021-07-28T06:47:04Z")
So you didn’t change anything? If so `fp16` couldn’t work because it is not parsed in `train.py`. Can you share the logs of your script? There you should be able to see which batch size was used by the `Trainer` ---
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [July 28, 2021, 8:30am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/9 "2021-07-28T08:30:02Z")
@philschmid I didn’t change anything apart from the config above in the estimator (batch size, fp16, instance type). Did you try and see that batch 256 works fine on p3.16x? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [July 28, 2021, 11:26am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/10 "2021-07-28T11:26:36Z")
Hey, found the issue. the `train.py` expected the hyperparamter to be `train-batch-size` and we always passed in `train_batch_size` so it used the default of `32` and not the values passed in. I fixed all of the examples and tested `256` and it failed. If you want to test `fp16` you would need to change the `train.py` ---
**Author:** ![sd3ntato](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sd3ntato](https://discuss.huggingface.co/u/sd3ntato)\ **Post date:** [June 29, 2023, 9:13am UTC](https://discuss.huggingface.co/t/fp16-doesnt-reduce-trainer-training-time/8517/11 "2023-06-29T09:13:59Z")
Hi, So how would I change the code to enable fp16 computation in this case?