# Why is use\_cache incompatible with gradient checkpointing?

**URL:** <https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811>\
**Category:** 🤗Transformers\
**Created:** [June 7, 2022, 10:25am UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811 "2022-06-07T10:25:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ratishsp](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ratishsp](https://discuss.huggingface.co/u/ratishsp)\
**Post date:** [June 7, 2022, 10:25am UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/1 "2022-06-07T10:25:46Z")

</div>

Hi,  
I see the below snippet in modeling\_t5.py. I wanted to understand why use\_cache is incompatible with gradient checkpointing.

> <https://github.com/huggingface/transformers/blob/9aa230aa2fb17e4a7da4c61fccaddafe410f7ed1/src/transformers/models/t5/modeling_t5.py#L1008-L1011>

Thanks.

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [June 8, 2022, 5:46am UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/2 "2022-06-08T05:46:28Z")

</div>

Hi, there. I face the same problem in `run_clm.py` when I set `--gradient_checkpointing true`. However I do not find any config that I can set on `run_clm.py`, does anyone know?

---

<div class="post-metadata">

**Author:** ![luisarmando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/luisarmando/32/24205_2.png) [@luisarmando](https://discuss.huggingface.co/u/luisarmando)\
**Post date:** [October 27, 2022, 9:11pm UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/3 "2022-10-27T21:11:09Z")

</div>

> [@lianghsun](#):
>
> –gradient\_checkpointing true

Anybody knows how to fix this?

---

<div class="post-metadata">

**Author:** ![rs2992](https://avatars.discourse-cdn.com/v4/letter/r/6bbea6/32.png) [@rs2992](https://discuss.huggingface.co/u/rs2992)\
**Post date:** [March 24, 2023, 4:33pm UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/4 "2023-03-24T16:33:03Z")

</div>

Honestly, I’ve just ignored it. It automatically disables the use\_cache function. I’m about to remove the warning altogether.

`use_cache=True` is incompatible with gradient checkpointing. **Setting `use_cache=False`**

---

<div class="post-metadata">

**Author:** ![kira](https://avatars.discourse-cdn.com/v4/letter/k/c4cdca/32.png) [@kira](https://discuss.huggingface.co/u/kira)\
**Post date:** [July 7, 2023, 12:44pm UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/5 "2023-07-07T12:44:16Z")

</div>

its related to `past_key_values`, you can disable this warning by setting  
`model.config.use_cache = False`, when using gradient checkpointing during training.  
but during inference make sure to set it back to `True`.

---

<div class="post-metadata">

**Author:** ![mmiakashs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mmiakashs/32/1959_2.png) [@mmiakashs](https://discuss.huggingface.co/u/mmiakashs)\
**Post date:** [August 25, 2023, 9:32pm UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/6 "2023-08-25T21:32:46Z")

</div>

Could you please share what are the purposes of `use_cache`? Thanks.

---

<div class="post-metadata">

**Author:** ![kira](https://avatars.discourse-cdn.com/v4/letter/k/c4cdca/32.png) [@kira](https://discuss.huggingface.co/u/kira)\
**Post date:** [September 15, 2023, 1:39am UTC](https://discuss.huggingface.co/t/why-is-use-cache-incompatible-with-gradient-checkpointing/18811/7 "2023-09-15T01:39:01Z")

</div>

> **[Generation](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig.use_cache)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.