# NLP Pretrained model model doesn’t use GPU when making inference **URL:** \ **Category:** 🤗Transformers\ **Created:** [October 7, 2020, 6:01am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395 "2020-10-07T06:01:36Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![yashugupta786](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yashugupta786](https://discuss.huggingface.co/u/yashugupta786)\ **Post date:** [October 7, 2020, 6:01am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/1 "2020-10-07T06:01:36Z")
I am using Marian MT Pretrained model for Inference for machine Translation task integrated with a flask Service . I am running the Model on Cuda enabled device .While inferencing the model not using the GPU ,it is using the CPU only .I don’t want to use the cpu for inference as it is taking very long time for processing the request. Even if i am passing 1 sentence it is taking very long . Please help on this . Below is the code snippet and model i am using model\_name = ‘Helsinki-NLP/opus-mt-ROMANCE-en’ tokenizer = MarianTokenizer.from\_pretrained(model\_name) print(tokenizer.supported\_language\_codes) model = MarianMTModel.from\_pretrained(model\_name) translated = model.generate(\*\*tokenizer.prepare\_translation\_batch(src\_text)) tgt\_text = [tokenizer.decode(t, skip\_special\_tokens=True) for t in translated] I have downloaded the pytorch model.bin and other tokenizer files from s3 environment and saved on my local …Please help on this how i can put the things on GPU for faster inference ---
**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\ **Post date:** [October 7, 2020, 6:09am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/2 "2020-10-07T06:09:56Z")
Have you tried model.to(‘cuda’), to make the model use the GPU? ---
**Author:** ![yashugupta786](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yashugupta786](https://discuss.huggingface.co/u/yashugupta786)\ **Post date:** [October 7, 2020, 6:14am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/3 "2020-10-07T06:14:16Z")
Hi Karthik Thanks for replying yes i have used please find below the snippet and please correct me where i am doing wrong **torch\_device = ‘cuda’ if torch.cuda.is\_available() else 'cpu’** **print(torch\_device)** model\_name = ‘Helsinki-NLP/opus-mt-ROMANCE-en’ tokenizer = MarianTokenizer.from\_pretrained(model\_name) print(tokenizer.supported\_language\_codes) model = MarianMTModel.from\_pretrained(model\_name)**.to(torch\_device)** translated = model.generate(\*\*tokenizer.prepare\_translation\_batch(src\_text)) tgt\_text = [tokenizer.decode(t, skip\_special\_tokens=True) for t in translated] Thanks in advance ---
**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\ **Post date:** [October 7, 2020, 6:39am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/4 "2020-10-07T06:39:46Z")
You have the model on GPU but how about the tokenizer? Change this line: translated = model.generate(\*\*tokenizer.prepare\_translation\_batch(src\_text)) To: translated = model.generate(\*\*tokenizer.prepare\_translation\_batch(src\_text).to(‘cuda’)) ---
**Author:** ![yashugupta786](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yashugupta786](https://discuss.huggingface.co/u/yashugupta786)\ **Post date:** [October 7, 2020, 6:48am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/5 "2020-10-07T06:48:22Z")
Sure kartik thanks i will check also should i also change this below line tokenizer = MarianTokenizer.from\_pretrained(model\_name).to(‘cuda’) or translated = model.generate(\*\*tokenizer.prepare\_translation\_batch(src\_text).to(‘cuda’)) only to the above one which you told ---
**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\ **Post date:** [October 7, 2020, 6:51am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/6 "2020-10-07T06:51:54Z")
tokenizer = MarianTokenizer.from\_pretrained(model\_name).to(‘cuda’) - Do you get an error here, like, "‘MarianTokenizer’ object has no attribute ‘to’? If so, you can give as I have mentioned. Try it out. ---
**Author:** ![yashugupta786](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yashugupta786](https://discuss.huggingface.co/u/yashugupta786)\ **Post date:** [December 2, 2020, 6:53am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/7 "2020-12-02T06:53:08Z")
hi @Karthik12 i am able to use the gpu ,but inference is very slow and time consuming .Is there a way to make the inference fast .I have the done the same changes as you suggested still the inference is very slow and it takes time to process one request ---
**Author:** ![fciannel](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@fciannel](https://discuss.huggingface.co/u/fciannel)\ **Post date:** [December 30, 2020, 9:42pm UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/8 "2020-12-30T21:42:15Z")
Hi guys, I am having the same issue, did you figure out what the issue is? The execution is very slow, althugh the model seems to be on a GPU. It used to be different before on GPU, although I don’t recall exactly the transformers version I was using. ---
**Author:** ![R00](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r00/32/2009_2.png) [@R00](https://discuss.huggingface.co/u/R00)\ **Post date:** [June 27, 2021, 4:53pm UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/9 "2021-06-27T16:53:48Z")
Hello, have you managed to solve the issue of slow translation ? ---
**Author:** ![yashugupta786](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yashugupta786](https://discuss.huggingface.co/u/yashugupta786)\ **Post date:** [November 23, 2021, 11:48am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/10 "2021-11-23T11:48:30Z")
Still getting the same speed. Are you able to manage the faster inference ? ---
**Author:** ![yashugupta786](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yashugupta786](https://discuss.huggingface.co/u/yashugupta786)\ **Post date:** [November 23, 2021, 11:49am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/11 "2021-11-23T11:49:16Z")
are you able to manage the inference speed on gpu for marian ? ---
**Author:** ![raphaelmerx](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/raphaelmerx/32/56182_2.png) [@raphaelmerx](https://discuss.huggingface.co/u/raphaelmerx)\ **Post date:** [March 11, 2022, 6:30am UTC](https://discuss.huggingface.co/t/nlp-pretrained-model-model-doesn-t-use-gpu-when-making-inference/1395/12 "2022-03-11T06:30:28Z")
Inference is markedly faster for me on GPU, using the `device` option to `pipeline`: ```auto model_checkpoint = "Helsinki-NLP/opus-mt-ROMANCE-en" translator = pipeline("translation", model=model_checkpoint, device=0) ```