# Generation Config for ByT5

**URL:** <https://discuss.huggingface.co/t/generation-config-for-byt5/40490>\
**Category:** 🤗Transformers\
**Created:** [May 20, 2023, 8:10pm UTC](https://discuss.huggingface.co/t/generation-config-for-byt5/40490 "2023-05-20T20:10:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![salokr](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@salokr](https://discuss.huggingface.co/u/salokr)\
**Post date:** [May 20, 2023, 8:10pm UTC](https://discuss.huggingface.co/t/generation-config-for-byt5/40490/1 "2023-05-20T20:10:19Z")

</div>

I am training a ByT5 model for a generation task. My initial observation was that there are a lots of repeats and the generation quality is bad. So I started doing a contrastive search mentioned here: [https://huggingface.co/blog/introducing-csearch](https://huggingface.co/blog/introducing-csearch). But t seems I cannot override the generation config…

The steps I follow are:  
I initialize the model using:

```auto
model = AutoModelForSeq2SeqLM.from_pretrained(
        model_args.model_name_or_path,
        config=config,
        cache_dir=model_args.cache_dir
    )

```

Next, my trainer looks like this:

```auto
trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        train_dataset = train_dataset['train'] if training_args.do_train else None,
        eval_dataset = eval_dataset['train'] if training_args.do_eval else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

```

in the above, I try to override the generation configuration using:

```auto
gc = GenerationConfig.from_pretrained("google/byt5-base")
gc.penalty_alpha=0.6
gc.top_k=4 
training_args.generation_config = gc

```

But whenever logs are printed I get warning like:

`You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)`

and when the configuration files are saved, I get:

```auto
{
  "_from_model_config": true,
  "decoder_start_token_id": 0,
  "eos_token_id": 1,
  "pad_token_id": 0,
  "transformers_version": "4.29.2"
}

```

and my generation still has lots of repetitions and bad quality. How can I use the generation config properly? @sgugger