# Using custom models (not necessarily transformer based) with generate() and sampling

**URL:** <https://discuss.huggingface.co/t/using-custom-models-not-necessarily-transformer-based-with-generate-and-sampling/15168>\
**Category:** Beginners\
**Created:** [February 26, 2022, 9:38pm UTC](https://discuss.huggingface.co/t/using-custom-models-not-necessarily-transformer-based-with-generate-and-sampling/15168 "2022-02-26T21:38:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SamPr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@SamPr](https://discuss.huggingface.co/u/SamPr)\
**Post date:** [February 26, 2022, 9:38pm UTC](https://discuss.huggingface.co/t/using-custom-models-not-necessarily-transformer-based-with-generate-and-sampling/15168/1 "2022-02-26T21:38:18Z")

</div>

I’m comparing a number of Seq2Seq models for a particular (non NLP) task. I’ve got a version of this working in pure PyTorch, but I’m interested in porting the research to HuggingFace to take advantage of the existing sampling and beam search capabilities. Most of my existing models are autoregressive LMs, but I also have an autoregressive LSTM baseline that would need to be ported to use the same sampling infrastructure.

Is this practical, or am I going to run into problems trying to use a non transformer model in Transformers?

I assume that the way to do this is to derive from `PreTrainedModel` and return `CausalLMOutput`, but some of the optional parameters on the output (particularly `attentions`) are transformer specific. Am I safe leaving these as `None`, or are there going to be times when the generation infrastructure relies on these values? Is there a better way to make use of Transformers’ generation/sampling/beam capabilities with custom models?

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](https://discuss.huggingface.co/u/dblakely)\
**Post date:** [February 28, 2022, 6:03pm UTC](https://discuss.huggingface.co/t/using-custom-models-not-necessarily-transformer-based-with-generate-and-sampling/15168/2 "2022-02-28T18:03:15Z")

</div>

I think what you’re trying to do is feasible. As you mentioned, you could inherit `PreTrainedModel`, but could perhaps also just inherit [GenerationMixin](https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L378). You’d also want to make sure the `forward` method of your custom class resembles the `forward` methods of Huggingface models.

> but some of the optional parameters on the output (particularly `attentions` ) are transformer specific. Am I safe leaving these as `None`, or are there going to be times when the generation infrastructure relies on these values?

You should be fine setting those to `None`. Most of the things in the return objects are only there so users can access them if desired. In fact, the `attentions` field of the return objects is usually `None`, because it’s controlled by the `output_attentions` arg, which is `False` by default.

---

<div class="post-metadata">

**Author:** ![SamPr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@SamPr](https://discuss.huggingface.co/u/SamPr)\
**Post date:** [March 1, 2022, 4:59am UTC](https://discuss.huggingface.co/t/using-custom-models-not-necessarily-transformer-based-with-generate-and-sampling/15168/3 "2022-03-01T04:59:55Z")

</div>

Just inheriting `GenerationMixin` is a nice idea! I’ve got a version based on `PreTrainedModel` implemented now, but I haven’t made it fully compatible with everything that class expects.

I’ve opted encapsulate the whole `PreTrainedModel` class inside a `generate` function to hide the details and only expose the few features that I’ve actually implemented.