# Token Classification Models on (Very) Long Text **URL:** \ **Category:** Models\ **Created:** [August 30, 2022, 7:58pm UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318 "2022-08-30T19:58:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![nottakumasato](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@nottakumasato](https://discuss.huggingface.co/u/nottakumasato)\ **Post date:** [August 30, 2022, 7:58pm UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/1 "2022-08-30T19:58:34Z")
Hi everyone, From what I have seen, most token classification models out there have max token lengths less than 1k. Are there any models out there that can be used (i.e. customized) to be used with very long texts (long-form documents? Assuming a model’s max token length is customizable, I assume its memory footprint has to be light for it to be able to batch a large number of embeddings&weights in GPU memory? Any help/recommendation would be greatly appreciated in tackling this problem. Thanks! ---
**Author:** ![ccdv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ccdv/32/28048_2.png) [@ccdv](https://discuss.huggingface.co/u/ccdv)\ **Post date:** [August 31, 2022, 1:25pm UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/2 "2022-08-31T13:25:02Z")
HI @nottakumasato Most models have a 512 tokens limit and cannot extrapolate to longer sequences. Memory footprint also increases quadratically with sequence length because standard attention is O(n²). Best way to handle long sequences is to use a custom attention mecanism. You can try this [repo](https://github.com/ccdv-ai/convert_checkpoint_to_lsg) with a small model and a small block size, you should be able to process 16k tokens sequences. ---
**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](https://discuss.huggingface.co/u/AndreaSottana)\ **Post date:** [August 31, 2022, 3:43pm UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/3 "2022-08-31T15:43:57Z")
The BART model goes up to 1024 tokens. Then there are models which can take up to 16k tokens but they’re more custom and not always available out of the box on HuggingFace. One of these is the [Longformer](https://arxiv.org/pdf/2004.05150.pdf) for example. Their model can be accessed via HuggingFace as shown [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18551/en/model_doc/longformer). You may also want to take a look at [this recent paper](https://arxiv.org/pdf/2208.04347.pdf) from Google. It is a model specific for text generation (not exactly classification as you asked, but gives you an idea for what’s possible) and they have also made their code available (you can see more details [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18551/en/model_doc/pegasus_x) and [here](https://github.com/huggingface/transformers/pull/18551) - there is still an open PR which will be merged into the main HuggingFace branch soon, so right now you’d have to take their code from the fork) ---
**Author:** ![nottakumasato](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@nottakumasato](https://discuss.huggingface.co/u/nottakumasato)\ **Post date:** [September 2, 2022, 10:38am UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/4 "2022-09-02T10:38:54Z")
Thank you for the replies @ccdv & @AndreaSottana ! So I guess there are two ways to tackle this: 1. Split up the input text into segments that are less than the model’s max sequence (token) length 2. Find a model like Pegasus-X or Longformer that can handle all the samples (based on their sequence length) in my dataset Option #1 seems more plausible and will give it a try. Thanks! ---
**Author:** ![nottakumasato](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@nottakumasato](https://discuss.huggingface.co/u/nottakumasato)\ **Post date:** [September 20, 2022, 1:33am UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/5 "2022-09-20T01:33:49Z")
Is it also possible to use RNN (non-Transformer) based models? I assume the tradeoff is model “accuracy” vs the max sequence length? ---
**Author:** ![nottakumasato](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@nottakumasato](https://discuss.huggingface.co/u/nottakumasato)\ **Post date:** [September 20, 2022, 3:20am UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/6 "2022-09-20T03:20:16Z")
> Best way to handle long sequences is to use a custom attention mecanism. Is there a specific reason that you didn’t recommend using earlier RNN-based models? Since they don’t have an attention mechanism, their memory footprint should theoretically be linear to the sequence lenght, right? ---
**Author:** ![nottakumasato](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@nottakumasato](https://discuss.huggingface.co/u/nottakumasato)\ **Post date:** [September 20, 2022, 3:21am UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/7 "2022-09-20T03:21:25Z")
> You can try this [repo](https://github.com/ccdv-ai/convert_checkpoint_to_lsg) Is there a paper about this LSG attention mechanism? Looks interesting and any further info would be appreciated to understand it a bit more. ---
**Author:** ![Vignesh1997](https://avatars.discourse-cdn.com/v4/letter/v/3be4f8/32.png) [@Vignesh1997](https://discuss.huggingface.co/u/Vignesh1997)\ **Post date:** [March 7, 2023, 6:22am UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/8 "2023-03-07T06:22:53Z")
Hey did you try the paper if possible can you share us the results of how did it work ---
**Author:** ![nottakumasato](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@nottakumasato](https://discuss.huggingface.co/u/nottakumasato)\ **Post date:** [March 9, 2023, 5:30pm UTC](https://discuss.huggingface.co/t/token-classification-models-on-very-long-text/22318/9 "2023-03-09T17:30:55Z")
Unfortunately not