# Summarization on long documents

**URL:** <https://discuss.huggingface.co/t/summarization-on-long-documents/920>\
**Category:** 🤗Transformers\
**Created:** [August 29, 2020, 5:57pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920 "2020-08-29T17:57:00Z")\
**Posts on this page:** 1\
**Showing post:** 30

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\
**Post date:** [December 2, 2020, 9:49am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/30 "2020-12-02T09:49:29Z")

</div>

Hey @MoritzLaurer @echatzikyriakidis

The regex `.(?!\d)|\n` works for Python, it just says to split where there is a full stop (not followed by a number, to avoid splitting floating points) or a new line. Consider changing it to what’s more suitable to you. For example I do not have any URL in my text, otherwise it would be a problem.

`num_tok` is the number of tokens of the entire text `text`.

The `tokenizer` is either Bart or Pegasus, works for both. I use the `tokenize` function so that i do not get BOS and EOS for each sentence.

---

_[View the full topic](https://discuss.huggingface.co/t/summarization-on-long-documents/920)._