Reduce memory consumption of summarizer

[horpto]

By a request of @menshikh-iv:

• I found a couple of questions (this and this about memory of summarization when I was looking for an information related to my previous PR with graph nodes removing.
• I've taken first part of the english translated War and Peace. It consists of 14426 unique docs (number of nodes). I've tested on computer with 16 GiB of memory, Intel i5, 64-bit Windows 10, Python 3.7. Original version have eaten 14 GiB of memory after 7 minutes of execution and failed with MemoryError. My version have eaten 2 GiB of memory and taken ~2min (108 sec), but failed to process full version War and Peace (~57000 unique docs) with MemoryError too (failed on the phase pagerank).

[horpto]

I need #2263 to add some tests about graph.

[menshikh-iv]

#2263 merged, feel free to continue

[menshikh-iv]

Thanks for PR @horpto, can you also please update your first comment in PR with

• what was a reason for these changes (links to complaints about RAM)
• little benchmark (time & RAM) 3.6.0 vs summarization-refactoring to better understand, how this improved

[menshikh-iv]

In that case len(scores) <= self.corpus_size, why?

[horpto]

Because it's actually quite sparse array, isn't it?
In summarizer._set_graph_edge_weights such documents with little weight will be dropped anyway, so there is no reason to waste extra memory. And what's more, if we are needed a dense array we can uncompactify this bow.

[menshikh-iv]

I don't get, how we understand ids of documents that have 0 scores in that case?

[horpto]

Easy. Like words with 0 weight in bow. We have ids of docs with not zero weight. They are saved in bag-of-docs (I should rename function name part from bow - bag-of-weights to bod - bag-of-docs). If doc id isn't in bag-of-docs, so weight of doc is 0.

[menshikh-iv]

I've taken first part of the english translated War and Peace. It consists of 14426 unique docs (number of nodes). I've tested on computer with 16 GiB of memory, Intel i5, 64-bit Windows 10, Python 3.7. Original version have eaten 14 GiB of memory after 7 minutes of execution and failed with MemoryError. My version have eaten 2 GiB of memory and taken ~2min (108 sec), but failed to process full version War and Peace (~57000 unique docs) with MemoryError too (failed on the phase pagerank).

Awesome result, nice improvement for new release, thanks @horpto 👍