Skip to content

Remove outdated bz2 examples from tutorials#1867

Merged
menshikh-iv merged 1 commit into
developfrom
fix-outdated-bz2
Feb 1, 2018
Merged

Remove outdated bz2 examples from tutorials#1867
menshikh-iv merged 1 commit into
developfrom
fix-outdated-bz2

Conversation

@menshikh-iv

@menshikh-iv menshikh-iv commented Feb 1, 2018

Copy link
Copy Markdown
Contributor

MmReader support only filename as input (not file-like object), but in the old documentation (wiki.rst/dist_lsi.rst) we used file-like object too as input.

Current PR remove this outdated usage from examples.

Based on mailing list post

@menshikh-iv menshikh-iv merged commit 5342153 into develop Feb 1, 2018
@menshikh-iv menshikh-iv deleted the fix-outdated-bz2 branch February 1, 2018 05:39
Comment thread docs/src/dist_lsi.rst
>>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt')
>>> # load corpus iterator
>>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')
>>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this line doesn't sound right -- we still support bz2!

Just remove the (superfluous) bz2.BZFile wrapper. Dtto below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@piskvorky fixed in #1868

@piskvorky

Copy link
Copy Markdown
Owner

Why don't we support file-like objects in MmCorpus? That sounds like a bug, I see no reason.

@menshikh-iv

Copy link
Copy Markdown
Contributor Author

@piskvorky I see code for support file-like, but also see some buggy behavior, look at example

from gensim.corpora import MmCorpus
import bz2

f = bz2.BZ2File("testcorpus.mm.bz2")
print(f.closed)  # 0
corpus = MmCorpus(f)
print(f.closed)  # 1 ???

for this reason, if we try to read from this, we'll receive an exception suggested in mailing list.

@menshikh-iv

Copy link
Copy Markdown
Contributor Author

@piskvorky UPD, I found what's a reason for this behavior:
https://github.com/RaRe-Technologies/gensim/blob/5342153eb4f4b02bb45bfa3951eef8250ac9f6b6/gensim/matutils.py#L1274

in this line, we using with here, that automatically call .close when we exit from with scope, for this reason, after Mmreader.__init__ we closed passed file-descriptor.

This is a bug anyway (because internally we use file_or_filename, instead of simple smart_open -> It was assumed that everything should work for file-like)

menshikh-iv added a commit that referenced this pull request Feb 1, 2018
* Revert "Remove outdated `bz2` + `MmCorpus` examples from tutorials (#1867)"

This reverts commit 5342153.

* remove bz2 wrapper

* remove bz2 wrapper[2]
sj29-innovate pushed a commit to sj29-innovate/gensim that referenced this pull request Feb 21, 2018
sj29-innovate pushed a commit to sj29-innovate/gensim that referenced this pull request Feb 21, 2018
* Revert "Remove outdated `bz2` + `MmCorpus` examples from tutorials (piskvorky#1867)"

This reverts commit 5342153.

* remove bz2 wrapper

* remove bz2 wrapper[2]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants