Add segment-wiki script#1483
Conversation
| os.write(sys.stdout.fileno(), u"\t".join(printed_components).encode('utf-8') + b"\n") | ||
|
|
||
|
|
||
| # noinspection PyUnresolvedReferences |
There was a problem hiding this comment.
Noise, please remove (here and elsewhere).
|
@menshikh-iv the original script was Python3 only -- has this been tested on Python2? We're aiming at dual compatibility (e.g. using |
|
@piskvorky of course, need to first check all another wiki scripts (as suggested in #1584), after it I'll add python2 compatibility here. |
piskvorky
left a comment
piskvorky
left a comment
There was a problem hiding this comment.
Minor code-style comments; great script!
How as this tested for scale/stability? I remember some issues with multiprocessing.
| article_title<tab>section_heading<tab>section_content<tab>section_heading<tab>section_content | ||
|
|
||
| """ | ||
| with open(output_file, 'wb') as outfile: |
| The documents are extracted on-the-fly, so that the whole (massive) dump | ||
| can stay compressed on disk. | ||
|
|
||
| >>> wiki = WikiCorpus('enwiki-20100622-pages-articles.xml.bz2') # create word->word_id mapping, takes almost 8h |
There was a problem hiding this comment.
Docstring out of date (different class).
There was a problem hiding this comment.
Done (updated all docstrings, converted to numpy-style, removed outdated things).
| Parse the content inside a page tag, returning its content as a list of tokens | ||
| (utf8-encoded strings). | ||
|
|
||
| Returns a 2-tuple (str, list) - |
There was a problem hiding this comment.
Neither google nor numpy docstring format.
|
|
||
| """ | ||
| elem = cElementTree.fromstring(page_xml) | ||
| filter_namespaces = ('0',) |
There was a problem hiding this comment.
Deserves a comment -- what is this?
|
About performance, approximately 5 minutes for 100,000 articles in the output file (SSD, i7 6700HQ, ruwiki), sometimes problems with |
| if self.lemmatize: | ||
| num_total_tokens += len(utils.lemmatize(section_content)) | ||
| else: | ||
| num_total_tokens += len(tokenize(section_content)) |
There was a problem hiding this comment.
Btw I think for the purposes of gensim-data, we shouldn't do any tokenization or normalization. We should present the sections "as they are", so people can use their own sentence detection / token detection etc. Only remove newlines and tabs just before printing, because of the output format.
It's easy to go from raw section_content => tokenize, but impossible to go from tokenize => raw. @menshikh-iv
There was a problem hiding this comment.
This is only for filtering very short articles, all content provided "as is".
tefimov
left a comment
tefimov
left a comment
There was a problem hiding this comment.
Sorry for misguidance, here's the right way.
| The documents are extracted on-the-fly, so that the whole (massive) dump can stay compressed on disk. | ||
|
|
||
| """ | ||
| def __init__(self, fileobj, processes=None, lemmatize=utils.has_pattern(), filter_namespaces=('0',)): |
There was a problem hiding this comment.
Don't mix __init__ and class annotations. I propose to annotate __init__ from now on.
| lemmatize : bool | ||
| If `pattern` package is installed, use fancier shallow parsing to get token lemmas. | ||
| Otherwise, use simple regexp tokenization. | ||
| filter_namespaces : tuple(int) |
|
|
||
| Yields | ||
| ------ | ||
| tuple(str, list of tuple(str, str)) |
|
|
||
| Returns | ||
| ------- | ||
| tuple(str, list of tuple(str, str)) |
|
|
||
| Yields | ||
| ------ | ||
| tuple(str, list of tuple(str, str)) |
* add segment wiki script * fix indentation error * Add output file and logging + small fixes * add smart_open * Add numpy-style docstrings & fix .rst * Fix types * Fix docstrings + output file format (json-lines) * Upd .rst
|
Continued in #1694 |
3 participants
Construct a corpus from a Wikipedia (or other MediaWiki-based) database dump and extract sections of pages from it.
CC @piskvorky