Add article interlinks to the output of `gensim.scripts.segment_wiki`. Fix #1712 by steremma · Pull Request #1839 · piskvorky/gensim

steremma · 2018-01-13T17:39:35Z

Interlinks

This PR adds the feature requested in #1712
The segment_wiki script now includes the interlinks (links to other articles in the dump) found in each article it processes.

Implementation

The interlink markup parsing takes place in the filtered text rather than the raw one in order to avoid code and logic duplication (similar markup would need to be removed again which is already done in filter_wiki). In order to achieve this the function filter_wiki must not promote the uncaught markup to plain text. The usefulness of this operation was already doubted (see comment in old version) so I removed the promotion altogether:

# the following is needed to make the tokenizer see '[[socialist]]s' as a single word 'socialists'
# TODO is this really desirable?
text = text.replace('[', '').replace(']', '')

Removing this line was important because:

Phrases like "Computer Science" which has a meaning as a phrase will lose its meaning if its promoted to plain text (2 tokens)
the [[ and ]] are the markup used for interlinks. So if we promote them then we lose this information

Since the output of the script and its helper functions changed, I modified the documentation (source comments) accordingly.

* New output format is (str, list of (str, str), list of str, reflecting structure (title, [(section_heading, section_content), ...], [interlink, ...]) * `filter_wiki` in WikiCorpus will not promote uncaught markup to plain text as this will give up valuable information for the interlink discovery

menshikh-iv

Great start @steremma!
It will be really nice if you add several simple tests for segment_wiki script (you already have small wiki dump as test data)

menshikh-iv · 2018-01-15T06:30:18Z

-        for idx, (article_title, article_sections) in enumerate(article_stream):
-            output_data = {"title": article_title, "section_titles": [], "section_texts": []}
+        for idx, (article_title, article_sections, article_interlinks) in enumerate(article_stream):
+            output_data = {"title": article_title,


please use hanging indents (instead of vertical)

menshikh-iv · 2018-01-15T06:31:14Z


    logger.info("finished running %s", sys.argv[0])
+
+    print("-----Now checking output--------\n\n\n")


Why? This isn't needed here.

menshikh-iv · 2018-01-15T06:33:55Z

+            output_data = {"title": article_title,
+                           "section_titles": [],
+                           "section_texts": [],
+                           "section_interlinks": article_interlinks


If you don't split interlinks by sections, this should name as "interlinks" instead of "section_interlinks".

…re disregarded * Due to preprocessing in `filter_wiki` interlinks containing alternative names had one of the 2 `[` and `]` characters removed. The regex now takes that into account.

* Initiate unit testing for all scripts. * Check for expected len given article filtering (namespace, size in characters and redirections). * Check for yielded title, section headings and texts as well as interlinks yielded from generator. * Check that the same is correctly persisted in JSON. * Fix PEP 8

* Refactored filtering functions in ``wikicorpus.py` so that uncaught markup can be optionally promoted to plain text * Interlink extraction logic moved to `wikicorpus.py` * Unit tests modified accordingly

…interlinks

napsternxg · 2018-01-19T19:05:45Z

+    Find all interlinks to other articles in the dump. `raw` is either unicode
+    or utf-8 encoded string.
+    """
+    interlink_regex_capture = r"\[{1,2}(.*?)\]{1,2}"


I think this regex can be compiled outside the function.

…ual interlink text * Used boolean argument with default argument in `filter_wiki`. The default value keeps the old functionality so that existing code does not brake * Overriding the default argument causes interlinks to not be simplified and lets `find_interlinks` create the mappings

menshikh-iv · 2018-01-22T11:40:03Z

@steremma please don't forget to

resolve the merge conflict
run segment_wiki on full wiki dump (with python2 and python3), this needs some time

…mand line argument

…into interlinks * Kept documentation improvements from upstream * Kept interlink support and updated signatures from my branch * Added documentation from my extra arguments in correct format

menshikh-iv · 2018-01-31T05:29:35Z

Looks great, thank you @steremma!

steremma · 2018-01-31T10:19:15Z

👍

…Fix piskvorky#1712 (piskvorky#1839) * promoting the markup gives up information needed to find the intelinks * Add interlinks to the output of `segment_wiki` * New output format is (str, list of (str, str), list of str, reflecting structure (title, [(section_heading, section_content), ...], [interlink, ...]) * `filter_wiki` in WikiCorpus will not promote uncaught markup to plain text as this will give up valuable information for the interlink discovery * Fixed PEP 8 * Refactoring identation and variable names * Removed debugging code from script * Fixed a bug where interlinks with a description or multiple names where disregarded * Due to preprocessing in `filter_wiki` interlinks containing alternative names had one of the 2 `[` and `]` characters removed. The regex now takes that into account. * Now stripping whitespace off section titles * Unit test `gensim.scripts.segment_wiki` * Initiate unit testing for all scripts. * Check for expected len given article filtering (namespace, size in characters and redirections). * Check for yielded title, section headings and texts as well as interlinks yielded from generator. * Check that the same is correctly persisted in JSON. * Fix PEP 8 * Fix Python 3.5 compatibility * Section text now completely clean from wiki markup * Refactored filtering functions in ``wikicorpus.py` so that uncaught markup can be optionally promoted to plain text * Interlink extraction logic moved to `wikicorpus.py` * Unit tests modified accordingly * Added extra logging info to troublehsoot weird Travis behavior * Fix PEP 8 * pin workers for segment_and_write_all_articles * Get rid of debugging stuff * Get rid of global logger * Interlinks are now mapping from the linked article's title to the actual interlink text * Used boolean argument with default argument in `filter_wiki`. The default value keeps the old functionality so that existing code does not brake * Overriding the default argument causes interlinks to not be simplified and lets `find_interlinks` create the mappings * Moved regex outside function * Interlink extraction is now optional and controlled with the `-i` command line argument * PEP 8 long lines * made scripts tests aware of the optional interlinks argument * Updated script help output for interlinks

steremma added 3 commits January 13, 2018 17:49

promoting the markup gives up information needed to find the intelinks

f90cd9c

Fixed PEP 8

acc5221

menshikh-iv suggested changes Jan 15, 2018

View reviewed changes

steremma added 7 commits January 15, 2018 19:48

Refactoring identation and variable names

0057c7b

Removed debugging code from script

107d7f7

Fixed a bug where interlinks with a description or multiple names whe…

4adcf86

…re disregarded * Due to preprocessing in `filter_wiki` interlinks containing alternative names had one of the 2 `[` and `]` characters removed. The regex now takes that into account.

Now stripping whitespace off section titles

9bf6b87

Fix Python 3.5 compatibility

cd37315

Section text now completely clean from wiki markup

c681a60

* Refactored filtering functions in ``wikicorpus.py` so that uncaught markup can be optionally promoted to plain text * Interlink extraction logic moved to `wikicorpus.py` * Unit tests modified accordingly

steremma closed this Jan 16, 2018

steremma reopened this Jan 16, 2018

steremma and others added 6 commits January 16, 2018 13:54

Added extra logging info to troublehsoot weird Travis behavior

ead5386

Fix PEP 8

193861c

pin workers for segment_and_write_all_articles

e170c06

Merge branch 'interlinks' of https://github.com/steremma/gensim into …

b68507b

…interlinks

Get rid of debugging stuff

0884f6d

Get rid of global logger

58f63ca

menshikh-iv changed the title ~~Add article interlinks to the output of gensim.scripts.segment_wiki()~~ Add article interlinks to the output of gensim.scripts.segment_wiki. Fix #1712 Jan 18, 2018

napsternxg reviewed Jan 19, 2018

View reviewed changes

steremma added 2 commits January 20, 2018 19:56

Moved regex outside function

3b13d3b

steremma closed this Jan 21, 2018

steremma reopened this Jan 21, 2018

steremma added 3 commits January 25, 2018 19:42

Interlink extraction is now optional and controlled with the -i com…

e038f52

…mand line argument

Merge branch 'develop' of https://github.com/RaRe-Technologies/gensim …

68ca8b1

…into interlinks * Kept documentation improvements from upstream * Kept interlink support and updated signatures from my branch * Added documentation from my extra arguments in correct format

PEP 8 long lines

94c2b3d

steremma closed this Jan 25, 2018

steremma reopened this Jan 25, 2018

steremma and others added 2 commits January 25, 2018 21:38

made scripts tests aware of the optional interlinks argument

3c838a6

Updated script help output for interlinks

7f9ed71

menshikh-iv merged commit aa10f79 into piskvorky:develop Jan 31, 2018

pyup-bot mentioned this pull request Feb 5, 2018

Scheduled weekly dependency update for week 05 workforce-data-initiative/skills-ml#120

Closed

steremma deleted the interlinks branch March 1, 2018 10:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add article interlinks to the output of `gensim.scripts.segment_wiki`. Fix #1712#1839

Add article interlinks to the output of `gensim.scripts.segment_wiki`. Fix #1712#1839
menshikh-iv merged 23 commits into
piskvorky:developfrom
steremma:interlinks

steremma commented Jan 13, 2018 •

edited

Loading

Uh oh!

menshikh-iv left a comment •

edited

Loading

Uh oh!

menshikh-iv Jan 15, 2018

Uh oh!

menshikh-iv Jan 15, 2018

Uh oh!

menshikh-iv Jan 15, 2018

Uh oh!

napsternxg Jan 19, 2018

Uh oh!

menshikh-iv commented Jan 22, 2018

Uh oh!

menshikh-iv commented Jan 31, 2018

Uh oh!

steremma commented Jan 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		logger.info("finished running %s", sys.argv[0])

		print("-----Now checking output--------\n\n\n")

Uh oh!

Conversation

steremma commented Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Interlinks

Implementation

Uh oh!

menshikh-iv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

menshikh-iv Jan 15, 2018

Choose a reason for hiding this comment

Uh oh!

menshikh-iv Jan 15, 2018

Choose a reason for hiding this comment

Uh oh!

menshikh-iv Jan 15, 2018

Choose a reason for hiding this comment

Uh oh!

napsternxg Jan 19, 2018

Choose a reason for hiding this comment

Uh oh!

menshikh-iv commented Jan 22, 2018

Uh oh!

menshikh-iv commented Jan 31, 2018

Uh oh!

steremma commented Jan 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

steremma commented Jan 13, 2018 •

edited

Loading

menshikh-iv left a comment •

edited

Loading