[WIP] Gensim docker container by parulsethi · Pull Request #1368 · piskvorky/gensim

parulsethi · 2017-05-27T17:56:56Z

TODO:

Run gensim tests in docker container

parulsethi · 2017-05-27T17:58:34Z

+# Installs python, pip and setup tools (with fixed versions)
+RUN apt-get update \
+    && apt-get install -y \
+    ant=1.9.6-1ubuntu1 \


@tmylk @menshikh-iv Should I remove version pinning from here also?

If fixation is not required, then, of course, you can remove it

Pinning versions in a container is actually very good. It guarantees that it keeps running.

Just don't pin the gensim version.

So please keep the pins here

Guys, the fixation is needed due best practices with docker. Without these versions fixed the container can broke when a components changes it versions and when someone try to get it from a docker pull.

Another thing: In my previous PR I forgot to add a modification in dockerfile that fix some dependencies for git dependencies to avoid this problems.

I was following the docker contribution guidelines to turn this image into an official one in the future.

thanks @danielbdias, I've kept the pinned versions which were already there in your PR and will add version pinning for rest of the packages which I added later.

tmylk

The container is a very needed feature, just left some comments to make things more clear

tmylk · 2017-05-29T22:24:41Z

+    && mv ./gensim-$GENSIM_VERSION/* /gensim \
+    && rm -rf /gensim/download \
+    && cd /gensim \
+    && python setup.py install


please checkout master branch first

It will already download the package from branch which is specified in GENSIM_VERSION variable (and I use a branch on my personal fork for now which has the docker folder).

tmylk · 2017-05-29T22:30:12Z

+ENV WR_HOME gensim_dependencies/wordrank
+ENV MALLET_HOME gensim_dependencies/mallet
+ENV DTM_PATH gensim_dependencies/dtm/bin/dtm-linux64
+ENV VOWPAL_WABBIT_PATH gensim_dependencies/vowpal_wabbit/vowpalwabbit/vw


Need to add varembed path in order to run varembed tests.

According to test_varembed_wrapper.py varembed path is not required. It only tests on these test_data files.

tmylk · 2017-05-29T22:43:03Z

+import sys
+
+try:
+    from gensim.models.word2vec_inner import FAST_VERSION, MAX_WORDS_IN_BATCH


there is no need for max_words_in_batch

Removed in latest commit

…into gensim_docker

menshikh-iv · 2017-06-22T04:59:52Z

@parulsethi What's a status of PR? Have you any problems with container?

parulsethi · 2017-06-22T17:27:03Z

Update: Proper installation of wordrank dependencies is left now (need to install OpenMPI with multithreading enabled).
All other gensim tests (except wordrank) passes.

menshikh-iv · 2017-06-23T05:00:11Z

+
+ENV GENSIM_REPOSITORY https://github.com/parulsethi/gensim/archive
+ENV GENSIM_BRANCH gensim_docker
+


Add gensim version as env variable

menshikh-iv · 2017-06-23T05:03:07Z

+    && python3 setup.py install
+
+# Set ENV variables for wrappers
+ENV FT_HOME gensim_dependencies/fastText


Replace relative to absolute (for all *_PATH and *_HOME), it's very important.

menshikh-iv · 2017-06-23T05:08:30Z

+
+try:
+    from gensim.models.word2vec_inner import FAST_VERSION
+


Please add additional info for this script (like python version, numpy/scipy/gensim version) and create alias in docker for this script

It does run from here in dockerfile with both python 2 and 3. And only the pinned versions of numpy/scipy/gensim installed in the container are used

menshikh-iv · 2017-06-23T05:08:56Z

@@ -0,0 +1,7 @@
+version: '2'


Why you need docker-compose file?

menshikh-iv · 2017-06-23T05:10:25Z

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+printf "1. clean up workspace\n"


Why do you add this script here? This script from wordrank repo.

menshikh-iv · 2017-06-23T05:16:51Z

+
+# Install fastText
+RUN cd /gensim/gensim_dependencies \
+    && git clone https://github.com/facebookresearch/fastText.git \


Please pin version for FastText/Wordrank/etc (you can use commit hash or version)

menshikh-iv · 2017-06-28T17:15:30Z

+    && cd /gensim/gensim_dependencies/fastText \
+    && make
+
+# Install WordRank


Comment all things connected with wordrank in dockerfile (ompi problem)

menshikh-iv · 2017-06-28T17:16:56Z

+python2 setup.py test
+```
+
+To push the image to docker hub:


No needed this block (about push to dockerhub)

menshikh-iv · 2017-06-28T17:17:33Z

+docker push [my_user]/gensim
+```
+
+# Run gensim image from anywhere


Replace to Run ipython notebook with installed gensim

…into gensim_docker

menshikh-iv · 2017-06-30T10:48:00Z

Nice feature, thank you @parulsethi 🥇

* added dockerfile * remove fasttext from pip installs * remove syntax errors * remove unused imports * modified dockerfile * add subversion, locales * use both python2 and python3 * upgrade numpy version * add readme with relevant commands * add fixed versions for wrapper dependencies * made requested changes * update readme * change vw pin and remove docker-yml * change vw version and make absolute paths for wrappers * specify original gensim repo for download * change maintainer * correct missing slash * use git clone for gensim * correct gensim folder sequences

parulsethi added 4 commits May 27, 2017 16:12

added dockerfile

e148719

remove fasttext from pip installs

b5e27b5

remove syntax errors

e653d3f

remove unused imports

a40b1df

parulsethi commented May 27, 2017

View reviewed changes

parulsethi changed the title ~~Gensim docker container~~ [WIP] Gensim docker container May 27, 2017

menshikh-iv mentioned this pull request May 29, 2017

Added function to replace gensim check_output with subprocess.check_output #1182

Closed

tmylk suggested changes May 29, 2017

View reviewed changes

tmylk reviewed May 29, 2017

View reviewed changes

modified dockerfile

330b9ad

menshikh-iv mentioned this pull request Jun 13, 2017

Docker instructions in README.rst #579

Closed

parulsethi added 4 commits June 13, 2017 22:00

Merge branch 'develop' of https://github.com/RaRe-Technologies/gensim …

a371056

…into gensim_docker

add subversion, locales

800aae6

use both python2 and python3

498ae79

upgrade numpy version

9afaca7

menshikh-iv mentioned this pull request Jun 22, 2017

[WIP] Adding docker containers in Gensim. Fix #497 #1121

Closed

add readme with relevant commands

19ce3ca

menshikh-iv suggested changes Jun 23, 2017

View reviewed changes

menshikh-iv reviewed Jun 23, 2017

View reviewed changes

add fixed versions for wrapper dependencies

ee3c4cf

menshikh-iv reviewed Jun 28, 2017

View reviewed changes

parulsethi and others added 5 commits June 29, 2017 17:18

made requested changes

9fd041e

update readme

1cc3398

change vw pin and remove docker-yml

4825a47

Merge branch 'gensim_docker' of https://github.com/parulsethi/gensim …

e3a9699

…into gensim_docker

change vw version and make absolute paths for wrappers

d0f09bb

parulsethi added 5 commits June 30, 2017 14:45

specify original gensim repo for download

36a5025

change maintainer

2db28f4

correct missing slash

25865a9

use git clone for gensim

1add68d

correct gensim folder sequences

ce4656d

menshikh-iv merged commit bd6db9a into piskvorky:develop Jun 30, 2017


		ENV GENSIM_REPOSITORY https://github.com/parulsethi/gensim/archive
		ENV GENSIM_BRANCH gensim_docker

Uh oh!

Conversation

parulsethi commented May 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

parulsethi May 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmylk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

menshikh-iv commented Jun 22, 2017

Uh oh!

parulsethi commented Jun 22, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

menshikh-iv Jun 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

menshikh-iv Jun 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

menshikh-iv commented Jun 30, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

parulsethi commented May 27, 2017 •

edited

Loading

parulsethi May 27, 2017 •

edited

Loading

menshikh-iv Jun 28, 2017 •

edited

Loading

menshikh-iv Jun 28, 2017 •

edited

Loading