Conversation
nkemnitz
requested changes
Apr 22, 2021
Contributor
nkemnitz
left a comment
There was a problem hiding this comment.
objectscriberneeds to be added to the requirements- This line in
MiplessCloudVolume.__init__is problematic:Only the scheduler should create/overwrite the info file - right now all workers will try to rewrite the info file simultaneously, which is causing e.g. GCS to panic with 429 errors.corgie/corgie/mipless_cloudvolume.py
Lines 91 to 92 in d6a973a
Is there a good way to figure out if the job was initialized by the worker or the scheduler?
Contributor
|
Collaborator
Author
|
Thanks for the feedback! I'll address these issues asap |
Collaborator
Author
|
Added a global |
nkemnitz
requested changes
May 3, 2021
Contributor
nkemnitz
left a comment
There was a problem hiding this comment.
Just one typo - see suggestion
Besides objectscriber, does this PR also require a minimal mazepa version? If so, that should be specified.
Co-authored-by: Nico Kemnitz <nkemnitz@princeton.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrating the basic version of scriber into
corgieandmazepa.Verified to produce correct results with
NormalizationJobandAlignBlockJob.MergeRender tasks shrunk to 2KB
To test: use
mazepa-e git+https://github.com/supersergiy/scheduler.git@8905783b680cafcebfec52fec7f0e68e4be0e5fd#egg=mazepa`