Skip to content

Terminate tee subprocesses before waiting#718

Closed
EspenHa wants to merge 1 commit intoIDSIA:masterfrom
EspenHa:terminate-tee
Closed

Terminate tee subprocesses before waiting#718
EspenHa wants to merge 1 commit intoIDSIA:masterfrom
EspenHa:terminate-tee

Conversation

@EspenHa
Copy link

@EspenHa EspenHa commented Feb 13, 2020

Send termination signals to tee_stdout and tee_stderr, before waiting.
This is needed because on some systems there seems to be deadlock issues related to using multiprocessing combined with fd capture. #705 (comment) shows an example of this.

This change reliably fixes the problem on the system I am using, and hopefully for others as well. There should be no ill effects from terminating the subprocesses because this is done after the
stdin pipe is closed.

Fixes #705

Send termination signals to tee_stdout and tee_stderr, before waiting.
This is needed because on some systems there seems to be deadlock
issues related to using multiprocessing combined with fd capture.
IDSIA#705 (comment)
shows an example of this.

This change reliably fixes the problem on the system I am using,
and hopefully for others as well. There should be no ill effects
from terminating the subprocesses because this is done after the
stdin pipe is closed.

Fixes IDSIA#705
@Qwlouse
Copy link
Collaborator

Qwlouse commented Feb 19, 2020

Hey, thanks for tackling this. It seems like one of the stdout capturing tests is failing now:

    def test_fd_tee_output(capsys):
        expected_lines = {
            "captured stdout",
            "captured stderr",
            "stdout from C",
            "and this is from echo",
        }
    
        capture_mode, capture_stdout = get_stdcapturer("fd")
        output = ""
        with capsys.disabled():
            print("before (stdout)")
            print("before (stderr)")
            with capture_stdout() as out:
                print("captured stdout")
                print("captured stderr", file=sys.stderr)
                output += out.get()
                libc.puts(b"stdout from C")
                libc.fflush(None)
                os.system("echo and this is from echo")
                output += out.get()
    
            output += out.get()
    
            print("after (stdout)")
            print("after (stderr)")
    
>           assert set(output.strip().split("\n")) == expected_lines
E           AssertionError: assert {'and this is...tdout from C'} == {'and this is ...tdout from C'}
E             Extra items in the right set:
E             'captured stderr'
E             Use -v to get the full diff

tests/test_stdout_capturing.py:57: AssertionError 

To me it looks like stderr somehow goes missing now.
I should warn you that this is a nasty bit of code, in that it tends to produce stochastic errors that are sometimes hard to reproduce. My best guess is that the terminate takes effect before the stderr capturing is properly flushed.

@EspenHa
Copy link
Author

EspenHa commented Feb 19, 2020

The test passes for the systems I have access to (three different linux systems).
It will be challenging to fix a bug I am not able to reproduce, but I guess I can use Azure Pipelines to test.

@stale
Copy link

stale bot commented May 19, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

capuring stdout timeout

2 participants