Skip to content

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zooba merged 21 commits into from
Jun 19, 2019
Merged

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1d33fe6
bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh
May 20, 2019
3aadf52
add +d to match test.bat
May 21, 2019
7f17d29
100% test pass when manually running test.bat
May 22, 2019
9aaa2c4
fix file destination
May 22, 2019
3f38d18
only skip ssl tests on Windows IoT
May 23, 2019
4e147de
Avoid mixing debug/release CRT on Windows
May 23, 2019
9819647
test for win32
May 23, 2019
365701c
propagate exit code
May 23, 2019
69c40d7
fix 2 more exit 0 calls
May 23, 2019
086a1b9
fixes for test_regrtest
May 24, 2019
042e0e9
merge upstream/master
May 24, 2019
e230795
add tools/parser for test_tools
May 24, 2019
1a1f12d
copy entire Tools directory
May 24, 2019
29601d8
skip check on Windows ARM32
May 24, 2019
786194d
restore 64-bit check in test_regrtest
May 24, 2019
cd2d5c5
merge upstream/master
Jun 11, 2019
f5c917a
Update Lib/test/test_math.py
paulmon Jun 19, 2019
e2fb14b
Update Lib/test/test_ssl.py
paulmon Jun 19, 2019
ce3dfbc
Merge remote-tracking branch 'upstream/master' into arm32_pythoninfo
Jun 19, 2019
5c355ab
fix mismatched paren
Jun 19, 2019
f1ce1bf
fix Py_DEBUG_WIN32 spelling
Jun 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Lib/test/support/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,7 +485,9 @@ def _is_gui_available():
if hasattr(_is_gui_available, 'result'):
return _is_gui_available.result
reason = None
if sys.platform.startswith('win'):
if sys.platform.startswith('win') and platform.win32_is_iot():
reason = "gui is not available on Windows IoT Core"
elif sys.platform.startswith('win'):
# if Python is running as a service (such as the buildbot service),
# gui interaction may be disallowed
import ctypes
Expand Down
2 changes: 2 additions & 0 deletions Lib/test/test_math.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -465,6 +465,8 @@ def testCos(self):
self.assertRaises(ValueError, math.cos, NINF)
self.assertTrue(math.isnan(math.cos(NAN)))

@unittest.skipIf(sys.platform=='win32' and (platform.machine()=='ARM' or platform.machine()=='ARM64'),
"Windows UCRT is off by 2 ULP this test requires accuracy within 1 ULP")
def testCosh(self):
self.assertRaises(TypeError, math.cosh)
self.ftest('cosh(0)', math.cosh(0), 1)
Expand Down
4 changes: 4 additions & 0 deletions Lib/test/test_ssl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@

ssl = support.import_module("ssl")

Py_DEBUG = hasattr(sys, 'gettotalrefcount')

PROTOCOLS = sorted(ssl._PROTOCOL_NAMES)
HOST = support.HOST
Expand Down Expand Up @@ -1345,6 +1346,7 @@ def test_load_verify_cadata(self):
ctx.load_verify_locations(cadata=b"broken")


@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")
Copy link
Member

@zooba zooba May 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is certainly not a general issue - can we scope it down to the platform(s) where it crashes?

Copy link
Contributor Author

@paulmon paulmon May 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will scope it down.

I think it's limited to UCRT. The problem is that SSL has FILE* from the ucrtbase.dll table and python has a FILE* from the ucrtbased.dll table.

I think we could limit it to win32+arm or platform.win32_is_iot( ). Unless there's a way to check if python is using UCRT? I couldn't find one.

Copy link
Member

@zooba zooba May 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it's because of mixing FILE* across debug/release builds? In that case we scope it down to Windows and enhance the skip message to say "Avoid mixing debug/release CRT on Windows"

Copy link
Contributor Author

@paulmon paulmon May 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

def test_load_dh_params(self):
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_dh_params(DHFILE)
Expand Down Expand Up @@ -1645,6 +1647,7 @@ def test_str(self):
self.assertEqual(str(e), "foo")
self.assertEqual(e.errno, 1)

@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")
def test_lib_reason(self):
# Test the library and reason attributes
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
Expand Down Expand Up @@ -3825,6 +3828,7 @@ def test_compression_disabled(self):
sni_name=hostname)
self.assertIs(stats['compression'], None)

@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")
def test_dh_params(self):
# Check we can get a connection with ephemeral Diffie-Hellman
client_context, server_context, hostname = testing_context()
Expand Down
46 changes: 46 additions & 0 deletions Tools/buildbot/remoteDeploy.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
@echo off
rem Used by the buildbot "remotedeploy" step.
setlocal

set here=%~dp0
set arm32_ssh=

:CheckOpts
if "%1"=="-arm32" (set arm32_ssh=true) & shift & goto CheckOpts
if NOT "%1"=="" (echo unrecognized option %1) & goto Arm32SshHelp

if "%arm32_ssh%"=="true" goto :Arm32Ssh

:Arm32Ssh
if "%SSH_SERVER%"=="" goto :Arm32SshHelp
if "%SSH%"=="" if EXIST %WINDIR%\System32\OpenSSH\ssh.exe (set SSH=%WINDIR%\System32\OpenSSH\ssh.exe)
if "%SCP%"=="" if EXIST %WINDIR%\System32\OpenSSH\scp.exe (set SCP=%WINDIR%\System32\OpenSSH\scp.exe)
echo SSH = %SSH%
echo SCP = %SCP%
if "%PYTHON_SOURCE%"=="" (set PYTHON_SOURCE=%here%..\..\)
if "%REMOTE_PYTHON_DIR%"=="" (set REMOTE_PYTHON_DIR=C:\python\)
%SSH% %SSH_SERVER% "if EXIST %REMOTE_PYTHON_DIR% (rd %REMOTE_PYTHON_DIR% /s/q)"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%PCBuild\arm32"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%temp"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%Modules"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%PC"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.bat /b`) do @%SCP% PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.py /b`) do @%SCP% PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.exe /b`) do @%SCP% PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.pyd /b`) do @%SCP% PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.dll /b`) do @%SCP% PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
%SCP% -r "%PYTHON_SOURCE%Include" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Include"
%SCP% -r "%PYTHON_SOURCE%Lib" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Lib"
%SCP% "%PYTHON_SOURCE%Modules\Setup" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Modules"
%SCP% "%PYTHON_SOURCE%PC\pyconfig.h" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PC"

exit /b 0

:Arm32SshHelp
echo SSH_SERVER environment variable must be set to administrator@[ip address]
echo where [ip address] is the address of a Windows IoT Core ARM32 device.
echo.
echo The test worker should have the SSH agent running.
echo Also a key must be created with ssh-keygen and added to both the buildbot worker machine
echo and the ARM32 worker device: see https://docs.microsoft.com/en-us/windows/iot-core/connect-your-device/ssh
exit /b 127
31 changes: 31 additions & 0 deletions Tools/buildbot/remotePythonInfo.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
@echo off
rem Used by the buildbot "remotedeploy" step.
setlocal

set here=%~dp0
set arm32_ssh=
set suffix=_d
:CheckOpts
if "%1"=="-arm32" (set arm32_ssh=true) & (set prefix=c:\python\pcbuild\arm32) & shift & goto CheckOpts
if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts
if "%1"=="+d" (set suffix=) & shift & goto CheckOpts
if NOT "%1"=="" (echo unrecognized option %1) & goto Arm32SshHelp

if "%arm32_ssh%"=="true" goto :Arm32Ssh

:Arm32Ssh
if "%SSH_SERVER%"=="" goto :Arm32SshHelp
if "%SSH%"=="" if EXIST %WINDIR%\System32\OpenSSH\ssh.exe (set SSH=%WINDIR%\System32\OpenSSH\ssh.exe)
set PYTHON_EXE=%prefix%\python%suffix%.exe
echo on
%SSH% %SSH_SERVER% %PYTHON_EXE% -m test.pythoninfo
exit /b 0

:Arm32SshHelp
echo SSH_SERVER environment variable must be set to administrator@[ip address]
echo where [ip address] is the address of a Windows IoT Core ARM32 device.
echo.
echo The test worker should have the SSH agent running.
echo Also a key must be created with ssh-keygen and added to both the buildbot worker machine
echo and the ARM32 worker device: see https://docs.microsoft.com/en-us/windows/iot-core/connect-your-device/ssh
exit /b 127
14 changes: 2 additions & 12 deletions Tools/buildbot/test.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ if "%1"=="+d" (set rt_opts=%rt_opts:-d=%) & shift & goto CheckOpts
if "%1"=="+q" (set rt_opts=%rt_opts:-q=%) & shift & goto CheckOpts
if NOT "%1"=="" (set regrtest_args=%regrtest_args% %1) & shift & goto CheckOpts

echo on
if "%arm32_ssh%"=="true" goto :Arm32Ssh

call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW --slowest --timeout=1200 --fail-env-changed %regrtest_args%
Expand All @@ -28,20 +27,11 @@ set dashU=-unetwork,decimal,subprocess,urlfetch,tzdata
if "%SSH_SERVER%"=="" goto :Arm32SshHelp
if "%PYTHON_SOURCE%"=="" (set PYTHON_SOURCE=%here%..\..\)
if "%REMOTE_PYTHON_DIR%"=="" (set REMOTE_PYTHON_DIR=C:\python\)
if "%SSH%"=="" if EXIST %WINDIR%\System32\OpenSSH\ssh.exe (set SSH=%WINDIR%\System32\OpenSSH\ssh.exe)
set TEMP_ARGS=--temp %REMOTE_PYTHON_DIR%temp
ssh %SSH_SERVER% "if EXIST %REMOTE_PYTHON_DIR% (rd %REMOTE_PYTHON_DIR% /s/q)"
ssh %SSH_SERVER% "md %REMOTE_PYTHON_DIR%PCBuild\arm32"
ssh %SSH_SERVER% "md %REMOTE_PYTHON_DIR%temp"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.bat /b`) do @scp PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.py /b`) do @scp PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.exe /b`) do @scp PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.pyd /b`) do @scp PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.dll /b`) do @scp PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
scp -r "%PYTHON_SOURCE%Include" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Include"
scp -r "%PYTHON_SOURCE%Lib" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Lib"

set rt_args=%rt_opts% %dashU% -rwW --slowest --timeout=1200 --fail-env-changed %regrtest_args% %TEMP_ARGS%
ssh %SSH_SERVER% "set TEMP=%REMOTE_PYTHON_DIR%temp& %REMOTE_PYTHON_DIR%PCbuild\rt.bat" %rt_args%
%SSH% %SSH_SERVER% "set TEMP=%REMOTE_PYTHON_DIR%temp& %REMOTE_PYTHON_DIR%PCbuild\rt.bat" %rt_args%
exit /b 0

:Arm32SshHelp
Expand Down