You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Possible bug: dare_test_tjm_2 and dare_test_tjm_3 with accelerate/mkl #84
On my laptop, I got the following output when running the tests this morning:
======================================================================
ERROR: quantecon.tests.test_ricatti.dare_test_tjm_2
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/pablo/Programming/quantecon/quant-econ/quantecon/tests/test_ricatti.py", line 50, in dare_test_tjm_2
X = solve_discrete_riccati(A, B, Q, R)
File "/home/pablo/Programming/quantecon/quant-econ/quantecon/matrix_eqn.py", line 162, in solve_discrete_riccati
Q_tilde = - Q + dot(C.T, solve(Z, C + gamma * BTA)) + gamma * I
File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 381, in solve
r = gufunc(a, b, signature=signature, extobj=extobj)
File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 90, in _raise_linalgerror_singular
raise LinAlgError("Singular matrix")
LinAlgError: Singular matrix
======================================================================
ERROR: quantecon.tests.test_ricatti.dare_test_tjm_3
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/pablo/Programming/quantecon/quant-econ/quantecon/tests/test_ricatti.py", line 66, in dare_test_tjm_3
X = solve_discrete_riccati(A, B, Q, R)
File "/home/pablo/Programming/quantecon/quant-econ/quantecon/matrix_eqn.py", line 162, in solve_discrete_riccati
Q_tilde = - Q + dot(C.T, solve(Z, C + gamma * BTA)) + gamma * I
File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 381, in solve
r = gufunc(a, b, signature=signature, extobj=extobj)
File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 90, in _raise_linalgerror_singular
raise LinAlgError("Singular matrix")
LinAlgError: Singular matrix
The problem disappeared when I upgraded Anaconda, and removed the MKL and accelerate packages (since I don't have a license anymore):
The following packages will be REMOVED:
accelerate: 1.5.1-np18py27_p0
mkl: 11.1-np18py27_p2
The following packages will be UPDATED:
numexpr: 2.3.1-np19py27_p0 [mkl] --> 2.3.1-np19py27_0
numpy: 1.9.0-py27_p0 [mkl] --> 1.9.0-py27_0
scikit-learn: 0.15.2-np19py27_p0 [mkl] --> 0.15.2-np19py27_0
scipy: 0.14.0-np19py27_p0 [mkl] --> 0.14.0-np19py27_0
Has anybody run the tests with the accelerate/mkl packages ?
@albop I replicated this when I ran tests with Python 3.4 using MKL numpy through the Anaconda accelerate package.
It looks like the following is happening:
In line 160 of matrix_eqn.py we are checking the condition of a matrix Z to make sure that we can invert it in line 162. In both dare_test_tjm_2 and dare_test_tjm_3 (the tests that fail) the condition of Z under gamma = 0. is on the magnitude of 1e17, which seems to suggest that it is likely ill-conditioned. @jstac Should we change this to something like cn < BIGNUMBER?
I might look into it a little more later, I just wanted to at least figure out what was happening before I forgot about it.
The reason we don't get this problem when we aren't on MKL is because the default linear algebra pacakge is LAPACK and it finds that the condition of the matrices is inf which is what we are checking for.
I have updated the TravisCI environment. It seems to pass all tests suggesting an upstream fix. If someone had some time to run nosetests over the QuantEcon.py package after updating conda to the latest version - that would be appreciated.
I guess this is a separate issue but here's my output
======================================================================
ERROR: Failure: ImportError (The code previously contained in the quantecon.models subpackage has been migrated to the QuantEcon.applications (https://github.com/QuantEcon/QuantEcon.applications) repo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/john/anaconda3/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/john/anaconda3/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/home/john/anaconda3/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/john/anaconda3/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/john/anaconda3/lib/python3.5/imp.py", line 244, in load_module
return load_package(name, filename)
File "/home/john/anaconda3/lib/python3.5/imp.py", line 216, in load_package
return _load(spec)
File "<frozen importlib._bootstrap>", line 693, in _load
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/john/sync_dir/books/quant-econ/QuantEcon.py/quantecon/models/__init__.py", line 1, in <module>
raise ImportError("The code previously contained in the quantecon.models subpackage has been migrated to the QuantEcon.applications (https://github.com/QuantEcon/QuantEcon.applications) repo")
ImportError: The code previously contained in the quantecon.models subpackage has been migrated to the QuantEcon.applications (https://github.com/QuantEcon/QuantEcon.applications) repo
----------------------------------------------------------------------
Ran 263 tests in 31.321s
FAILED (SKIP=1, errors=1)
Closing remaining open files:/home/john/sync_dir/books/quant-econ/QuantEcon.py/quantecon/tests/data/testing_data.h5...done
nosetests --exclude=models +1938 10:47 ❰─┘
.........................................................................................................................S.............................................................................................................................................
----------------------------------------------------------------------
Ran 262 tests in 31.575s
OK (SKIP=1)
Closing remaining open files:/home/john/sync_dir/books/quant-econ/QuantEcon.py/quantecon/tests/data/testing_data.h5...done
OK thanks for reporting @oyamad. Maybe it's time to add CircleCI and Appveyor here. We have these on conda-forge but that isn't run as frequently as this master repo.
Activity
cc7768 commentedon Oct 17, 2014
@albop I replicated this when I ran tests with Python 3.4 using MKL
numpythrough the Anaconda accelerate package.It looks like the following is happening:
In line 160 of
matrix_eqn.pywe are checking the condition of a matrixZto make sure that we can invert it in line 162. In bothdare_test_tjm_2anddare_test_tjm_3(the tests that fail) the condition ofZundergamma = 0.is on the magnitude of 1e17, which seems to suggest that it is likely ill-conditioned. @jstac Should we change this to something likecn < BIGNUMBER?I might look into it a little more later, I just wanted to at least figure out what was happening before I forgot about it.
cc7768 commentedon Oct 17, 2014
The reason we don't get this problem when we aren't on
MKLis because the default linear algebra pacakge isLAPACKand it finds that the condition of the matrices isinfwhich is what we are checking for.sglyon commentedon Jan 28, 2015
Is still still an issue with the re-vamped matrix tools functions?
cc7768 commentedon Jan 28, 2015
We have done nothing to address these issues for now. It has to do with LAPACK and MKL being slightly different.
oyamad commentedon Feb 18, 2016
The same tests failed recently: https://travis-ci.org/QuantEcon/QuantEcon.py/builds/109609660.
Is it because
Numpyand related packages now depend onMKL? See https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations.mmcky commentedon Feb 24, 2016
This is now high priority due to tests failing under the latest anaconda distribution. Is anyone aware of a root cause of this issue before I dive in?
I cannot seem to replicate this on a fresh
py35environment. Is it possible this has been fixed upstream in eithernumpyor bycontinuum?mmcky commentedon Feb 25, 2016
I have updated the TravisCI environment. It seems to pass all tests suggesting an upstream fix. If someone had some time to run
nosetestsover the QuantEcon.py package after updating conda to the latest version - that would be appreciated.Then I propose we close the issue.
jstac commentedon Feb 25, 2016
I guess this is a separate issue but here's my output
mmcky commentedon Feb 25, 2016
Thanks @jstac this is actually an error that results in the notice for removing the models subpackage. So when models loads it throws the ImportError.
I should have been more specific with nosetests, Travis runs the following:
jstac commentedon Feb 25, 2016
With latest anaconda:
mmcky commentedon Feb 25, 2016
That looks like the test suite passes on your machine also.
I plan to close this issue tomorrow - unless anyone objects.
oyamad commentedon Oct 23, 2017
I got the same errors today.
mmcky commentedon Oct 23, 2017
Hey @oyamad I wasn't able to replicate this
testfailure.I created the latest
anacondaenvironment withoyamad commentedon Oct 23, 2017
I am on Mac:
mmcky commentedon Oct 23, 2017
OK thanks for reporting @oyamad. Maybe it's time to add
CircleCIandAppveyorhere. We have these onconda-forgebut that isn't run as frequently as this master repo.oyamad commentedon Oct 23, 2017
Re #84 (comment), on my machine I have
cn = 3.591469197922365e+17forgamma = 0.(for bothdare_test_tjm_2anddare_test_tjm_3).