Skip to content

Possible bug: dare_test_tjm_2 and dare_test_tjm_3 with accelerate/mkl #84

Closed

Description

@albop

On my laptop, I got the following output when running the tests this morning:

======================================================================
ERROR: quantecon.tests.test_ricatti.dare_test_tjm_2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/pablo/Programming/quantecon/quant-econ/quantecon/tests/test_ricatti.py", line 50, in dare_test_tjm_2
    X = solve_discrete_riccati(A, B, Q, R)
  File "/home/pablo/Programming/quantecon/quant-econ/quantecon/matrix_eqn.py", line 162, in solve_discrete_riccati
    Q_tilde = - Q + dot(C.T, solve(Z, C + gamma * BTA)) + gamma * I
  File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 381, in solve
    r = gufunc(a, b, signature=signature, extobj=extobj)
  File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 90, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
LinAlgError: Singular matrix

======================================================================
ERROR: quantecon.tests.test_ricatti.dare_test_tjm_3
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/pablo/Programming/quantecon/quant-econ/quantecon/tests/test_ricatti.py", line 66, in dare_test_tjm_3
    X = solve_discrete_riccati(A, B, Q, R)
  File "/home/pablo/Programming/quantecon/quant-econ/quantecon/matrix_eqn.py", line 162, in solve_discrete_riccati
    Q_tilde = - Q + dot(C.T, solve(Z, C + gamma * BTA)) + gamma * I
  File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 381, in solve
    r = gufunc(a, b, signature=signature, extobj=extobj)
  File "/home/pablo/.local/opt/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 90, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
LinAlgError: Singular matrix

The problem disappeared when I upgraded Anaconda, and removed the MKL and accelerate packages (since I don't have a license anymore):

The following packages will be REMOVED:

    accelerate:   1.5.1-np18py27_p0 
    mkl:          11.1-np18py27_p2  

The following packages will be UPDATED:

    numexpr:      2.3.1-np19py27_p0  [mkl] --> 2.3.1-np19py27_0 
    numpy:        1.9.0-py27_p0      [mkl] --> 1.9.0-py27_0     
    scikit-learn: 0.15.2-np19py27_p0 [mkl] --> 0.15.2-np19py27_0
    scipy:        0.14.0-np19py27_p0 [mkl] --> 0.14.0-np19py27_0

Has anybody run the tests with the accelerate/mkl packages ?

Activity

  1. cc7768 commented on Oct 17, 2014

    @cc7768
    Member

    @albop I replicated this when I ran tests with Python 3.4 using MKL numpy through the Anaconda accelerate package.

    It looks like the following is happening:

    In line 160 of matrix_eqn.py we are checking the condition of a matrix Z to make sure that we can invert it in line 162. In both dare_test_tjm_2 and dare_test_tjm_3 (the tests that fail) the condition of Z under gamma = 0. is on the magnitude of 1e17, which seems to suggest that it is likely ill-conditioned. @jstac Should we change this to something like cn < BIGNUMBER?

    I might look into it a little more later, I just wanted to at least figure out what was happening before I forgot about it.

  2. cc7768 commented on Oct 17, 2014

    @cc7768
    Member

    The reason we don't get this problem when we aren't on MKL is because the default linear algebra pacakge is LAPACK and it finds that the condition of the matrices is inf which is what we are checking for.

  3. sglyon commented on Jan 28, 2015

    @sglyon
    Member

    Is still still an issue with the re-vamped matrix tools functions?

  4. cc7768 commented on Jan 28, 2015

    @cc7768
    Member

    We have done nothing to address these issues for now. It has to do with LAPACK and MKL being slightly different.

  5. oyamad commented on Feb 18, 2016

    @oyamad
    Member

    The same tests failed recently: https://travis-ci.org/QuantEcon/QuantEcon.py/builds/109609660.
    Is it because Numpy and related packages now depend on MKL? See https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations.

  6. mmcky commented on Feb 24, 2016

    @mmcky
    Contributor

    This is now high priority due to tests failing under the latest anaconda distribution. Is anyone aware of a root cause of this issue before I dive in?

    I cannot seem to replicate this on a fresh py35 environment. Is it possible this has been fixed upstream in either numpy or by continuum?

  7. mmcky commented on Feb 25, 2016

    @mmcky
    Contributor

    I have updated the TravisCI environment. It seems to pass all tests suggesting an upstream fix. If someone had some time to run nosetests over the QuantEcon.py package after updating conda to the latest version - that would be appreciated.

    Then I propose we close the issue.

  8. jstac commented on Feb 25, 2016

    @jstac
    Contributor

    I guess this is a separate issue but here's my output

    ======================================================================
    ERROR: Failure: ImportError (The code previously contained in the quantecon.models subpackage has been migrated to the QuantEcon.applications (https://github.com/QuantEcon/QuantEcon.applications) repo)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/john/anaconda3/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest
        raise self.exc_val.with_traceback(self.tb)
      File "/home/john/anaconda3/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName
        addr.filename, addr.module)
      File "/home/john/anaconda3/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath
        return self.importFromDir(dir_path, fqname)
      File "/home/john/anaconda3/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir
        mod = load_module(part_fqname, fh, filename, desc)
      File "/home/john/anaconda3/lib/python3.5/imp.py", line 244, in load_module
        return load_package(name, filename)
      File "/home/john/anaconda3/lib/python3.5/imp.py", line 216, in load_package
        return _load(spec)
      File "<frozen importlib._bootstrap>", line 693, in _load
      File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 662, in exec_module
      File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
      File "/home/john/sync_dir/books/quant-econ/QuantEcon.py/quantecon/models/__init__.py", line 1, in <module>
        raise ImportError("The code previously contained in the quantecon.models subpackage has been migrated to the QuantEcon.applications (https://github.com/QuantEcon/QuantEcon.applications) repo")
    ImportError: The code previously contained in the quantecon.models subpackage has been migrated to the QuantEcon.applications (https://github.com/QuantEcon/QuantEcon.applications) repo
    
    ----------------------------------------------------------------------
    Ran 263 tests in 31.321s
    
    FAILED (SKIP=1, errors=1)
    Closing remaining open files:/home/john/sync_dir/books/quant-econ/QuantEcon.py/quantecon/tests/data/testing_data.h5...done
    
  9. mmcky commented on Feb 25, 2016

    @mmcky
    Contributor

    Thanks @jstac this is actually an error that results in the notice for removing the models subpackage. So when models loads it throws the ImportError.

    I should have been more specific with nosetests, Travis runs the following:

    nosetests --exclude=models
  10. jstac commented on Feb 25, 2016

    @jstac
    Contributor

    With latest anaconda:

    nosetests --exclude=models                                +1938 10:47 ❰─┘
    .........................................................................................................................S.............................................................................................................................................
    ----------------------------------------------------------------------
    Ran 262 tests in 31.575s
    
    OK (SKIP=1)
    Closing remaining open files:/home/john/sync_dir/books/quant-econ/QuantEcon.py/quantecon/tests/data/testing_data.h5...done
    
  11. mmcky commented on Feb 25, 2016

    @mmcky
    Contributor

    That looks like the test suite passes on your machine also.
    I plan to close this issue tomorrow - unless anyone objects.

  12. oyamad commented on Oct 23, 2017

    @oyamad
    Member

    I got the same errors today.

    anaconda                  5.0.0            py36hd9bc8a5_0
    
    mkl                       2018.0.0             hc285769_4
    
    numpy                     1.13.1           py36h93d791d_2
    
  13. reopened this on Oct 23, 2017
  14. mmcky commented on Oct 23, 2017

    @mmcky
    Contributor

    Hey @oyamad I wasn't able to replicate this test failure.
    I created the latest anaconda environment with

    mkl                       2018.0.0             hb491cac_4 
    anaconda                  5.0.0.1          py36hfb0b618_1
    numpy                     1.13.1           py36h5bc529a_2
    
  15. oyamad commented on Oct 23, 2017

    @oyamad
    Member

    I am on Mac:

    $ python
    Python 3.6.2 |Anaconda, Inc.| (default, Sep 21 2017, 18:29:43) 
    [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
    
  16. mmcky commented on Oct 23, 2017

    @mmcky
    Contributor

    OK thanks for reporting @oyamad. Maybe it's time to add CircleCI and Appveyor here. We have these on conda-forge but that isn't run as frequently as this master repo.

  17. oyamad commented on Oct 23, 2017

    @oyamad
    Member

    Re #84 (comment), on my machine I have cn = 3.591469197922365e+17 for gamma = 0. (for both dare_test_tjm_2 and dare_test_tjm_3).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions