deadlock in softhsm2 when testing head of openssl

[nhorman]

When building softhsm2 with the pcks11 provider and openssl-4.0-dev, I'm seeing a deadlock in the softhsm2 code:

command:

openssl req -new -key 'pkcs11:model=SoftHSM%20v2;object=test-key;pin-value=1234' -out ./req.pem -text -x509 -subj /CN=libp11-DEP8

Analysis:
The following stacktrace:

#0  OSLockMutex (mutex=0x548ee0) at ../../../../src/lib/common/osmutex.cpp:102
#1  0x00007ffff7531191 in Token::loginUser (this=0x548e90, pin=...) at ../../../../src/lib/slot_mgr/Token.cpp:155
#2  0x00007ffff74d5225 in SoftHSM::C_Login (this=0x5316e0, hSession=hSession@entry=2, userType=userType@entry=1, pPin=pPin@entry=0x54b530 "1234", 
    ulPinLen=ulPinLen@entry=4) at ../../../src/lib/SoftHSM.cpp:1557
#3  0x00007ffff74b750f in C_Login (hSession=2, userType=1, pPin=0x54b530 "1234", ulPinLen=4) at ../../../src/lib/main.cpp:407
#4  0x00007ffff758edde in p11prov_Login (ctx=0x514ce0, hSession=2, userType=1, pPin=0x54b530 "1234", ulPinLen=4) at ../src/interface.gen.c:451
#5  0x00007ffff75b873f in token_login (session=0x54a840, uri=0x54b550, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638, 
    slot=0x549710, user_type=1) at ../src/session.c:592
#6  0x00007ffff75b95ae in slot_login (slot=0x549710, uri=0x54b550, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638, reqlogin=true, 
    _session=0x0) at ../src/session.c:921
#7  0x00007ffff75b9a15 in p11prov_get_session (provctx=0x514ce0, slotid=0x7fffffffce88, next_slotid=0x7fffffffce80, uri=0x54b550, 
    mechtype=18446744073709551615, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638, reqlogin=true, rw=false, _session=0x546078)
    at ../src/session.c:1042
#8  0x00007ffff75c6fe2 in store_fetch (ctx=0x545ff0, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638) at ../src/store.c:124
#9  0x00007ffff75c74e1 in p11prov_store_load (pctx=0x545ff0, object_cb=0x7ffff7aaa1c0 <ossl_store_handle_load_result>, object_cbarg=0x7fffffffd060, 
    pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638) at ../src/store.c:255
#10 0x00007ffff7aa79f9 in OSSL_STORE_load () from /home/nhorman/opt/lib64/libcrypto.so.4
#11 0x000000000046dea8 in load_key_certs_crls ()
#12 0x000000000046f177 in load_key ()
#13 0x000000000043ace9 in req_main ()
#14 0x000000000042f757 in do_cmd ()
#15 0x0000000000406f5e in main ()

Acquires the tokenMutex lock, to be unlocked via the MutexLocker destructor when the loginUser function returns.

However, during the call to Token:loginUser, we enter this call path:

#0  OSLockMutex (mutex=0x548ee0) at ../../../../src/lib/common/osmutex.cpp:102
#1  0x00007ffff7530fd8 in Token::isSOLoggedIn (this=0x548e90) at ../../../../src/lib/slot_mgr/Token.cpp:95
#2  0x00007ffff752ecb5 in SessionManager::openSession (this=0x549420, slot=0x548e60, flags=flags@entry=4, pApplication=pApplication@entry=0x54d050, 
    notify=notify@entry=0x7ffff75b6c39 <token_session_callback>, phSession=phSession@entry=0x54d068) at ../../../../src/lib/session_mgr/SessionManager.cpp:83
#3  0x00007ffff74d3cea in SoftHSM::C_OpenSession (this=0x5316e0, slotID=slotID@entry=1455144078, flags=flags@entry=4, 
    pApplication=pApplication@entry=0x54d050, notify=notify@entry=0x7ffff75b6c39 <token_session_callback>, phSession=phSession@entry=0x54d068)
    at ../../../src/lib/SoftHSM.cpp:1433
#4  0x00007ffff74b73d3 in C_OpenSession (slotID=1455144078, flags=4, pApplication=0x54d050, notify=0x7ffff75b6c39 <token_session_callback>, phSession=0x54d068)
    at ../../../src/lib/main.cpp:317
#5  0x00007ffff758e1f6 in p11prov_OpenSession (ctx=0x514ce0, slotID=1455144078, flags=4, pApplication=0x54d050, 
    Notify=0x7ffff75b6c39 <token_session_callback>, phSession=0x54d068) at ../src/interface.gen.c:304
#6  0x00007ffff75b6d80 in token_session_open (session=0x54d050, flags=4) at ../src/session.c:77
#7  0x00007ffff75b9bc5 in p11prov_get_session (provctx=0x514ce0, slotid=0x7fffffffc7c0, next_slotid=0x0, uri=0x0, mechtype=592, pw_cb=0x0, pw_cbarg=0x0, 
    reqlogin=false, rw=false, _session=0x54d040) at ../src/session.c:1100
#8  0x00007ffff757cb3e in p11prov_digest_init (ctx=0x54d030, params=0x0) at ../src/digests.c:280
#9  0x00007ffff750bbbc in OSSLEVPHashAlgorithm::hashInit (this=0x54b490) at ../../../../src/lib/crypto/OSSLEVPHashAlgorithm.cpp:61
#10 0x00007ffff7515eb8 in RFC4880::PBEDeriveKey (password=..., salt=..., ppKey=ppKey@entry=0x7fffffffc908) at ../../../../src/lib/data_mgr/RFC4880.cpp:74
#11 0x00007ffff7517df8 in SecureDataManager::login (this=this@entry=0x549060, passphrase=..., encryptedKey=...)
    at ../../../../src/lib/data_mgr/SecureDataManager.cpp:264
#12 0x00007ffff751837d in SecureDataManager::loginUser (this=0x549060, userPIN=...) at ../../../../src/lib/data_mgr/SecureDataManager.cpp:317
#13 0x00007ffff7531253 in Token::loginUser (this=0x548e90, pin=...) at ../../../../src/lib/slot_mgr/Token.cpp:176
#14 0x00007ffff74d5225 in SoftHSM::C_Login (this=0x5316e0, hSession=hSession@entry=2, userType=userType@entry=1, pPin=pPin@entry=0x54b530 "1234", 
    ulPinLen=ulPinLen@entry=4) at ../../../src/lib/SoftHSM.cpp:1557
#15 0x00007ffff74b750f in C_Login (hSession=2, userType=1, pPin=0x54b530 "1234", ulPinLen=4) at ../../../src/lib/main.cpp:407
#16 0x00007ffff758edde in p11prov_Login (ctx=0x514ce0, hSession=2, userType=1, pPin=0x54b530 "1234", ulPinLen=4) at ../src/interface.gen.c:451
#17 0x00007ffff75b873f in token_login (session=0x54a840, uri=0x54b550, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638, 
    slot=0x549710, user_type=1) at ../src/session.c:592
#18 0x00007ffff75b95ae in slot_login (slot=0x549710, uri=0x54b550, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638, reqlogin=true, 
    _session=0x0) at ../src/session.c:921
#19 0x00007ffff75b9a15 in p11prov_get_session (provctx=0x514ce0, slotid=0x7fffffffce88, next_slotid=0x7fffffffce80, uri=0x54b550, 
    mechtype=18446744073709551615, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638, reqlogin=true, rw=false, _session=0x546078)
    at ../src/session.c:1042
#20 0x00007ffff75c6fe2 in store_fetch (ctx=0x545ff0, pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638) at ../src/store.c:124
#21 0x00007ffff75c74e1 in p11prov_store_load (pctx=0x545ff0, object_cb=0x7ffff7aaa1c0 <ossl_store_handle_load_result>, object_cbarg=0x7fffffffd060, 
    pw_cb=0x7ffff796af30 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x54b638) at ../src/store.c:255
#22 0x00007ffff7aa79f9 in OSSL_STORE_load () from /home/nhorman/opt/lib64/libcrypto.so.4
#23 0x000000000046dea8 in load_key_certs_crls ()
#24 0x000000000046f177 in load_key ()
--Type <RET> for more, q to quit, c to continue without paging--
#25 0x000000000043ace9 in req_main ()
#26 0x000000000042f757 in do_cmd ()
#27 0x0000000000406f5e in main ()

Which attempts to acquire the same tokenMutex, resulting in deadlock.

Unsure as to what the proper fix is here

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• HandleManager 'No such file or directory', segmentation fault #773
• Key mismatch with OpenSSL and 2 prime256v1 keys in token #784
• softhsm2 crashes in context of openssl 3 pkcs11 provider #780
• Valgrind issues in SoftHSM #785
• SoftHsm2 does not behave the same on ARM64 compared to X64 #803

👤 Suggested Assignees

• NickDarvey
• benoiton
• petrovr
• frankmorgner
• qathy

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[nhorman]

Note: This only occurs when the default provider isn't loaded, as SoftHSM2 expects that it is, so it may not be a particularly critical issue, but it would be nice if SoftHSM could fail more gracefully than a deadlock.