Introduce --enable-gnutls-relax-mode, apply it in cupsHashData()

[zdohnal]

#5601

[michaelrsweet]

@zdohnal Is the GNUTLS_FIPS140_SET_LAX_MODE call not an actual function? I would prefer to just auto-detect it at configure time and call it if it is defined. The more knobs we get the less likely it is that it will work... :/

[zdohnal]

It is actually macro for:

if (gnutls_fips140_mode_enabled())
gnutls_fips140_set_mode(GNUTLS_FIPS140_LAX, GNUTLS_FIPS140_SET_MODE_THREAD);

See https://gnutls.org/manual/html_node/FIPS140_002d2-mode.html

Ok, I'll see what I can do in cups-ssl.m4 about it.

[michaelrsweet]

@zdohnal Based on the documentation for that function, we can safely do a configure check for gnutls_fips140_set_mode and then do:

unsigned oldmode = gnutls_fips140_mode_enabled();
gnutls_fips140_set_mode(GNUTLS_FIPS140_LAX, GNUTLS_FIPS140_SET_MODE_THREAD);

...

gnutls_fips140_set_mode(oldmode, GNUTLS_FIPS140_SET_MODE_THREAD);

This will save and restore the mode around the hashing calls, and hopefully not step on any other users of this API (since we'd restore the previous setting, not just put it back to strict...)

[michaelrsweet]

Pushed changes for both 2.3.x and 2.2.x. Please review and let me know if this addresses your issues...

[master 9ec6124] GNU TLS FIPS140 support (Issue #5601, Issue #5622)

[branch-2.2 2f26c6b] GNU TLS FIPS140 support (Issue #5601, Issue #5622)