Multiprocessing DeprecationWarning after upgrading to 0.6.1

[streichsbaer]

I have just upgraded to the latest version of Magika and am using the Python library.

Regardless of whether I run it in the debugger or directly I now get this DeprecationWarning:

.../lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=91146) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()

This message appears as long as I have from magika import Magika in my code.

Any idea why that is the case and how I can remove that message?

[reyammer]

Thanks for reporting. Not sure what's going on, the newer version should not have any modification related to that. For context, magika's python module doesn't do any explicit multithreading, but onnxruntime likely does so.

Here are some questions to help debug this:

• I guess that warning comes from your own code that likely does something like os.fork()?
• Can you confirm that, in your pip/uv/etc. environment, if you just do import magika within a new python/ipython shell without additional code, you do NOT see that warning?
• What's your OS version? python version (seems 3.12 from your log)?
• When upgrading magika, did you also upgrade anything else (example: were you maybe using python 3.11 before, and now you upgrade to python 3.12)?
• Can it be that by installing the new version of magika, you also got a new version of onnxruntime (which may have change something under the hood)? It would help to know which version of onnxruntime you had when using magika 0.5.1 and which one you have now.
• Do you have a minimal test case that you can share so that I can reproduce this?

Thanks!

[reyammer]

I think I figured it out... in the new version we are emitting a couple of deprecation warnings, and we added this line:

magika/python/src/magika/types/content_type_info.py

Line 8 in 9db5e75

warnings.simplefilter("always", DeprecationWarning)

, which says to always output deprecation warnings.

My current theory is thus: the deprecation warning you see about multiprocess has always been there, but somehow your setup is such so that it was not printed... and this "always" filter from magika now makes it visible.

What I don't understand is why you wouldn't see that deprecation warning before... I would assume the default is to show them? can it be that somewhere in your code there is "disable deprecation warnings" statement, which then gets overwritten by magika?

Here are the docs about the filters: https://docs.python.org/3/library/warnings.html#the-warnings-filter

Is it possible for you to check that if we would change that line with "default" (instead of "always"), then your deprecation warning about multiprocess would go away?

[reyammer]

TIL: by default, DeprecationWarning are ignored for every module except __main__. [1] By adding that always line, we effectively overwrote the default for all libraries, showing all DeprecationWarnings, which is not nice.

I see two paths forward:

1. Keep Magika's deprecation enabled (without affecting existing modules, like multiprocess). One way to implement this is to use stacklevel to make it seem the warning comes from user code [2] (In this way, we don't need to touch the warnings' filter.)
2. Make Magika's deprecation hidden by default, and leave it up to the clients to explicitly enabled them during testing.

Since Magika mostly targets developers, I would lean towards option 1.

I will think about this for a couple of days; in the meantime, feedback is welcome!

[1] https://docs.python.org/3/library/warnings.html#updating-code-for-new-versions-of-dependencies
[2] https://docs.python.org/3/library/warnings.html#warnings.warn

[streichsbaer]

Thanks for the thorough investigation @reyammer!

I initially did a few checks and was certain that the issue was somehow related to the update of Magika to 0.6.1.
I even pinned the onnxruntime runtime to the previous version, but it didn't make a difference.

I agree, it's probably best to go with option 1 and keep all Magika's deprecation warnings enabled, but not overwrite the settings for the underlying app.

For now, the simple workaround on my end is to overwrite the warnings filter after importing Magika.

from magika import Magika
import warnings
warnings.simplefilter("ignore", DeprecationWarning)
...

I'm unblocked and happy that I can use the new version.
It's even better at correctly identifying files without extensions, great job Magika team!

[reyammer]

Fixed in #1017. Went with option 1 (even if it's not perfect; the warnings would be visible only to direct users of the deprecated property. I've added more context in the PR comment.).

Thanks again for reporting!

[streichsbaer]

Nice, thanks for your support!