Make multiprocessing import local to support pyodide

[twiecki]

Description

We almost have PyMC working natively under pyodide. But upon import we get an error importing multiprocessing which isn't included in pyodide:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[5], line 1
----> 1 import pymc as pm
      2 pm.__version__

File /lib/python3.12/site-packages/pymc/__init__.py:72
     70 from pymc.pytensorf import *
     71 from pymc.sampling import *
---> 72 from pymc.smc import *
     73 from pymc.stats import *
     74 from pymc.step_methods import *

File /lib/python3.12/site-packages/pymc/smc/__init__.py:16
      1 #   Copyright 2024 The PyMC Developers
      2 #
      3 #   Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     12 #   See the License for the specific language governing permissions and
     13 #   limitations under the License.
     15 from pymc.smc.kernels import IMH, MH
---> 16 from pymc.smc.sampling import sample_smc
     18 __all__ = ("sample_smc",)

File /lib/python3.12/site-packages/pymc/smc/sampling.py:21
     18 import warnings
     20 from collections import defaultdict
---> 21 from concurrent.futures import ProcessPoolExecutor, wait
     22 from typing import Any
     24 import cloudpickle

File /lib/python312.zip/concurrent/futures/__init__.py:44, in __getattr__(name)
     41 global ProcessPoolExecutor, ThreadPoolExecutor
     43 if name == 'ProcessPoolExecutor':
---> 44     from .process import ProcessPoolExecutor as pe
     45     ProcessPoolExecutor = pe
     46     return pe

File /lib/python312.zip/concurrent/futures/process.py:55
     52 # This import is required to load the multiprocessing.connection submodule
     53 # so that it can be accessed later as `mp.connection`
     54 import multiprocessing.connection
---> 55 from multiprocessing.queues import Queue
     56 import threading
     57 import weakref

File /lib/python312.zip/multiprocessing/queues.py:23
     19 import errno
     21 from queue import Empty, Full
---> 23 import _multiprocessing
     25 from . import connection
     26 from . import context

ModuleNotFoundError: No module named '_multiprocessing'

If we made that optional, it will work out of the box.

[ricardoV94]

I don't think it makes sense to go out of our way to make a standard python lib import optional. It's an anti-pattern to import things inside functions.

[twiecki]

I agree it's an anti-pattern, but I think it certainly makes sense here because with a pretty simple change we enable PyMC in the browser. So I think the added tech debt is worth it. We can also make it optional if we're running on pyodide.

[ricardoV94]

We can also make it optional if we're running on pyodide.

How does that look like?

[twiecki]

if 'pyodide' in sys.modules:...

…

On Wed, Oct 2, 2024, 18:23 Ricardo Vieira ***@***.***> wrote: We can also make it optional if we're running on pyodide. How does that look like? — Reply to this email directly, view it on GitHub <#7519 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFETGC4KO56WKJ7IOHONKDZZPCQ3AVCNFSM6AAAAABPHH3TUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBYGMYDANZQGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ricardoV94]

That's even uglier

[twiecki]

So what do we do?

[ricardoV94]

Local import or nothing. Should an issue be open with pyodide as well?

[twiecki]

OK, I don't mind which way. What issue would that be?

[ricardoV94]

OK, I don't mind which way. What issue would that be?

Shouldn't pyodide support the python standard library?

[twiecki]

Shouldn't pyodide support the python standard library?

I don't think it's an oversight, they just haven't figured out multiprocessing yet.

[adithyalaks]

Looking at this from the PyData NYC sprint. I don't really see how we could unblock this given that multiprocessing is used in various other modules in PyMC. Would we make the import local to all of the modules that use it??

[twiecki]

Where else is it being used? I guess that'd be the only way then, depends in how many modules we use it.

[AtulBoyal]

Hi, I’d love to work on this issue as my first contribution. Is it still available?

[twiecki]

Yes!

[AtulBoyal]

Hi @twiecki @ricardoV94,

Updated PR #7736 to fix #7519 by moving multiprocessing import to run_chains. Also enhanced test_mcbackend.py.

Local tests failed (CompileError), Colab passed test_censored.py. Fixing MinGW next.

Thanks,
Atul Boyal

[AtulBoyal]

Hi @twiecki @ricardoV94,

I’m 17 until June 30, missing GSoC 2025 registration (ends April 8). PR #7736 is in, and I fixed test_censored.py locally with 64-bit MinGW (all tests pass now). I’d love to keep working on PyTensor compilation fixes for Windows—any chance for unofficial mentorship this summer?

Thanks,
Atul Boyal
(AnInnovativeCoder)