multiprocessing.pool.MaybeEncodingError with Python 3.11/Apple Silicon using Pool.starmap()

[zopyx]

I have this small script

https://proxy-ga.blitzz.co/proxy/123456/github.com/zopyx/print-css-rocks/blob/master/lessons/generate_all.py

on Python 3.11

Python 3.11.2 (main, Feb 16 2023, 02:55:59) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

which generates some PDFs and Images using a processor pool. The script works fine on various Linux system
but fails in a reproducible way on Mac M1.

The cannot pickle message does not make much sense since the return value from the execute() method is just a simple dict with some strings - no difference between Mac and Linux.

Traceback (most recent call last):
  File "/Users/ajung/src/print-css-rocks/lessons/generate_all.py", line 169, in <module>
    main()
  File "/Users/ajung/src/print-css-rocks/lessons/generate_all.py", line 144, in main
    result = pool.starmap(process_target, jobs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x1060dd510>'. Reason: 'TypeError("cannot pickle '_io.BufferedRandom' object")'

[ned-deily]

Note that on macOS, the multiprocessing module defaults to using the spawn start method as opposed to the fork start method which is the default on most other Unix platforms. The spawn start method imposes additional restrictions on programs, including those described here. Please check that you are not running into one of these gotchas.

[zopyx]

Using set_start_method() with fork or forkserver makes no difference. Same error also with Python 3.10.

[ronaldoussoren]

My guess would be that the subprocess raises an error that contains an open file in its traceback.

That should be easy enough to check by adding some more logging to process_target, for example by adding the following below your current definition of process_target (to make it easier to remove the extra code later on):

import traceback
orig_process_target = process_target
def process_target(*args, **kwds):
      try:
          return orig_process_target(*args, **kwds)
      except:
          traceback.print_exc()
          return { "error": "Unexpected exception in process_target }

If that doesn't help, try capturing the original return value and printing that to check for unexpected values.

[zopyx]

The error in my case on Mac is the missing mutool binary which caused this error.

Traceback (most recent call last):
  File "/Users/ajung/src/print-css-rocks/lessons/generate_all.py", line 41, in execute
    return _execute(cmd, log_fn, verbose)
  File "/Users/ajung/src/print-css-rocks/lessons/generate_all.py", line 51, in _execute
    p = EasyProcess(cmd).call()
  File "/Users/ajung/src/print-css-rocks/web/lib/python3.10/site-packages/easyprocess/__init__.py", line 147, in call
    self.start().wait(timeout=timeout)
  File "/Users/ajung/src/print-css-rocks/web/lib/python3.10/site-packages/easyprocess/__init__.py", line 186, in start
    raise EasyProcessError(self, "start error")
easyprocess.EasyProcessError: start error <EasyProcess cmd_param=mutool convert -F png -O resolution=150 -o "/Users/ajung/src/print-css-rocks/lessons/lesson-basic/images/bfo/bfo.png" "/Users/ajung/src/print-css-rocks/lessons/lesson-basic/bfo.pdf" cmd=['mutool', 'convert', '-F', 'png', '-O', 'resolution=150', '-o', '/Users/ajung/src/print-css-rocks/lessons/lesson-basic/images/bfo/bfo.png', '/Users/ajung/src/print-css-rocks/lessons/lesson-basic/bfo.pdf'] oserror=[Errno 2] No such file or directory: 'mutool' return_code=None stdout="None" stderr="None" timeout_happened=False>
Traceback (most recent call last):
  File "/Users/ajung/src/print-css-rocks/web/lib/python3.10/site-packages/easyprocess/__init__.py", line 176, in start
    self.popen = subprocess.Popen(
  File "/opt/homebrew/Cellar/python@3.10/3.10.10_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/homebrew/Cellar/python@3.10/3.10.10_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1847, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'mutool'

Problem solved for me...perhaps this error situtation might deserve a better error message/handling?

[zitterbewegung]

@zopyx Since mutool was the culprit of why you were getting the error we don't fix problems with external workflows. This issue should be closed.

[ronaldoussoren]

from multiprocessing import Pool
import subprocess

def doit(n):
    fd = open("/dev/null")
    raise RuntimeError(fd)

def main():
    with Pool(4) as pool:
        pool.starmap(doit, ((n,) for n in range(1)))


if __name__ == "__main__":
    main()

This reproduces the error without 3th-party code. The OP uses the easyprocess package which raises an exception containing an object with an open file as one of its attributes.

I'm leaving it to one of the multiprocessing experts to decide if it is worthwhile to pursue trying to give a more helpful error message here.

[2sn]

I see the same issue that MP code that runs fine on Linux fails on Apple silicon since recently.
It seems independent of 3rd part code.

[ronaldoussoren]

I see the same issue that MP code that runs fine on Linux fails on Apple silicon since recently.
It seems independent of 3rd part code.

Do you have a reproducer for this? The code in the original message and my small reproducer above should fail on Linux as well as on macOS.

Also, what's "recently"? Did you upgrade from Python 3.7 to some newer version?

[ronaldoussoren]

I checked the pyLIMA issue linked above and that's not directly related to this issue, but due to at least some of the PyLIMA scripts not being coded in a way that works with the spawn method (e.g. needs to be fixed on their end). I've added a comment to ebachelet/pyLIMA#66 to explain this.

The open question for this issue is whether or not it is useful (and possible!) to give a nicer error than the multiprocessing.pool.MaybeEncodingError exception in the OP when sending back a result or exception fails.

Also not that this is not a Mac specific issue and is not related to the spawn method, the exception in the OP also happens with start method "fork" as can be seen from a slight modification from my reproducer above:

from multiprocessing import Pool, set_start_method
import subprocess

set_start_method("fork")

def doit(n):
    fd = open("/dev/null")
    raise RuntimeError(fd)

def main():
    with Pool(4) as pool:
        pool.starmap(doit, ((n,) for n in range(1)))


if __name__ == "__main__":
    main()

I'm dropping the os-mac label because this isn't Mac specific.