multiprocessing package in interactive Python

Question

I have the following code test.py:

#multiprocessing in the interactive Python 

import time
from multiprocessing import Process, Pipe

def MyProcess(a):

    while(1):
       time.sleep(1)
       a.send("tic")    

if __name__ == "__main__":

    a, b = Pipe() 

    p = Process(target=MyProcess, args=(a,))
    p.start()

    while(1):
       msg=b.recv()
       print(msg)

It works fine if I execute it in the DOS shell "python test.py" But it doesn't work if I use "Execute File" from IEP (Pyzo).

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:pyzo2014a_64blibmultiprocessingspawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:pyzo2014a_64blibmultiprocessingspawn.py", line 116, in _main
    self = pickle.load(from_parent)
AttributeError: Can't get attribute 'MyProcess' on <module '__main__' (built-in)>

I found that this is a documented 'issue'. Please check the answer of the link below.

multiprocessing breaks in interactive mode

Does it mean that I should not use multiprocessing package from the interactive Python? Does it mean I can not create a process from the IPython console? Any clarification on this will be highly appreciated

Answer 1

Correct, you can't use multiprocessing from the interpreter… primarily because pickle doesn't know how to serialize interactive functions. However, if you use a multiprocessing fork, called pathos.multiprocessing, you can do what you want from the interpreter. This works because pathos.multiprocessing uses dill, and dill knows how to serialize functions (and other objects) that are defined in the interpreter.

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> 
>>> p = Pool(4)
>>> def squared(x):
...   return x**2
... 
>>> def pow(x,y):
...   return x**y
... 
>>> a = range(10)
>>> b = range(10,0,-1)
>>> 
>>> p.map(squared, a)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> res = p.amap(pow, a, b)
>>> print "asynchronous, and with multiple inputs!"
asynchronous, and with multiple inputs!
>>> res.get()
[0, 1, 256, 2187, 4096, 3125, 1296, 343, 64, 9]

Get pathos here: https://github.com/uqfoundation