Easier multiprocessing
Project description
multiprocessing_wrap
A simple interface for writing concurrent scripts.
Installation
To install using pip:
pip install multiprocess
Usage
A simple example that prints out three 1
s:
from multiprocessing_wrap import Multiprocess
m = Multiprocess(show_loading_bar=False)
def f():
print(1)
m.add_tasks(f, [(), (), ()])
m.do_tasks()
m.close()
A more involved example of sorting numbers using sleep
. Note that you only have as many workers as you have threads, so if you have 4 threads you will only be able to sort up to 4 numbers with this approach:
from multiprocessing_wrap import Multiprocess, Queue
from time import sleep
def sleep_sort():
m = Multiprocess()
q = Queue()
def f(q, x):
sleep(x)
q.push(x)
m.add_tasks(f, [(q, 1,), (q, 2,), (q, 1.5,)])
m.do_tasks()
m.close()
print('SORTED')
while not q.empty():
print(q.pop())
sleep_sort()
Error handling
Errors from within a process are propagated back to the parent with stack information. For example
from multiprocessing_wrap import Multiprocess
m = Multiprocess()
def f(x):
raise ValueError('bad error')
m.add_tasks(f, [(1,)])
m.do_tasks()
m.close()
Outputs:
Traceback (most recent call last):
File "./example3.py", line 9, in <module>
m.do_tasks()
File "/Users/dom/Documents/git/Multiprocess/src/multiprocess.py", line 53, in do_tasks
self._check_for_exceptions()
File "/Users/dom/Documents/git/Multiprocess/src/multiprocess.py", line 71, in _check_for_exceptions
"\n".join(['ERROR: ' + str(e) for e in exceptions]))
multiprocess.MultiprocessException: 1 errors occurred:
ERROR: Error in function call "f((1,))"
Traceback (most recent call last):
File "/Users/dom/Documents/git/Multiprocess/src/multiprocess.py", line 85, in my_worker
fn(*rem_args)
File "./example3.py", line 6, in f
raise ValueError('bad error')
ValueError: bad error
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for multiprocessing_wrap-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf96ddc9b54752fc6801093209f68ee35993835cf0c35fee1d55b7ee468b079c |
|
MD5 | 12422fd95ccaecf831c8e2a4990946c6 |
|
BLAKE2b-256 | 8d5757702d5fa4d343d8dd84353d512b9a603536b9cab4dbe9e11a9efe546c93 |
Close
Hashes for multiprocessing_wrap-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05240c7b66146dfab19f81f1d7bda9058153a76de42c118c5da6d725972c5fb8 |
|
MD5 | 1e9ab3eeabbee1dc464053a42eff2cdc |
|
BLAKE2b-256 | e34d3ee5dc9ccaf8b97f9cf073c5ac1d2394b588ac2bd9287a7956fa7f81059a |