Extended pickling support for Python objects
Project description
# cloudpickle
[](https://travis-ci.org/cloudpipe/cloudpickle)
[](https://codecov.io/github/cloudpipe/cloudpickle?branch=master)
`cloudpickle` makes it possible to serialize Python constructs not supported
by the default `pickle` module from the Python standard library.
`cloudpickle` is especially useful for **cluster computing** where Python
code is shipped over the network to execute on remote hosts, possibly close
to the data.
Among other things, `cloudpickle` supports pickling for **lambda functions**
along with **functions and classes defined interactively** in the
`__main__` module (for instance in a script, a shell or a Jupyter notebook).
**`cloudpickle` uses `pickle.HIGHEST_PROTOCOL` by default**: it is meant to
send objects between processes running the **same version of Python**.
Using `cloudpickle` for **long-term object storage is not supported and
discouraged.**
Installation
------------
The latest release of `cloudpickle` is available from
[pypi](https://pypi.python.org/pypi/cloudpickle):
pip install cloudpickle
Examples
--------
Pickling a lambda expression:
```python
>>> import cloudpickle
>>> squared = lambda x: x ** 2
>>> pickled_lambda = cloudpickle.dumps(squared)
>>> import pickle
>>> new_squared = pickle.loads(pickled_lambda)
>>> new_squared(2)
4
```
Pickling a function interactively defined in a Python shell session
(in the `__main__` module):
```python
>>> CONSTANT = 42
>>> def my_function(data):
... return data + CONSTANT
...
>>> pickled_function = cloudpickle.dumps(my_function)
>>> pickle.loads(pickled_function)(43)
85
```
Running the tests
-----------------
- With `tox`, to test run the tests for all the supported versions of
Python and PyPy:
pip install tox
tox
or alternatively for a specific environment:
tox -e py37
- With `py.test` to only run the tests for your current version of
Python:
pip install -r dev-requirements.txt
PYTHONPATH='.:tests' py.test
History
-------
`cloudpickle` was initially developed by [picloud.com](http://web.archive.org/web/20140721022102/http://blog.picloud.com/2013/11/17/picloud-has-joined-dropbox/) and shipped as part of
the client SDK.
A copy of `cloudpickle.py` was included as part of PySpark, the Python
interface to [Apache Spark](https://spark.apache.org/). Davies Liu, Josh
Rosen, Thom Neale and other Apache Spark developers improved it significantly,
most notably to add support for PyPy and Python 3.
The aim of the `cloudpickle` project is to make that work available to a wider
audience outside of the Spark ecosystem and to make it easier to improve it
further notably with the help of a dedicated non-regression test suite.
[](https://travis-ci.org/cloudpipe/cloudpickle)
[](https://codecov.io/github/cloudpipe/cloudpickle?branch=master)
`cloudpickle` makes it possible to serialize Python constructs not supported
by the default `pickle` module from the Python standard library.
`cloudpickle` is especially useful for **cluster computing** where Python
code is shipped over the network to execute on remote hosts, possibly close
to the data.
Among other things, `cloudpickle` supports pickling for **lambda functions**
along with **functions and classes defined interactively** in the
`__main__` module (for instance in a script, a shell or a Jupyter notebook).
**`cloudpickle` uses `pickle.HIGHEST_PROTOCOL` by default**: it is meant to
send objects between processes running the **same version of Python**.
Using `cloudpickle` for **long-term object storage is not supported and
discouraged.**
Installation
------------
The latest release of `cloudpickle` is available from
[pypi](https://pypi.python.org/pypi/cloudpickle):
pip install cloudpickle
Examples
--------
Pickling a lambda expression:
```python
>>> import cloudpickle
>>> squared = lambda x: x ** 2
>>> pickled_lambda = cloudpickle.dumps(squared)
>>> import pickle
>>> new_squared = pickle.loads(pickled_lambda)
>>> new_squared(2)
4
```
Pickling a function interactively defined in a Python shell session
(in the `__main__` module):
```python
>>> CONSTANT = 42
>>> def my_function(data):
... return data + CONSTANT
...
>>> pickled_function = cloudpickle.dumps(my_function)
>>> pickle.loads(pickled_function)(43)
85
```
Running the tests
-----------------
- With `tox`, to test run the tests for all the supported versions of
Python and PyPy:
pip install tox
tox
or alternatively for a specific environment:
tox -e py37
- With `py.test` to only run the tests for your current version of
Python:
pip install -r dev-requirements.txt
PYTHONPATH='.:tests' py.test
History
-------
`cloudpickle` was initially developed by [picloud.com](http://web.archive.org/web/20140721022102/http://blog.picloud.com/2013/11/17/picloud-has-joined-dropbox/) and shipped as part of
the client SDK.
A copy of `cloudpickle.py` was included as part of PySpark, the Python
interface to [Apache Spark](https://spark.apache.org/). Davies Liu, Josh
Rosen, Thom Neale and other Apache Spark developers improved it significantly,
most notably to add support for PyPy and Python 3.
The aim of the `cloudpickle` project is to make that work available to a wider
audience outside of the Spark ecosystem and to make it easier to improve it
further notably with the help of a dedicated non-regression test suite.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cloudpickle-0.8.1.tar.gz
(30.8 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file cloudpickle-0.8.1.tar.gz.
File metadata
- Download URL: cloudpickle-0.8.1.tar.gz
- Upload date:
- Size: 30.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3ea6fd33b7521855a97819b3d645f92d51c8763d3ab5df35197cd8e96c19ba6f
|
|
| MD5 |
1d3738ffae998e429795c7bd3831352a
|
|
| BLAKE2b-256 |
339cb785fe61fd43fb673532caca18d6c398e86faaf7d283266c7ed23db4ee5e
|
File details
Details for the file cloudpickle-0.8.1-py2.py3-none-any.whl.
File metadata
- Download URL: cloudpickle-0.8.1-py2.py3-none-any.whl
- Upload date:
- Size: 16.1 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
70b492d5e06b779707c00b334311abcd4ac144c56910068aeeda0079e6244ef0
|
|
| MD5 |
7cdbd231296ef6435b3ac4822235aa86
|
|
| BLAKE2b-256 |
6ebc67f13115adcce4efc5e4d7f8220fb9a50aaa2b5c7ed460b26cbb76aa76ad
|