Skip to main content

Dask Cluster objects in Saturn Cloud

Project description

dask-saturn

Python library for interacting with Dask clusters in Saturn Cloud.

Dask-Saturn mimics the API of Dask-Kubernetes, but allows the user to interact with clusters created within Saturn Cloud.

Start cluster

In order to interact with a Dask cluster, the cluster must first be created in the Saturn User Interface. Then, from within a Jupyter notebook, you can start the cluster and adjust the number of workers.

from dask_saturn import SaturnCluster

cluster = SaturnCluster()
cluster

Adjust number of workers

Once you have a cluster you can interact with it via the jupyter widget, or using the scale and adapt methods.

For example, to manually scale up to 20 workers:

cluster.scale(20)

To create an adaptive cluster that controls its own scaling:

cluster.adapt(minimum=1, maximum=20)

Interact with client

To submit tasks to the cluster, you sometimes need access to the Client object. Instantiate this with the cluster as the only argument:

from distributed import Client

client = Client(cluster)
client

Close cluster

To terminate all resources associated with a cluster, use the close method:

cluster.close()

Development

Create/update a dask-saturn conda environment:

make conda-update

Set environment variables to run dask-saturn with a local atlas server:

export BASE_URL=http://dev.localtest.me:8888/
export SATURN_TOKEN=<JUPYTER_SERVER_SATURN_TOKEN>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-saturn-0.0.3.tar.gz (21.7 kB view hashes)

Uploaded Source

Built Distribution

dask_saturn-0.0.3-py3-none-any.whl (6.4 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page