dbx 0.0.9

DataBricks CLI eXtensions aka dbx

DataBricks CLI eXtensions - aka dbx is a CLI tool for advanced Databricks jobs management.

• Concept

• Requirements

• Installation

• Quickstart

• Documentation

• Differences from other tools

• Limitations

• Legal Information

• Feedback

• Contributing

Concept

dbx simplifies jobs launch and deployment process across multiple environments. It also helps to package your project and deliver it to your Databricks environment in a versioned fashion. Designed in a CLI-first manner, it is built to be actively used both inside CI/CD pipelines and as a part of local tooling for fast prototyping.

Requirements

• Python Version > 3.6

• pip or conda

Installation

• with pip:

pip install dbx

• with conda:

conda install dbx

Quickstart

As a prerequisite, you need to install databricks-cli with a configured profile. In this instruction we’re based on Databricks Runtime 7.3 LTS ML. If you don’t need to use ML libraries, we still recommend to use ML-based version due to %pip magic support.

For Python-based deployments, we recommend to use cicd-templates for quickstart. However, if you don’t like the project structure defined in cicd-templates, feel free to use the instruction below for full customization.

After configuring the profile, please do the following in the root of your project:

• Configure your project environments and storage locations:

dbx configure

• Create a conf/deployment.json file with specs defined in documentation

• Run dbx deploy to perform an initial deployment

• Run dbx launch --job=<job-name> --trace to launch the job and trace it’s status

Documentation

Please refer to the docs provided in the docs folder.

Differences from other tools

Tool	Comment
databricks-cli	dbx is NOT a replacement for databricks-cli. Quite the opposite - dbx is heavily dependent on databricks-cli and uses most of the APIs exactly from databricks-cli SDK.
mlflow cli	dbx is NOT a replacement for mlflow cli. dbx uses some of the MLflow APIs under the hood to store serialized job objects, but doesn’t use mlflow CLI directly.
Databricks Terraform Provider	While dbx is primarily oriented on versioned job management, Databricks Terraform Provider provides much wider set of infrastructure settings. In comparison, dbx doesn’t provide infrastructure management capabilities, but brings more flexible deployment and launch options.
cicd-templates	cicd-templates is a Python project template, which actively uses dbx for jobs management and CI-related operations. You can choose, whenever you would like to use this template, or use dbx separately and choose the project structure on your own.
Databricks Stack CLI	Databricks Stack CLI is a great component for managing a stack of objects. dbx concentrates on the versioning and packaging jobs together, not treating files and notebooks as a separate component.

Limitations

• Python > 3.6

• dbx execute can only be used on clusters with Databricks ML Runtime 7.X

Legal Information

This software is provided as-is and is not officially supported by Databricks through customer technical support channels. Support, questions, and feature requests can be communicated through the Issues page of this repo. Please see the legal agreement and understand that issues with the use of this code will not be answered or investigated by Databricks Support.

Feedback

Issues with dbx? Found a bug? Have a great idea for an addition? Feel free to file an issue.

Contributing

Please find more details about contributing to dbx in the contributing doc.