Build large-scale task workflows using luigi, remote job submission, remote targets, and environment
Project description
Note: This project is currently under development. Version 1.0.0 will be the first, fully documented beta release, targetted for mid 2022.
Use law to build complex and large-scale task workflows. It is build on top of luigi and adds abstractions for run locations, storage locations and software environments. Law strictly disentangles these building blocks and ensures they remain interchangeable and resource-opportunistic.
Key features:
CLI with auto-completion and interactive status and dependency inspection.
- Remote targets with automatic retries and local caching
WebDAV, HTTP, Dropbox, SFTP, all WLCG protocols (srm, xrootd, rfio, dcap, gsiftp, …)
- Automatic submission to batch systems from within tasks
HTCondor, LSF, gLite, ARC, Slurm
- Environment sandboxing, configurable on task level
Docker, Singularity, Sub-Shells, Python-venv
Contents
First steps
Installation and dependencies
Install via pip:
pip install law
This command also installs luigi and six.
The (default) remote target implementation also requires gfal2 and gfal2-python (optional, also via pip) to be installed.
Usage at CERN
See the wiki.
Overcomplete example config
See law.cfg.example.
Projects using law
- CMS B-Tag SF Measurement:
Automated workflow for deriving shape-calibrating b-tag scale factors, starting at MiniAOD-level
- CMS Tau POG ML Tools:
Preprocessing pipeline for ML trainings in the TAU group
- CMS HLT Config Parser:
Collects information from various databases (HLT, bril, etc.) and shows menus, triggers paths, filter names for configurable MC datasets or data runs
- UHH-CMS Analysis Framework:
Python based, fully automated, columnar framework, including job submission, resolution of systematics and ML pipelines, starting at NanoAOD-level with an optimized multi-threaded column reader
- RWTH-CMS Analysis Framework:
Basis for multiple CMS analyses ranging from Di-Higgs, to single Higgs and b-tag SF measurements, starting at NanoAOD-level and based on coffea processors
- CIEMAT-CMS Analysis Framework:
Python and RDataFrame based framework starting from NanoAOD and targetting multiple CMS analyses
YOUR REPOSITORY HERE
If your project uses law but is not yet listed here, feel free to open a pull request or mention your project details in a new issue and it will be added.
Examples
All examples can be run either in a Jupyter notebook or a dedicated docker container. For the latter, do
docker run -ti riga/law:example <example_name>
loremipsum: The hello world example of law.
workflows: Law workflows.
dropbox_targets: Working with targets that are stored on Dropbox.
wlcg_targets: Working with targets that are stored on WLCG storage elements (dCache, EOS, …). TODO.
htcondor_at_vispa: HTCondor workflows at the VISPA service.
htcondor_at_cern: HTCondor workflows at the CERN batch infrastructure.
htcondor_at_naf: HTCondor workflows at German National Analysis Facility (NAF).
slurm_at_maxwell: Slurm workflows at the Desy Maxwell cluster.
grid_at_cern: Workflows that run jobs and store data on the WLCG.
lsf_at_cern: LSF workflows at the CERN batch infrastructure.
docker_sandboxes: Environment sandboxing using Docker. TODO.
singularity_sandboxes: Environment sandboxing using Singularity. TODO.
subshell_sandboxes: Environment sandboxing using Subshells. TODO.
parallel_optimization: Parallel optimization using scikit optimize.
notifications: Demonstration of slack and telegram task status notifications..
CMS Single Top Analysis: Simple physics analysis using law.
Further topics
Auto completion on the command-line
bash
source "$( law completion )"
zsh
zsh is able to load and evaluate bash completion scripts via bashcompinit. In order for bashcompinit to work, you should run compinstall to enable completion scripts:
autoload -Uz compinstall && compinstall
After following the instructions, these lines should be present in your ~/.zshrc:
# The following lines were added by compinstall
zstyle :compinstall filename '~/.zshrc'
autoload -Uz compinit
compinit
# End of lines added by compinstall
If this is the case, just source the law completion script (which internally enables bashcompinit) and you’re good to go:
source "$( law completion )"
Tests
To run and test law, there are three docker images available on the DockerHub, corresponding to Python versions 2.7, and 3.7 to 3.10. They are based on CentOS 7 and ship with the dependencies listed above, including gfal2.
docker run -ti riga/law:latest
Tags:
py310: Latest Python 3.10
py39, py3, latest: Latest Python 3.9
py38: Latest Python 3.8
py37: Latest Python 3.7
py2, py27: Python 2.7
example: Example runner, based on latest (see below)
Development
Source hosted at GitHub
Report issues, questions, feature requests on GitHub Issues
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.