tucan 0.0.0

Easy creation of workflows for recursive and farming HPC jobs

Lemmings is a 1991 video game where the player try to herd small animals, the "lemmings" out of a a 2D puzzle. Lemmings are clueless about their surroundings, walk blindly, and will eventually fall, burn, be crushed, ... well die, unless the player personally take care of them. The "Lemmings Jobs ", introduced here, are the same : by nature, these unsupervised job submission often end up in dramatic failures. Human oversight is compulsory when you are dealing with chained runs.

Lemmings

Idea

Lemmings is an open-source code designed to simplify the submission of multiple inter-dependent jobs on the schedulers of HPC clusters. While originally developed within the context of Computational Fluid Dynamics (CFD) applications, it is adapted to many recursive jobs. A farming mode is present to help the replication of these recursive jobs for a parametric study.

Installation

Lemmings is open-source and can be pip-installed :

pip install lemmings-hpc

End user POV

The end-user of lemmings is someone making a lot of simulations with a repetitive pattern. This repetition (eg. resubmit the job until simulated time reaches 1ms) is automated by a lemmings "workflow", a python file gathering all the logic of the application. This "workflow" was created by a super user using lemmings.

Here The end-user (John Doe) adds the workflow (sandcastle) file where he usually launches the run, then run the lemmings run command:

>lemmings run --machine-file sandbox.yml --job-prefix funtask sandcastle
INFO - 
##############################
Starting Lemmings 0.8.0...
##############################

INFO -     Job name     :funtask_PAJI77
INFO -     Loop         :1
INFO -     Status       :start
INFO -     Worflow path :/Users/johndoe/productionpath/sandcastle.py
INFO -     Imput path   :/Users/johndoe/productionpath/sandcastle.yml
INFO -     Machine path :/Users/johndoe/productionpath/sandbox.yml
INFO -     Farming mode :False
INFO -     Lemmings START (1/3)
INFO -          Check on startTrue (False -> Exit)
INFO -          Prior to job
INFO -     Lemmings SPAWN (2/3)
INFO -          Prepare run
INFO -          Submit batch 74148 
INFO -          Submit batch post job 74149

This execution will be called funtask_PAJI77 and will automatically submit runs through the job schedulers. On the job scheduler, he will find something like

+----------------+---------------+-------+----------+-------------------+---------+
|    job name    |     queue     | pid   |  state   |    last update    |  after  |
+----------------+---------------+-------+----------+-------------------+---------+
| funtask_PAJI77 |  long00:00:30 | 74148 |   done   | 06/13/22 15:22:52 |    -    |
| funtask_PAJI77 | short00:00:10 | 74149 |  running | 06/13/22 15:22:53 |  74148  |
+----------------+---------------+-------+----------+-------------------+---------+

Here jobs funtask_PAJI77_74148 and funtask_PAJI77_74149 are the two first dependent jobs of the workflow, but more will come. The decision to re-submit and the creation of the next job will be handled by funtask_PAJI77_74149 after completion. Therefore Lemmings does not "book" consecutive PID on start, only the next jobs are queued.

Finally lemmings is not moving/hiding log files automatically. By actively limiting such "black magic", it enforces an experience similar to manual re-submission

Creating a workflow

A super-user creates a workflow by injecting code into some parts of a baseline Loop. The default, simplified, lemmings job follows this algorithm:

                +-----------+                     +------------+True  
Start---------->|Prepare Run+--->Job submission--->Check on end+----------->Happy
            ^   +-----------+                     +------+-----+             End
            |                                            |
            |                                            |False
            |                                            |
            |                                            |
            +--------------------------------------------+                          

By adding code to Prepare Run phase (updates of input file) and to Check on end (when to stop the job), the super-user can customize it to his needs. Follows the HowTos for an extended explanation.

Resources

Lemmings documentation can be found following this link : lemmings documentation

Acknowledgements

Lemmings is a service created in the EXCELLERAT Center Of Excellence and is continued as part of the COEC Center Of Excellence. Both projects are funded by the European community.