Skip to main content

Reusable and maintained Luigi tasks to incorporate in bioinformatics pipelines

Project description

Bioluigi

Python package Documentation Status

Reusable and maintained Luigi tasks to incorporate in bioinformatics pipelines

Features

Provides Luigi tasks for tools from samtools, bcftools, STAR, RSEM, vcfanno, GATK, Ensembl VEP and much more!

Reuses as much as possible the ExternalProgramTask interface from the external_program contrib module and extends its feature to make it work on modern scheduler such as Slurm.

Provides basic resource management for a local scheduler: all tasks are annotated with reasonable default cpus and memory parameters that can be tuned and constrained via the [resources] configuration. In the case of externally scheduled tasks, the resource management is deferred.

Provides a command-line interface for interacting more conveniently with Luigi scheduler.

bioluigi list [--status STATUS] [--user USER] [--detailed] TASK_GLOB
bioluigi show TASK_ID

Tools

Here's a list of supported tools:

Schedulers

Examples

The most convenient way of using the pre-defined tasks is to yield them dynamically in the body of the run function. It's also possible to require them since they inherit from luigi.Task.

import luigi
from bioluigi.tasks import bcftools

def MyTask(luigi.Task)
    def input(self):
        return luigi.LocalTarget('source.vcf.gz')

    def run(self):
        yield bcftools.Annotate(self.input().path,
                                annotations_file,
                                self.output().path,
                                ...,
                                scheduler='slurm',
                                cpus=8)

    def output(self):
        return luigi.LocalTarget('annotated.vcf.gz')

You can define your own scheduled task by implementing the ScheduledExternalProgramTask class. Note that the default scheduler is local and will use Luigi's [resources] allocation mechanism.

import datetime
from bioluigi.scheduled_external_program import ScheduledExternalProgramTask

class MyScheduledTask(ScheduledExternalProgramTask):
    scheduler = 'slurm'
    walltime = datetime.timedelta(seconds=10)
    cpus = 1
    memory = 1

    def program_args(self):
        return ['sleep', '10']

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bioluigi-0.1.3.tar.gz (20.9 kB view hashes)

Uploaded Source

Built Distribution

bioluigi-0.1.3-py3-none-any.whl (28.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page