Skip to main content

Pytest plugin for testing notebooks

Project description

nbmake

codecov PyPI versions PyPI versions PyPI Downloads Gitpod ready-to-code

What? Pytest plugin for testing and releasing notebook documentation

Why? To raise the quality of scientific material through better automation

Who is this for? Research/Machine Learning Software Engineers who maintain packages/teaching materials with documentation written in notebooks.

Functionality

  1. Executes notebooks using pytest and nbclient, allowing parallel notebook testing
  2. Optionally writes back to the repo, allowing faster building of nbsphinx or jupyter book docs

Quick Start

pip install pytest nbmake
pytest --nbmake **/*ipynb

Configure Cell Timeouts

You can configure the cell timeout with the following pytest flag:

pytest --nbmake --nbmake-timeout=3000 # allows each cell 3000 seconds to finish

Allow Errors For a Whole Notebook

This configuration must be placed in the notebook's top-level metadata (not cell-level metadata).

Your notebook should look like this:

{
  "cells": [ ... ],
  "metadata": {
    "kernelspec": { ... },
    "execution": {
      "allow_errors": true,
      "timeout": 300
    }
  }
}

Allow a Cell to Throw an Exception

A cell with the following metadata can throw an exception without failing the test:

{
  "language": "python",
  "custom": {
    "metadata": {
      "tags": [
        "raises-exception"
      ]
    }
  }
}

Ignore a Code Cell

A cell with the following metadata will not be executed by nbmake

{
  "language": "python",
  "custom": {
    "metadata": {
      "tags": [
        "skip-execution"
      ]
    }
  }
}

Override Notebook Kernels when Testing

Regardless of the kernel configured in the notebook JSON, you can force nbmake to use a specific kernel when testing:

pytest --nbmake --nbmake-kernel=mycustomkernel

Add Missing Jupyter Kernel to Your CI Environment

If you are not using the flag above and are using a kernel name other than the default ‘python3’, you will see an error message when executing your notebooks in a fresh CI environment: Error - No such kernel: 'mycustomkernel'

Use ipykernel to install the custom kernel:

python -m ipykernel install --user --name mycustomkernel

If you are using another language such as c++ in your notebooks, you may have a different process for installing your kernel.

Parallelisation

Parallelisation with xdist is experimental upon initial release, but you can try it out:

pip install pytest-xdist

pytest --nbmake -n=auto

It is also possible to parallelise at a CI-level using strategies, see example

Build Jupyter Books Faster

Using xdist and the --overwrite flag let you build a large jupyter book repo faster:

pytest --nbmake --overwrite -n=auto examples
jb build examples

Advice on Usage

nbmake is best used in a scenario where you use the ipynb files only for development. Consumption of notebooks is primarily done via a docs site, built through jupyter book, nbsphinx, or some other means. If using one of these tools, you are able to write assertion code in cells which will be hidden from readers.

Pre-commit

Treating notebooks like source files lets you keep your repo minimal. Some tools, such as plotly may drop several megabytes of javascript in your output cells, as a result, stripping out notebooks on pre-commit is advisable:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/kynan/nbstripout
    rev: master
    hooks:
      - id: nbstripout

See https://pre-commit.com/ for more...

Disable Nbmake

Implicitly:

pytest

Explicitly:

pytest -p no:nbmake

See Also:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nbmake-1.2.1.tar.gz (10.7 kB view hashes)

Uploaded Source

Built Distribution

nbmake-1.2.1-py3-none-any.whl (11.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page