pytest-markdown-docs 0.9.1

Run markdown code fences through pytest

Pytest Markdown Docs

A plugin for pytest that uses markdown code snippets from markdown files and docstrings as tests.

Detects Python code fences (triple backtick escaped blocks) in markdown files as well as inline Python docstrings (similar to doctests) and runs them as tests.

Python file example:

# mymodule.py
class Foo:
    def bar(self):
        """Bar the foo

        This is a sample docstring for the bar method

        Usage:
        ```python
        import mymodule
        result = mymodule.Foo().bar()
        assert result == "hello"
        ```
        """
        return "hello"

Markdown file examples:

# Title

Lorem ipsum yada yada yada

```python
import mymodule
result = mymodule.Foo().bar()
assert result == "hello"
```

Usage

First, make sure to install the plugin:

pip install pytest-markdown-docs

To enable markdown python tests, pass the --markdown-docs flag to pytest:

pytest --markdown-docs

You can also use the markdown-docs flag to filter only markdown-docs tests:

pytest --markdown-docs -m markdown-docs

Detection conditions

Fence blocks (```) starting with the python, python3 or py language definitions are detected as tests in:

• Python (.py) files, within docstrings of classes and functions
• .md, .mdx and .svx files

Skipping tests

To exclude a Python code fence from testing, add a notest info string to the code fence, e.g:

```python notest
print("this will not be run")
```

Code block dependencies

Sometimes you might wish to run code blocks that depend on entities to already be declared in the scope of the code, without explicitly declaring them. There are currently two ways you can do this with pytest-markdown:

Injecting global/local variables

If you have some common imports or other common variables that you want to make use of in snippets, you can add them by creating a pytest_markdown_docs_globals hook in your conftest.py:

def pytest_markdown_docs_globals():
    import math
    return {"math": math, "myvar": "hello"}

With this conftest, you would be able to run the following markdown snippet as a test, without causing an error:

```python
print(myvar, math.pi)
```

Fixtures

You can use both autouse=True pytest fixtures in a conftest.py or named fixtures with your markdown tests. To specify named fixtures, add fixture:<name> markers to the code fence info string, e.g.,

```python fixture:capsys
print("hello")
captured = capsys.readouterr()
assert captured.out == "hello\n"
```

As you can see above, the fixture value will be injected as a global. For autouse=True fixtures, the value is only injected as a global if it's explicitly added using a fixture:<name> marker.

Depending on previous snippets

If you have multiple snippets following each other and want to keep the side effects from the previous snippets, you can do so by adding the continuation info string to your code fence:

```python
a = "hello"
```

```python continuation
assert a + " world" == "hello world"
```

Retrying Flaky Tests

For tests that may fail occasionally due to timing, network, or other transient issues, you can specify automatic retries using the retry:N syntax:

```python retry:3
import requests
response = requests.get("https://api.example.com")
assert response.status_code == 200
```

With retry:3, the test runs up to 4 times total (1 initial attempt + 3 retries). The test passes if any attempt succeeds.

Important notes:

• Fixtures are NOT re-run between retries - only the test code re-executes
• All exceptions trigger retries (AssertionError, RuntimeError, etc.)
• When using a continuation block, only the failing block retries

Compatibility with Material for MkDocs

Material for Mkdocs is not compatible with the default syntax.

But if the extension pymdownx.superfences is configured for mkdocs, the brace format can be used:

```{.python continuation}

You will need to call pytest with the --markdown-docs-syntax option:

pytest --markdown-docs --markdown-docs-syntax=superfences

MDX Comments for Metadata Options

In .mdx files, you can use MDX comments to provide additional options for code blocks. These comments should be placed immediately before the code block and take the following form:

{/* pmd-metadata: notest fixture:capsys */}
```python
print("hello")
captured = capsys.readouterr()
assert captured.out == "hello\n"

The following options can be specified using MDX comments:

• notest: Exclude the code block from testing.
• fixture:: Apply named pytest fixtures to the code block.
• continuation: Continue from the previous code block, allowing you to carry over state.
• retry:: Automatically retry the test up to the specified number of times if it fails.

This approach allows you to add metadata to the code block without modifying the code fence itself, making it particularly useful in MDX environments.

Customizing your own custom MarkdownIt parser

You can configure your own Markdown-it-py parser used by pytest-markdown-docs by defining a pytest_markdown_docs_markdown_it. For example, you can support mkdocs's admonitions with:

def pytest_markdown_docs_markdown_it():
    import markdown_it
    from mdit_py_plugins.admon import admon_plugin

    mi = markdown_it.MarkdownIt(config="commonmark")
    mi.use(admon_plugin)
    return mi

Testing of this plugin

You can test this module itself (sadly not using markdown tests at the moment) using pytest:

> poetry run pytest

Or for fun, you can use this plugin to include testing of the validity of snippets in this README.md file:

> poetry run pytest --markdown-docs

Known issues

• Code for docstring-inlined test discovery can probably be done better (similar to how doctest does it). Currently, seems to sometimes traverse into Python's standard library which isn't great...
• Traceback logic is extremely hacky, wouldn't be surprised if the tracebacks look weird sometimes
  • Line numbers are "wrong" for docstring-inlined snippets (since we don't know where in the file the docstring starts)
  • Line numbers are "wrong" for continuation blocks even in pure markdown files (can be worked out with some refactoring)
• There are probably more appropriate ways to use pytest internal APIs to get more features "for free" - current state of the code is a bit "patch it til' it works".
• Assertions are not rewritten w/ pretty data structure inspection like they are with regular pytest tests by default