redflag 0.1.8rc3

Raise red flags on machine learning datasets.

Redflag

🚩 redflag aims to be an automatic safety net for machine learning datasets. The vision is to accept input of a Pandas DataFrame or NumPy ndarray (one for each of the input X and target y in a machine learning task). redflag will provide an analysis of each feature, and of the target, including aspects such as class imbalance, outliers, anomalous data patterns, threats to the IID assumption, and so on. The goal is to complement other projects like pandas-profiling and greatexpectations.

This project is very rough and does not do much yet. The API will very likely change without warning. Please consider contributing!

Installation

You can install this package with pip:

pip install redflag

Installing scikit-learn allows you to access some extra options for outlier detection.

pip install redflag[sklearn]

For developers, there are also options for installing tests, docs and dev dependencies.

Example

redflag is currently just a collection of functions. Most of the useful ones take a single column of data (e.g. a 1D NumPy array) and run a single test. For example, we can do some outlier detection:

>>> import redflag as rf
>>> data = [-3, -2, -2, -1, 0, 0, 0, 1, 2, 2, 3]
>>> rf.has_outliers(data)
array([], dtype=int64)
>>> rf.has_outliers(3 * data + [100])
array([100])

See the notebook Using_redflag.ipynb for several other examples.

Documentation

The documentation is online.

Contributing

Please see CONTRIBUTING.md. There is also a section in the documentation about Development.

Testing

You can run the tests (requires pytest and pytest-cov) with

python run_tests.py

Most of the tests are run with doctest.