image-dataset-converter 0.1.0

Python3 library for converting between various image annotation dataset formats.

The image-dataset-converter library allows the conversion between various dataset formats for image annotation datasets. Filters can be supplied as well, e.g., for cleaning up the data.

Dataset formats:

• depth data: CSV (r/w), grayscale (r/w), numpy (r/w), PFM (r/w)

• image classification: ADAMS (r/w), sub-dir (r/w)

• image segmentation: blue-channel (r/w), grayscale (r/w), indexed PNG (r/w), layer segments (r/w)

• object detection: ADAMS (r/w), COCO (r/w), OPEX (r/w), ROI (r/w), VOC (r/w), YOLO (r/w)

Examples can be found here:

https://github.com/waikato-datamining/image-dataset-converter-examples

Changelog

0.1.0 (2025-10-31)

• added dims-to-metadata filter for transferring image dimension to metadata

• centralized comparison code in idc.api._comparison (available via idc.api)

• split-records filter now allows specifying the meta-data field in which to store the split name

• the tee meta-filter can now forward or drop the incoming data based on a meta-data evaluation

• added sub-process filter for processing data with sub-flow of filters, can be conditional based on meta-data evaluation

• the metadata-from-name filter can work on the path now as well (must be present)

• the label-present-is filter now lets the user to specify a minimum number of pixels that need to be present in the layers

• switched to kasperl library for base API and generic pipeline plugins

• requiring seppl>=0.3.0 now

• added @abc.abstractmethod decorator where appropriate

• the use-mask filter now sets the image format PNG in the generated output container

• the readers from-grayscale-dp, from-indexed-png-is, from-blue-channel-is and from-grayscale-is now support reading only the annotations

• added idc.api.binarize_image convenience function

• added idc.api.image_to_bytesio helper method

• the idc-exec tool now uses all remaining parameters as the pipeline components rather than having to specify them via the -p/–pipeline parameter, making it easy to simply prefix the idc-exec command to an existing idc-convert command-line

• added the text-file and csv-file generators that work off files to populate the variable(s)

• idc-exec can load pipelines from file now as well, useful when dealing with large pipelines

• added –load_pipeline option to idc-convert

• added from-text-file reader and to-text-file writer

• readers now locate files the first time the read() method gets called rather than in the initialized(), to allow more dynamic placeholders

• added block, stop filters for controlling the flow of data (via meta-data conditions)

• added email support with get-email reader and send-email writer

• added list-files reader for listing files in a directory

• added list-to-sequence stream filter that forwards list items one by one

• added console writer for outputting the data on stdout that is coming through

• added move-files filter which moves incoming files to a target directory

• added watch-dir meta-reader that uses the watchdog library to react to file-system events rather than using fixed-interval polling like poll-dir

• added delete-files writer

• added copy-files filter

• added count-specks filter that adds counts of small objects to meta-data

• added support for caching plugins via IDC_CLASS_CACHE environment variable

• od-to-is now handles labels or regexp pattern correctly

• added is-to-od filter that generates object detection annotations from contours determined in image segmentation layers

• added to-metadata writer that outputs the meta-data of an image

• added attach-metadata filter that loads meta-data from a directory and attaches it to the data passing through

• added load-data filter to turn file names into data containers

• added annotation-to-storage and annotation-from-storage filters

• annotation data is now being type-checked when setting it

• added delete-storage filter for removing objects from internal storage

0.0.13 (2025-07-15)

• requiring seppl>=0.2.20 now for improved help requests in idc-convert tool

0.0.12 (2025-07-11)

• dropped numpy<2.0.0 restriction

• added grayscale-to-binary filter

• fix: sort-pixels, rgb-to-grayscale filters

• added ensure_grayscale and grayscale_required_info convenience methods (package: idc.api)

• added ensure_binary and binary_required_info convenience methods (package: idc.api)

• added –dump_pipeline option to idc-convert for saving the pipeline command

• the rename filter now supports lower/upper case placeholders of name and extension as well

• requiring seppl>=0.2.17 now for skippable plugin support and avoiding deprecated use of pkg_resources

• added any-to-rgb filter for turning binary/grayscale images back into RGB ones

• using wai_common instead of wai.common now

• requiring fast_opex>=0.0.4 now

• added label-to-metadata filter for transferring labels into meta-data

• added metadata-to-placeholder filter for transferring meta-data into placeholders

• added basic support for images with associated depth information: DepthData, DepthInformation

• added depth-to-grayscale filter for converting depth information to grayscale image

• prefixed image segmentation methods like from_bluechannel and to_bluechannel with imgseg_

• added depth information readers from-grayscale-dp, from-numpy-dp, from-csv-dp and from-pfm-dp

• added depth information writers to-grayscale-dp, to-numpy-dp, to-csv-dp and to-pfm-dp

• added apply-ext-mask filter to applying external PNG masks to image containers (image and/or annotations)

• added apply-label-mask filter for applying image segmentation label masks to their base images

• added label-present-ic and label-present-is that ensure that certain label(s) are present or otherwise discard the image

• filter label-present was renamed to label-present-od but keeping label-present as alias for the time being

• fix: imgseg_to_bluechannel, imgseg_to_indexedpng and imgseg_to_grayscale now handle overlapping pixels correctly, no longer adding them up and introducing additional labels

• discard-by-name filter can use names of files in specified paths now as well

• fixed the construction of the error messages in the pyfunc reader/filter/writer classes

0.0.11 (2025-04-03)

• fix: idc-registry –list writers now returns writer plugins instead of reader ones

0.0.10 (2025-04-03)

• added set-placeholder filter for dynamically setting (temporary) placeholders at runtime

• added –resume_from option to relevant readers that allows resuming the data processing from the first file that matches this glob expression (e.g., */012345.png)

• requiring seppl>=0.2.14 now for resume support

• using underscores now instead of dashes in dependencies (setup.py)

• the array_to_image method no longer performs unnecessary conversions of Image objects

• the dirs generator can limit directories now to ones that have files matching a specific regexp (–file_regexp), to avoid the Failed to locate any files using: … error message when a reader doesn’t find any matching files

• requiring seppl>=0.2.15 now for new –split_group support

• added the from-multi meta-reader that combines multiple base readers and returns their output

• added the to-multi meta-writer that forwards the data to multiple base writers

• added the use-mask filter for using the image segmentation annotations (= mask) as the new base image

0.0.9 (2025-03-14)

• using wai_logging instead of wai.logging as dependency now

0.0.8 (2025-03-14)

• requiring seppl>=0.2.13 now for placeholder support

• added placeholder support to tools: idc-convert, idc-exec

• added placeholder support to readers: from-adams-ic, from-subdir-ic, from-blue-channel-is, from-grayscale-is, from-indexed-png-is, from-layer-segments-is, from-adams-od, from-coco-od, from-opex-od, from-roicsv-od, from-voc-od, from-yolo-od, from-data, from-pyfunc, poll-dir

• added placeholder support to filters: write-labels

• added placeholder support to writers: to-adams-ic, to-subdir-ic, to-blue-channel-is, to-grayscale-is, to-indexed-png-is, to-layer-segments-is, to-adams-od, to-coco-od, to-opex-od, to-roicsv-od, to-voc-od, to-yolo-od, to-data

0.0.7 (2025-03-12)

• added safe_deepcopy method to idc.api._utils which creates a deep copy of an object if not None

• added rgb-to-grayscale filter to convert color images into gray scale ones

• added sort-pixels filter for grayscale images

• the following filters can operate on lists of records now as well: inspec, metadata, metadata-from-name

• added metadata-od filter for filtering object-detection annotations based on their meta-data (e.g., scores from model predictions)

• the filters discard-negatives and discard-invalid-images now output how many were discarded/kept when processing finishes

0.0.6 (2025-02-26)

• LayerSegmentsImageSegmentationReader now suggest using –lenient flag in exception in case image not binary

• added the discard-by-name filter that allows user to discard images based on name, either exact match of regexp (matching sense can be inverted)

• requiring seppl>=0.2.10 now

• added support for aliases

• added to_bluechannel, to_grayscale and to_indexedpng image segmentation methods to idc.api

• added the generate_palette_list method to idc.api which turns a predefined palette name or comma-separated list of RGB values into a flat list of int values, e.g., used for indexed PNG files

• exposed method save_image through idc.api

• filter-labels now handles not specifying any labels and only regexp

• write-labels filter now allows specification of custom separator

• write-labels: fixed retrieval of image-segmentation labels

• using simple_palette_utils dependency now

• idc-convert tool now flags aliases on the help screen with *

• the from-voc-od reader now has the -r/–image_rel_path option which gets injected before the folder property from the XML file

0.0.5 (2025-01-13)

• added setuptools as dependency

• switched to underscores in project name

• using 90% as default quality for JPEG images now, can be overridden with environment variable IDC_JPEG_QUALITY

• added methods to idc.api module: jpeg_quality(), array_to_image(…), empty_image(…)

0.0.4 (2024-07-16)

• limiting numpy to <2.0.0 due to problems with imgaug library

0.0.3 (2024-07-02)

• switched to the fast-opex library

• helper method from_indexedpng was using incorrect label index (off by 1)

• Data.save_image method now ensures that source/target files exist before calling os.path.samefile

• requiring seppl>=0.2.6 now

• readers now support default globs, allowing the user to just specify directories as input (and the default glob gets appended)

• the to-yolo-od writer now has an option for predefined labels (for enforcing label order)

• the to-yolo-od writer now stores the labels/labels_cvs files in the respective output folders rather than using an absolute file name

• the bluechannel/grayscale/indexed-png image segmentation readers/writers can use a value other than 0 now for the background

• split filter has been renamed to split-records

0.0.2 (2024-06-13)

• added generic plugins that take user Python functions: from-pyfunc, pyfunc-filter, to-pyfunc

• added idc-exec tool that uses generator to produce variable/value pairs that are used to expand the provided pipeline template which then gets executed

• added polygon-simplifier filter for reducing number of points in polygons

• moved several geometry/image related functions from imgaug library into core library to avoid duplication

• added python-image-complete as dependency

• the ImageData class now uses the python-image-complete library to determine the file format rather than loading the image into memory in order to determine that

• the convert-image-format filter now correctly creates a new container with the converted image data

• the to-coco-od writer only allows sorting of categories when using predefined categories now

• the from-opex-od reader now handles absent meta-data correctly

• added the AnnotationsOnlyWriter mixin for writers that can skip the base image and just output the annotations

0.0.1 (2024-05-06)

• initial release