Library to handle sparse bytes within a virtual memory space
Project description
Library to handle sparse bytes within a virtual memory space.
Free software: BSD 2-Clause License
Objectives
This library aims to provide utilities to work with a virtual memory, which constsis in a virtual addressing space where sparse chunks of data can be stored.
In order to be easy to use, its interface should be close to that of a bytearray, which is the closest pythonic way to store dynamic data. The main downside of a bytearray is that it requires a contiguous data allocation starting from address 0. This is not good when sparse data have to be stored, such as when emulating the addressing space of a generic microcontroller.
The main idea is to provide a bytearray-like class with the possibility to internally hold the sparse blocks of data. A block is ideally a tuple (start, data) where start is the start address and data is the container of data items (e.g. bytearray). The length of the block is len(data). Those blocks are usually not overlapping nor contiguous, and sorted by start address.
Python implementation
This library is the Cython complement to the Python implementation provided by the bytesparse Python package. Please refer to its own documentation for more details.
Cython implementation
The library provides an experimental Cython implementation. It tries to mimic the same algorithms of the Python implementation, while exploiting the speedup of compiled C code.
Beware that the Cython implementation is meant to be potentially faster than the pure Python one, but there might be even faster ad-hoc implementations of virtual memory highly optimized for the underlying hardware.
The addressing space is limited to that of an uint_fast64_t (typically 32-bit or 64-bit as per the hosting machine), so it is not possible to have an infinite addressing space, nor negative addresses. To keep the implementation code simple enough, the highest address (e.g. 0xFFFFFFFF on a 32-bit machine) is reserved.
Block data chunks cannot be greater than the maximum ssize_t value (typically half of the addressing space).
The Cython implementation is optional, and potentially useful only when the Python implementation seems too slow for the user’s algorithms, within the limits stated above.
If in doubt about using the Cython implementation, just stick with the Python one, which is much easier to integrate and debug.
More details can be found within cbytesparse._c.
Documentation
For the full documentation, please refer to:
Installation
From PIP (might not be the latest version found on github):
$ pip install cbytesparse
From source root directory:
$ pip install .
Development
To regenerate the Cython files, run the following commands:
$ python scripts/cython_build_src.py
$ python scripts/cython_build_tests.py
To run the all the tests:
$ pip install tox
$ tox
Changelog
0.0.2 (2022-01-03)
Forced extension compilation.
Using explicit factory methods instead of constructor arguments.
Added block collapsing helper function.
Minor fixes.
Improved test suite.
0.0.1 (2021-12-27)
First release on PyPI.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.