Services for compression and transfer of aind-data to the cloud
Project description
aind-data-transfer
Tools for transferring large data to and between cloud storage providers.
Installation
To upload data to aws s3, you may need to install and configure awscli
. To upload data to gcp, you may need to install and configure gsutil
.
Imaging
- Run
pip install -e .[imaging]
- Run
./post_install.sh
Ephys
- Run
pip install -e .[ephys]
Full
- Run
pip install -e .[full]
- Run
./post_install.sh
Development
- Run
pip install -e .[dev]
- Run
./post_install.sh
Contributing
Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the coverage library, which will run the tests and log a coverage report:
coverage run -m unittest discover && coverage report
- Use interrogate to check that modules, methods, etc. have been documented thoroughly:
interrogate .
- Use flake8 to check that code is up to standards (no unused imports, etc.):
flake8 .
- Use black to automatically format the code into PEP standards:
black .
- Use isort to automatically sort import statements:
isort .
Pull requests
For internal members, please create a branch. For external members, please fork the repo and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:
<type>(<scope>): <short summary>
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- build: Changes that affect the build system or external dependencies (example scopes: pyproject.toml, setup.py)
- ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- docs: Documentation only changes
- feat: A new feature
- fix: A bug fix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- test: Adding missing tests or correcting existing tests
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aind-data-transfer-0.1.23.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f8c9d5b17f2d36834e8d2e23c623ca4c8c3e96d9c290123a5264aaa24b3a82c3 |
|
MD5 | 310f0ca27023ad8a97f6d1e148d6c7b6 |
|
BLAKE2b-256 | ffe61bf86379b9189a3f9d211f7c39503d0301558cac6d1f14db4ee7b6febfa2 |
Hashes for aind_data_transfer-0.1.23-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0cd377c30b1c4d351ece5d52a322f21dc1caa0276abfcee334e1306dfb66045a |
|
MD5 | a267bc60d0e6d6839f7c995e40cb5603 |
|
BLAKE2b-256 | 242884b8bf334344ddc5eb3142176d95b0f6679c1febf552e99b2e0a135fde93 |