aliyunoss2-autoupload 0.1

Monitor files by wildcard patterns, upload them to ALIYUN OSS, then move to backup directory.

Source:

https://github.com/tanbro/aliyunoss2-autoupload

Package:

https://pypi.org/project/aliyunoss2-autoupload/

aliyunoss2-autoupload

Monitor files by wildcard patterns, upload them to ALIYUN OSS, then move to backup directory.

Usage

Command Line

After the package installed, run the command in a terminator, show help messages

$ aliyunoss2-autoupload --help

usage: aliyunoss2-autoupload [-h] [--version] {run,echo_config_sample} ...

Watch files in a directory and upload them to Aliyun OSS on file writing
completed

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

subcommands:

  {run,echo_config_sample}
                        <sub_command --help> Print the help of sub_commands
    run                 Start to run then program. It will monitor and upload
                        files continuously.
    echo_config_sample  Echo configure file sample

$ aliyunoss2-autoupload run --help

usage: aliyunoss2-autoupload run [-h] [--only-once]
                                 [--config-file CONFIG_FILE]
                                 [--logging-config-file LOGGING_CONFIG_FILE]

optional arguments:
  -h, --help            show this help message and exit
  --only-once, -o       Upload only once, then exit. Will NOT monitor files.
                        (default=False)
  --config-file CONFIG_FILE, -c CONFIG_FILE
                        The program configuration file. The program will first
                        try to load configuration file by environment variable
                        ${ALIYUNOSS2_AUTOUPLOAD_CONF}. If the environment
                        variable not assigned, then try to load configuration
                        file "conf/aliyunoss2-autoupload.yml"
  --logging-config-file LOGGING_CONFIG_FILE, -l LOGGING_CONFIG_FILE
                        The logging configuration file. The program will first
                        try to load logging configuration file by environment
                        variable ${ALIYUNOSS2_AUTOUPLOAD_LOG_CONF}. If the
                        environment variable not assigned, then try to load
                        configuration file
                        "conf/aliyunoss2-autoupload.log.yml"

$ aliyunoss2-autoupload echo_config_sample --help

usage: aliyunoss2-autoupload echo_config_sample [-h] {prog,log}

positional arguments:
{prog,log}  Configure file to echo

optional arguments:
-h, --help  show this help message and exit

Configuration File

The program will first try to load configuration file from environment variable ALIYUNOSS2_AUTOUPLOAD_CONF. If the environment variable not assigned, then try to load configuration file “conf/aliyunoss2-autoupload.yml”.

The YAML file is like blow:

---

## Aliyun OSS configs
oss:
    ## Name of your Aliyun OSS bucket
    name: "your_bucket_name"
    ## Endpoint URL of Aliyun OSS bucket
    endpoint: "oss-xx-xxxxxx.aliyuncs.com"
    ## cname of the domain of Aliyun OSS bucket. Empty if no cname.
    cname: ""
    ## Access Key ID of Aliyun OSS bucket
    access_key_id: "your_access_key_id"
    ## Access Key Secret of Aliyun OSS bucket
    access_key_secret: "your_access_key_secret"

## Directory name configs
dir:
    ## Calculate uploading file relative name by this local directory
    rel_dir: ""
    ## Upload files to OSS in this dir
    oss_dir: ""
    ## Move uploaded file the the directory. It MUST be a different dir from where the files are. If not, the file will be uploaded again and again.
    bak_dir: ""

## watcher configs
watcher:
    ## The time interval(seconds) this program scan the directory
    interval: 30
    ## If the interval between the current time and the file\'s modification time is greater than this value, the write is considered complete.
    write_complete_time: 30
    ## Pattern of the files to watch and upload
    patterns: "files/*.*"
    ## If find patterns recursively
    recursive: false
    ## pool of at most max_workers threads to execute upload/backup tasks. If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5.
    max_workers: ~

Also, the program will first try to load logging configuration file by environment variable ALIYUNOSS2_AUTOUPLOAD_LOG_CONF. If the environment variable not assigned, then try to load configuration file “conf/aliyunoss2-autoupload.log.yml”.

Logging config file is also YAML. Go to https://docs.python.org/3/library/logging.config.html for more information about Python logging config.

---

---

Install

Install by pip

pip install aliyunoss2-autoupload

Install from code

git clone https://github.com/tanbro/aliyunoss2-autoupload.git
cd aliyunoss2-autoupload
path/of/your/python setup.py install

Changelog

0.1

Added

• Reload configuration every time before scanning files.

• –only-once command line option.

Optimized

• test case for main() function

0.1b2

Date:

2018-06-20

Adds

• More detailed documents.

Bug Fixes:

• Remove files in .gitignore, but tracked.

0.1b1

Date:

2018-06-19

Changes

• Support old Python 2.7, Python 3.4

• Config file name extension changed from ".yaml" to ".yml"

• Default config file environment variable name

• Default config file path if no environment variable

Adds

• Some simple test cases

• CircleCI

• Codacy

0.1a3

The first ever-usable version.

Date:

2018-04-18

Changes

• Now Python 3.5+ ONLY, because glob.iglob has no recursive argument and "**" wildcard in lower Python version

Bug Fixes

• Backup directory errors

• OSS exception

CONTRIBUTING

Contributions are welcome!

It’s advised to develop in venv

/path/of/your/python -m venv .venv
source .venv/bin/activate
git clone [e-mail]:tanbro/aliyunoss2-autoupload.git
cd aliyunoss2-autoupload
python setup.py develop

Authors

• liu xue yan <[e-mail]>