ETL flow framework based on Yaml configs in Python
Project description
ETL flow framework based on Yaml configs in Python
A light framework for creating data streams. Setting up streams through configuration in the Yaml file. There is a schedule, task pools, concurrency limitation. Works quickly, does not require a lot of resources. Runs on Windows and Linux. Flow run in parallel via threading library. Internally SQLite Database.
At the moment there are connectors to sources
- Yandex Metrika Management API
- Yandex Metrika Stats API
- Yandex Metrika Logs API
- Yandex Direct API
- Yandex Direct Report API
Storages
- Save to file
- Clickhouse
Documentation
Requirements
- python >=3.9
- virtual environment
Settings
It is highly recommended to install in a virtual environment.
Flowmaster needs a home, '{HOME}/FlowMaster' is the default,
but you can lay foundation somewhere else if you prefer
(optional)
For Windows
setx FLOWMASTER_HOME "{YOUR_PATH}"
For Linux
export FLOWMASTER_HOME={YOUR_PATH}
Installing
pip install flowmaster==0.3.1
Run
flowmaster run
args
flowmaster run --help
Support
Author
Pavel Maksimov
My contacts Telegram, Facebook
Удачи тебе, друг! Поставь звездочку ;)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.