BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Google Cloud BigQuery
Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Credits
Thanks to Bluecore engineering team for this usefull article.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for airflow-bigquerylogger-0.4.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | da610427c7eb5ec806d5b92285abf349579fe8642b75a2962a91c9702b457c0a |
|
MD5 | ce24124969ec8ae1162c473f94fa6ba9 |
|
BLAKE2b-256 | 91b52794f5600782057a0779c4d5e4072e8df0f597174aa28b025e13cf3b28fc |
Close
Hashes for airflow_bigquerylogger-0.4.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 143876f7e1c883c467f85a69e97f9b934ce1e4ddf8b8bf6f031a45c0bc483614 |
|
MD5 | 202d318314bf92fc8d5d4a336bb76d7b |
|
BLAKE2b-256 | 6de073017a3977bc338c335090af3c2a91b9f15500fdc961e396de0c138dddf1 |