BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Credits
Thanks to https://medium.com/bluecore-engineering/kubernetes-pod-logging-in-the-airflow-ui-ed9ca6f37e9d
Unable to delete logs from BigQuery {} because:\n\n *** Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for airflow-bigquerylogger-0.4.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2e67191a09ecf2f746a9c9cf8e98ece506004cfccc81e5cddf492ed82b2be48 |
|
MD5 | 2f1d589775c0de36731e7579f37c551c |
|
BLAKE2b-256 | 28497ada250eb9f190d4d623ad88fd038f0e9232f92c7ed991b2f585a9521282 |
Close
Hashes for airflow_bigquerylogger-0.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c31e2581c06fcddc0dc0c824a9fcc1d6be6172ec1bb57b3731613703fbdf2735 |
|
MD5 | 586399e71af3dcbf71edbb13903b9570 |
|
BLAKE2b-256 | bb8ee93d8e1c023ed5f010da4d0433ae4ef401bff0b3e74c809e8e5ecc801566 |