BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Google Cloud BigQuery
Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Credits
Thanks to Bluecore engineering team for this usefull article.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for airflow-bigquerylogger-0.4.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bcd66d096045af05be1626fb81faf6df343d19d84883e64f34b289507498837 |
|
MD5 | d750fd3fe9373131c0a2fecad63ae7e0 |
|
BLAKE2b-256 | 71e8448b3144d9f1955324c22f8d8a20fc52690f02805d64731549c588d608d2 |
Close
Hashes for airflow_bigquerylogger-0.4.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8dda82b74e38dd46cfeeee6b08c4089cac1d0e64f433367cf1054991006f22d0 |
|
MD5 | 22b6274f367ece59d503384ff30556fb |
|
BLAKE2b-256 | 6a409713d044875b14ea9831d4f2e314fd96d39ac7460f2017c41820502b0a8c |