BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Google Cloud BigQuery
Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Credits
Thanks to Bluecore engineering team for this usefull article.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for airflow-bigquerylogger-0.4.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c79a9b592674091cdddb9f85d1495c5116b407f6786905ffe6049fe1b0054625 |
|
MD5 | c8968a77fd74515bce1f87f4fe8483d5 |
|
BLAKE2b-256 | 66c5ab5d64d2221193c8db712fd7b4c919004444df249adba5590531f0e2e119 |
Close
Hashes for airflow_bigquerylogger-0.4.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6384969ee163710b57206029c04632da3b2dc08ab5b7cc09d0586b5a42f39f5e |
|
MD5 | e7434c6a8ebba34ef1535b76fc0f2fa6 |
|
BLAKE2b-256 | 27574d3e3b7c428b84c465bfacfdac9fa7fce51e7add3589942b5ae85154a544 |