BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Google Cloud BigQuery
Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Credits
Thanks to Bluecore engineering team for this usefull article.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for airflow-bigquerylogger-0.4.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ad516d6e053d21ddf15d7cd6b41c6125741b9099d28441c318b1824000b90069 |
|
MD5 | 96922fef04772c87c47c3fc3e3fb4e81 |
|
BLAKE2b-256 | 279f2ab5aa35933290f1ed06cf8578728e7b67150f07f5f2944310f3456477c0 |
Close
Hashes for airflow_bigquerylogger-0.4.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4f77f28033e4239d38b0ddf7aa5c59be3a6e57e9e0aa410dde49ecd0a740108 |
|
MD5 | e91188cc3657d38de64879671c4e56eb |
|
BLAKE2b-256 | 5b3dc1a4f7bdf2ff4697d683924945767277799576b6cb418e20e92d52355491 |