BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Google Cloud BigQuery
Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Credits
Thanks to Bluecore engineering team for this usefull article.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for airflow-bigquerylogger-0.4.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 475433c3bcc95374508f5f5a798a8ea1062d547750e15813251ddc13b2549fa6 |
|
MD5 | 8015313de8ee544fa987f949bd2317b7 |
|
BLAKE2b-256 | f19084722c7b6e55f8816aad646cf4d98d5bf59578576100d278dc0335821b71 |
Close
Hashes for airflow_bigquerylogger-0.4.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f94a3c09503c0d20a3d7ee5b50c8866699ed5837a867d0e207bab227f94708ea |
|
MD5 | b724bdf8ec784750c5b43b6d903b3c10 |
|
BLAKE2b-256 | 19caa4515005d8dd7f39f84666bd17317f396b3827f2d910497982ea028504f2 |