BigQuery logger handler for Airflow
Project description
BigQuery logger handler for Airflow
Installation
pip install airflow-bigquerylogger
Configuration
AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50
Credits
Thanks to https://medium.com/bluecore-engineering/kubernetes-pod-logging-in-the-airflow-ui-ed9ca6f37e9d
Unable to delete logs from BigQuery {} because:\n\n *** Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file airflow-bigquerylogger-0.3.0.tar.gz.
File metadata
- Download URL: airflow-bigquerylogger-0.3.0.tar.gz
- Upload date:
- Size: 3.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
02ef2d7c6d6dcb65692b4d7a2f7e1349824dfdffe1b5f58b0bfed2624abab92d
|
|
| MD5 |
ec2471a8fb1e2b5a20f116e7943baa15
|
|
| BLAKE2b-256 |
4d3cb1bc92173d3e3f002108a332fe0736e55a1718b9ee9aafe525576e299ecf
|
File details
Details for the file airflow_bigquerylogger-0.3.0-py3-none-any.whl.
File metadata
- Download URL: airflow_bigquerylogger-0.3.0-py3-none-any.whl
- Upload date:
- Size: 5.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a16a6cc316461ea39b1da55e5ab90ba601107bfbe3d687019781ccd66416ae0b
|
|
| MD5 |
85fbfdf939a671ac0511ac8c7948f465
|
|
| BLAKE2b-256 |
95497c8c98641ed4a67084343fd332c301c3071c037148ba7a9492ca508ad341
|
