airbyte-serverless 0.15

Airbyte made easy (no UI, no database, no cluster)

Airbyte made simple

---

🔍️ What is AirbyteServerless?

AirbyteServerless is a simple tool to manage Airbyte connectors, run them locally or deploy them in serverless mode.

💡 Why AirbyteServerless?

Airbyte is a must-have in your data-stack with its catalog of open-source connectors to move your data from any source to your data-warehouse.

To manage these connectors, Airbyte offers Airbyte-Open-Source-Platform which includes a server, workers, database, UI, orchestrator, connectors, secret manager, logs manager, etc.

AirbyteServerless aims at offering a lightweight alternative to Airbyte-Open-Source-Platform to simplify connectors management.

📝 Comparing Airbyte-Open-Source-Platform & AirbyteServerless

Airbyte-Open-Source-Platform	AirbyteServerless
Has a UI	Has NO UI Connections configurations are managed by documented yaml files
Has a database	Has NO database - Configurations files are versioned in git - The destination stores the state (the checkpoint of where sync stops) and logs which can then be visualized with your preferred BI tool
Has a transform layer Airbyte loads your data in a raw format but then enables you to perform basic transform such as replace, upsert, schema normalization	Has NO transform layer - Data is appended in your destination in raw format. - airbyte_serverless is dedicated to do one thing and do it well: Extract-Load.
NOT Serverless - Can be deployed on a VM or Kubernetes Cluster. - The platform is made of tens of dependent containers that you CANNOT deploy with serverless	Serverless - An Airbyte source docker image is upgraded with a destination connector - The upgraded docker image can then be deployed as an isolated Cloud Run Job (or Cloud Run Service) - Cloud Run is natively monitored with metrics, dashboards, logs, error reporting, alerting, etc - It can be scheduled or triggered by events
Is scalable with conditions Scalable if deployed on autoscaled Kubernetes Cluster and if you are skilled enough. 👉 Check that you are skilled enough with Kubernetes by watching this video 😁.	Is scalable Each connector is deployed independently of each other. You can have as many as you want.

💥 Getting Started with abs CLI

abs is the CLI (command-line-interface) of AirbyteServerless which facilitates connectors management.

Install abs 🛠️

pip install airbyte-serverless

Create your first Connection 👨‍💻

abs create my_first_connection --source="airbyte/source-faker:0.1.4" --destination="bigquery:my_project.my_dataset"

1. Docker is required. Make sure you have it installed.
2. source param can be any Public Docker Airbyte Source (here is the list). We recomend that you use faker source to get started.
3. destination param must be one of the following:
   • print
   • bigquery:my_project.my_dataset with my_project a GCP project where you can run BigQuery queries and my_dataset a BigQuery dataset where you have dataEditor permission.
   • contributions are welcome to offer more destinations 🤗
4. The command will create a configuration file ./connections/my_first_connection.yaml with initialized configuration.
5. Update this configuration file to suit your needs.

Run it! ⚡

abs run my_first_connection

1. The runcommmand will only work if you have correctly edited ./connections/my_first_connection.yaml configuration file.
2. If you chose bigquery destination, you must have gcloud installed on your machine with default credentials initialized with the command gcloud auth application-default login
3. Data is always appended at destination (not replaced nor upserted). It will be in raw format.
4. If the connector supports incremental extract (extract only new or recently modified data) then this mode is chosen.

Select only some streams 🧛🏼

You may not want to copy all the data that the source can get. To see all available streams run:

abs list-streams my_first_connection

Run extract-load for only stream1 and stream2 with:

abs run my_first_connection --streams="stream1,stream2"

If you want to persist the choice of these streams for all future extract-loads, run:

abs set-config my_first_connection --streams="stream1,stream2"

Get help 📙

$ abs --help
Usage: abs [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  deploy  Deploy BIGFUNCTION
  doc     Generate, serve and publish documentation
  test    Test BIGFUNCTION

Deploy 🚀

...

👋 Contribute

Any contribution is more than welcome 🤗!

• Add a ⭐ on the repo to show your support
• Raise an issue to raise a bug or suggest improvements
• Open a PR! Below are some suggestions of work to be done:
  • improve secrets management
  • implement a CLI
  • manage configurations as yaml files
  • implement the get_logs method of BigQueryDestination
  • add a new destination connector (Cloud Storage?)
  • add more serverless deployment examples.
  • implement optional post-processing (replace, upsert data at destination instead of append?)

🏆 Credits

• Big kudos to Airbyte for all the hard work on connectors!
• The generation of the sample connector configuration in yaml is heavily inspired from the code of octavia CLI developed by airbyte.