Skip to main content

Kafka integration for bluesky.

Project description

https://github.com/bluesky/bluesky-kafka/actions/workflows/tests.yml/badge.svg https://img.shields.io/pypi/v/bluesky-kafka.svg

Kafka integration for bluesky.

  • Free software: 3-clause BSD license

Features

  • BlueskyConsumer

  • MongoConsumer

  • Publisher

  • RemoteDispatcher

Test

Install docker and docker-compose.

Start a Kafka server:

$ cd bluesky-kafka/scripts
$ sudo docker-compose -f bitnami-kafka-docker-compose.yml up

Run tests:

$ cd bluesky-kafka
$ pytest

Optionally increase logging output to the console by specifying a logging level:

$ pytest --log-cli-level=DEBUG

Run a Mongo Consumer Group

Create a conda environment:

$ conda create -n consumers python=3.8
$ conda activate consumers

Install packages:

$ pip install bluesky-kafka supervisor

Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.

$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3
$ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"

Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.

Start the consumer processes:

$ supervisord -c bluesky_kafka/supervisor/supervisord.conf

Monitor the consumer processes:

$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bluesky-kafka-0.10.0.tar.gz (45.0 kB view hashes)

Uploaded Source

Built Distribution

bluesky_kafka-0.10.0-py3-none-any.whl (37.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page