Consume an AWS Kinesis Data Stream to look over the records from a terminal
Project description
aws-kinesis-consumer
Consume an AWS Kinesis Data Stream to look over the records from a terminal.
Demo
$ aws-kinesis-consumer --stream-name MyStream
<shard_id=shardId-000000000000, records=1>
Record-001
<shard_id=shardId-000000000001, records=2>
Record-002
Record-003
Usage
Pre-requirement
Connect to AWS and set the default AWS environment variables .
export AWS_DEFAULT_REGION=eu-central-1
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Alternatively, aws configure can help to set the environment variables.
Use with Python
# install
pip install aws-kinesis-consumer
# consume a stream
aws-kinesis-consumer --stream-name MyStream
Use with Docker
docker run \
-e AWS_DEFAULT_REGION=eu-central-1 \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
thinow/aws-kinesis-consumer --stream-name MyStream
Arguments
Argument | Default | Description |
---|---|---|
--stream-name (required) |
Name of the AWS Kinesis Stream. | |
--endpoint |
Custom AWS endpoint url to communicate with the AWS API. Could be used in order to specify a region (e.g. https://kinesis.us-east-1.amazonaws.com/ ). |
|
--iterator-type |
latest |
Defines how to start consuming records from the stream. Use latest to consume the new records only. Or use trim-horizon to consume all the records already existing in the stream. |
--help |
Prints the help message. | |
--version |
Prints the version of the binary. |
FAQ
What is the motivation ? What is the issue with AWS CLI ?
The AWS CLI is able to fetch records from Kinesis, but the users need to list the shards, to generate iterator tokens, use subsequent tokens, delay operations, and so on.
aws-kinesis-consumers
in contrary is able to get records by using the stream name, and only the stream name.
Therefore there is no need for an extra script.
How to consume a stream hosted in different regions ?
The environment variable AWS_DEFAULT_REGION
can be used to specify any AWS region.
AWS_DEFAULT_REGION=eu-central-1 aws-kinesis-consumer --stream-name MyGermanStream
How to filter the records ?
aws-kinesis-consumer
can be piped with other command such as grep,
or even jq to filter json records.
# all the records
$ aws-kinesis-consumer --stream-name MyStream
{"name":"foo", "status":"ok"}
{"name":"bar", "status":"pending"}
{"name":"baz", "status":"error"}
# records containing the text "ba" (e.g. "bar" and "baz", but not "foo")
$ aws-kinesis-consumer --stream-name MyStream | grep "ba"
{"name":"bar", "status":"pending"}
{"name":"baz", "status":"error"}
# records where the json property "status" has the value "error"
$ aws-kinesis-consumer --stream-name MyStream | jq 'select(.status == "error")'
{"name":"baz", "status":"error"}
What are the required AWS permissions ?
aws-kinesis-consumer
requires the following AWS permissions :
The following policy is an example which can be applied to an AWS user or an AWS role :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:ListShards",
"kinesis:GetShardIterator",
"kinesis:GetRecords"
],
"Resource": [
"arn:aws:kinesis:REGION:ACCOUNT-ID:stream/STREAM-NAME"
]
}
]
}
Changelog
See all the changes per release.
Special thanks
- Thanks to the contributors of the kinesalite project which make test and development of this project extremely easy and reliable!
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aws-kinesis-consumer-1.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 797dd9dcc63996e138e1135726e2104ae4e578344ef48c431de03cf80b991a37 |
|
MD5 | ad0cdef944bc8827912212e071aa59d2 |
|
BLAKE2b-256 | 2f3a87989e339bb494d0dcc3e80cf62dd184779cdf40b9f7c24494226df283af |
Hashes for aws_kinesis_consumer-1.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44024491c9b110113547362c96dc27c1b7065a8b30c12049f069a78e1e13d2e1 |
|
MD5 | d9b5ee9ef0139b0b7688bbe08515de35 |
|
BLAKE2b-256 | 288414477bde31e4f9f1ac9ed0dc63942e3bebc537346163019a9cab2911437d |