CloudWiki
Resource

Kinesis

Amazon Web Services
Analytics
Amazon Kinesis is a real-time data streaming service designed to process large scale data streams at any scale.
Terraform Name
terraform
aws_kinesis_stream
Kinesis
attributes:
  • name - (Required) A name to identify the stream. This is unique to the AWS account and region the Stream is created in.
  • shard_count – (Optional) The number of shards that the stream will use. If the stream_mode is PROVISIONED, this field is required. Amazon has guidelines for specifying the Stream size that should be referenced when creating a Kinesis stream. See Amazon Kinesis Streams for more.
  • retention_period - (Optional) Length of time data records are accessible after they are added to the stream. The maximum value of a stream's retention period is 8760 hours. Minimum value is 24. Default is 24.
  • shard_level_metrics - (Optional) A list of shard-level CloudWatch metrics which can be enabled for the stream. See Monitoring with CloudWatch for more. Note that the value ALL should not be used; instead you should provide an explicit list of metrics you wish to enable.
  • enforce_consumer_deletion - (Optional) A boolean that indicates all registered consumers should be deregistered from the stream so that the stream can be destroyed without error. The default value is false.
  • encryption_type - (Optional) The encryption type to use. The only acceptable values are NONE or KMS. The default value is NONE.
  • kms_key_id - (Optional) The GUID for the customer-managed KMS key to use for encryption. You can also use a Kinesis-owned master key by specifying the alias alias/aws/kinesis.
  • stream_mode_details - (Optional) Indicates the capacity mode of the data stream. Detailed below.
  • tags - (Optional) A map of tags to assign to the resource. If configured with a provider default_tags configuration block present, tags with matching keys will overwrite those defined at the provider-level.

stream_mode_details Configuration Block

  • stream_mode - (Required) Specifies the capacity mode of the stream. Must be either PROVISIONED or ON_DEMAND.

Associating resources with a
Kinesis
Resources do not "belong" to a
Kinesis
Rather, one or more Security Groups are associated to a resource.
Create
Kinesis
via Terraform:
The following HCL creates a Kinesis stream resource
Syntax:

resource "aws_kinesis_stream" "test_stream" {
 name             = "terraform-kinesis-test"
 shard_count      = 1
 retention_period = 48

 shard_level_metrics = [
   "IncomingBytes",
   "OutgoingBytes",
 ]

 stream_mode_details {
   stream_mode = "PROVISIONED"
 }

 tags = {
   Environment = "test"
 }
}

Create
Kinesis
via CLI:
Parametres:

create-stream
--stream-name <value>
[--shard-count <value>]
[--stream-mode-details <value>]
[--cli-input-json | --cli-input-yaml]
[--generate-cli-skeleton <value>]
[--debug]
[--endpoint-url <value>]
[--no-verify-ssl]
[--no-paginate]
[--output <value>]
[--query <value>]
[--profile <value>]
[--region <value>]
[--version <value>]
[--color <value>]
[--no-sign-request]
[--ca-bundle <value>]
[--cli-read-timeout <value>]
[--cli-connect-timeout <value>]
[--cli-binary-format <value>]
[--no-cli-pager]
[--cli-auto-prompt]
[--no-cli-auto-prompt]

Example:

aws kinesis create-stream \
   --stream-name samplestream \
   --shard-count 3

aws cost
Costs
The cost of using Kinesis depends on the number of reads, writes, and data transfer for the Kinesis data streams, as well as the number of Kinesis Video Streams and Kinesis Firehose delivery streams. Kinesis charges a fee for each shard hour and for each MB of data transferred, as well as for any additional data retention.
Direct Cost

OnDemand-StreamHour

Indirect Cost
No items found.
Best Practices for
Kinesis

Categorized by Availability, Security & Compliance and Cost

Low
Access allowed from VPN
No items found.
Low
Auto Scaling Group not in use
No items found.
Medium
Connections towards DynamoDB should be via VPC endpoints
No items found.
Medium
Container in CrashLoopBackOff state
No items found.
Low
EC2 with GPU capabilities
No items found.
Medium
EC2 with high privileged policies
No items found.
Medium
ECS cluster delete alarm
No items found.
Critical
ECS task with Admin access (*:*)
Medium
ECS task with high privileged policies
No items found.
Critical
EKS cluster delete alarm
No items found.
Medium
ElastiCache cluster delete alarm
No items found.
Medium
Ensure Container liveness probe is configured
No items found.
Medium
Ensure ECS task definition has memory limit
No items found.
Critical
Ensure EMR cluster master nodes are not publicly accessible
No items found.
More from
Amazon Web Services