VOOZH about

URL: https://repost.aws/questions/QUYGz9MZ3mTzehJsRtcEAvKw/maintaining-stable-shard-to-consumer-mapping-in-custom-node-js-kinesis-consumer

⇱ Maintaining stable shard-to-consumer mapping in custom Node.js Kinesis consumer | AWS re:Post

Skip to content

Maintaining stable shard-to-consumer mapping in custom Node.js Kinesis consumer

0

My previous post: We're building a custom Node.js consumer for Kinesis Data Streams and intentionally avoiding KCL due to its Java dependency and additional infrastructure requirements (DynamoDB). Our solution uses independent consumers that calculate their shard assignments using a round-robin distribution based on sorted shard IDs.

Current Implementation:

  • Sort all shards by shard ID lexicographically
  • Each consumer knows total consumer count and its instance ID
  • Consumer claims shards where (index % totalConsumers == instanceId)

`public getCurrentInstanceShardIds(allShards) { // Sort by shard ID for consistent ordering const sortedShards = _.sortBy(allShards, 'shardId');

// Get shards for this instance using cyclic distribution
return sortedShards.filter((_, index) => index % this.kinesisPollerCount === this.instanceId);

}`

This works well if we are sure about the shard naming convention before and after resharding. Our concerns:

Questions:

  1. Is there a documented convention/pattern for how new shard IDs are generated during resharding? This is crucial because:
    • Our distribution relies on lexicographic sorting of shard IDs
    • If new shards get IDs that sort before existing ones, it would shift our entire distribution mapping
    • Example: If shard "000000000003" splits and new shards get IDs like "000000000029" and "000000000030", the distribution remains stable. But if they get IDs that sort earlier, all subsequent assignments would shift
  2. What's the recommended approach for maintaining stable shard-to-consumer mapping during resharding when using a custom Node.js consumer?
  3. Are there any considerations around:
    • Parent-child shard relationship during transitions
    • Handling the period where both parent and child shards are active
    • Ensuring no data loss during redistribution

We're specifically looking for solutions that don't require external coordination services while maintaining predictable shard assignments during resharding operations.

Environment:

  • Node.js
  • "@aws-sdk/client-kinesis": "^3.74.0"
  • No external coordination (trying to keep infrastructure minimal)
1 Answer
  • Newest
  • Most votes
  • Most comments
Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.
0
This recommendation was generated using AWS Generative AI capabilities. You are responsible for evaluating the recommendation in your specific context and implementing appropriate oversight and safeguards. Learn more
This answer has not been reviewed by an expert

To address your concerns about maintaining stable shard-to-consumer mapping in a custom Node.js Kinesis consumer, especially during resharding operations, here are some insights and recommendations:

  1. Shard ID Convention: While there isn't a publicly documented convention for how new shard IDs are generated during resharding, Kinesis typically assigns new shard IDs that sort after the existing ones. This means your current lexicographic sorting approach should generally maintain stability. However, it's not guaranteed, and relying solely on this behavior could potentially lead to issues.

  2. Maintaining Stable Shard-to-Consumer Mapping: To achieve a more robust shard-to-consumer mapping, consider implementing a consistent hashing algorithm instead of relying on lexicographic sorting. Consistent hashing can help minimize the number of shards that need to be reassigned when resharding occurs. This approach would involve:

    • Creating a hash ring with a large number of virtual nodes
    • Mapping both consumers and shards to points on this ring
    • Assigning each shard to the nearest consumer in a clockwise direction on the ring

This method provides better stability during resharding events, as it limits the impact of adding or removing shards.

  1. Considerations for Resharding:

    • Parent-Child Relationship: When a shard splits or merges, maintain awareness of the parent-child relationships. Continue processing the parent shard until it's closed, then transition to the child shards.

    • Transition Period: During resharding, both parent and child shards will be active for a period. Your consumer should be able to handle this scenario by processing records from both until the parent is fully processed.

    • Data Loss Prevention: To ensure no data loss, implement checkpointing to track your progress within each shard. Store these checkpoints in a persistent store (like a database or even a local file if you're keeping things minimal).

Implementation Suggestions:

  1. Implement a consistent hashing mechanism for shard distribution.
  2. Use the DescribeStream API to periodically check for changes in the shard structure.
  3. Implement a simple checkpointing mechanism to track progress in each shard.
  4. When detecting new shards, gradually transition processing from parent to child shards.

While this approach doesn't require external coordination services, it does increase the complexity of your consumer implementation. However, it provides a more robust solution for handling resharding events and maintaining stable shard assignments.

Remember to thoroughly test your implementation, especially around resharding scenarios, to ensure it behaves as expected in all situations.

Sources
Best practices for distributing Kinesis shards across multiple pollers - handling resharding and balanced CPU load | AWS re:Post
Process Amazon Kinesis Data Streams records with Lambda - AWS Lambda
Introducing the new Amazon Kinesis source connector for Apache Flink | AWS Big Data Blog

answered a year ago

  • Samriddhi
    a year ago

    can someone from AWS team confirm on shard naming convention and behavior on resharding?

Relevant content