Event-driven architectures (EDA) are becoming the backbone of modern applications. By decoupling services and reacting to events rather than polling for state, systems achieve greater scalability, responsiveness, and resilience. However, one challenge remains: ensuring reliable message delivery. Misunderstandings about guarantees like at-least-once, at-most-once, and exactly-once can lead to bugs, data inconsistencies, or duplicate processing.

In this article, we’ll explore Kafka idempotency, the Outbox Pattern, and transactional event publishing, with practical examples.

1. Event-Driven Basics

In an event-driven system:

• Producers generate events when something happens (e.g., a new order is placed).
• Consumers react to those events (e.g., updating inventory, notifying users).
• Event brokers like Apache Kafka handle message delivery.

Challenge: How do we ensure that events are delivered and processed reliably, without duplicates or data loss?

2. Kafka Idempotency

Apache Kafka supports idempotent producers, which ensure that even if a producer retries due to network issues, each message is written exactly once to a partition.

Example (Java Kafka Producer):

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true"); // Enable idempotency

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

ProducerRecord<String, String> record = new ProducerRecord<>("orders", "order-123", "OrderCreated");
producer.send(record);
producer.close();

• enable.idempotence = true ensures that retries don’t result in duplicate events.
• Coupled with Kafka transactions, this allows exactly-once semantics across multiple partitions and topics.

3. The Outbox Pattern

The Outbox Pattern addresses the challenge of atomically saving data and publishing events. Without it, you risk inconsistencies if a database update succeeds but the message broker fails (or vice versa).

This diagram visually represents the flow described in the article, showing how a service writes to an outbox table within a transaction and a separate process then reliably publishes the event to Kafka, ensuring consistency and exactly-once semantics.

How it works:

1. Service writes data and a corresponding event to an outbox table in the same database transaction.
2. A separate process reads from the outbox and publishes events to Kafka.
3. Once published, the outbox entry is marked as processed.

Example (Outbox Table Schema):

CREATE TABLE outbox (
 id BIGINT AUTO_INCREMENT PRIMARY KEY,
 aggregate_id VARCHAR(255),
 event_type VARCHAR(255),
 payload JSON,
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
 processed BOOLEAN DEFAULT FALSE
);

Service Transaction Example (Java + JPA):

@Transactional
public void createOrder(Order order) {
 orderRepository.save(order);

 OutboxEvent event = new OutboxEvent();
 event.setAggregateId(order.getId());
 event.setEventType("OrderCreated");
 event.setPayload(order.toJson());
 outboxRepository.save(event);
}

A Kafka publisher process periodically reads unprocessed events:

List<OutboxEvent> events = outboxRepository.findUnprocessed();
for (OutboxEvent event : events) {
 producer.send(new ProducerRecord<>("orders", event.getAggregateId(), event.getPayload()));
 event.setProcessed(true);
 outboxRepository.save(event);
}

This guarantees that database and event changes stay in sync.

4. Kafka Transactions

Kafka transactions extend idempotency by allowing multiple writes across topics to either succeed or fail as a unit. This is especially useful when a service needs to write to both a database and Kafka.

Example (Java Transactional Producer):

producer.initTransactions();

try {
 producer.beginTransaction();
 producer.send(new ProducerRecord<>("orders", "order-123", "OrderCreated"));
 producer.send(new ProducerRecord<>("notifications", "order-123", "NotifyCustomer"));
 producer.commitTransaction();
} catch (ProducerFencedException | KafkaException e) {
 producer.abortTransaction();
}

• Either all messages are committed or none are.
• When combined with the outbox pattern, this guarantees exactly-once delivery.

5. Real-World Use Cases

1. E-Commerce:
   • Ensure each order triggers inventory updates, payment processing, and notifications exactly once.
   • Outbox pattern prevents missed or duplicated order events.
2. Banking / Finance:
   • Transaction events must be consistent. Kafka idempotency and transactions avoid double charges or missing ledger entries.
3. IoT & Sensor Networks:
   • Sensors generate frequent events. Outbox + Kafka guarantees reliable processing without overloading cloud services.

6. Best Practices

• Enable idempotent producers in Kafka (enable.idempotence = true).
• Use transactions when publishing multiple messages that must succeed together.
• Implement the outbox pattern to avoid distributed transaction issues.
• Monitor for unprocessed outbox events and retries to maintain reliability.
• Design consumer logic to be idempotent as well, in case of duplicates from upstream systems.

7. Writing Idempotent Consumers

So far, we’ve looked at Kafka’s guarantees, the Outbox Pattern, and transactional publishing. But even with those patterns, consumers must be idempotent — able to process the same message more than once without harmful side effects.

Why? Because duplicates can still appear:

• Network retries
• Consumer group rebalances
• Manual replays or catch-up runs
• Producer retries (even with idempotency enabled, depending on the delivery semantics)

Let’s walk through strategies for writing idempotent consumers, from the simplest (database deduplication) to advanced patterns (Kafka transactions, outbox replay).

7.1 Principles of Idempotency

• Assume at-least-once delivery: Kafka delivers messages at-least-once unless you explicitly use transactions with offset commits.
• Side effects must be safe: If your consumer writes to a database, calls an API, or sends an email, you need to ensure duplicates don’t cause harm.
• Detect and skip duplicates: Every processed message should be uniquely identifiable.

7.2 Database Deduplication with Unique Keys

The simplest and most reliable approach is to track processed messages in a table. You use a message ID (often the event key or a UUID inside the payload) as the unique identifier.

Schema Example (Postgres):

CREATE TABLE processed_messages (
 message_id TEXT PRIMARY KEY,
 topic TEXT,
 partition INT,
 offset BIGINT,
 processed_at TIMESTAMP DEFAULT now()
);

Consumer Example (Java + JDBC):

@Transactional
public void process(ConsumerRecord<String, String> record) {
 int inserted = jdbcTemplate.update(
 "INSERT INTO processed_messages(message_id, topic, partition, offset) " +
 "VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING",
 record.key(), record.topic(), record.partition(), record.offset()
 );

 if (inserted == 0) {
 // Duplicate -> skip
 return;
 }

 // Safe to apply side effects (e.g., update order, adjust inventory)
}

7.3 Outbox + Deduplication

When your consumer writes both to a database and to Kafka, use the Outbox Pattern.
The same approach applies on the consumer side: insert the message ID into a deduplication table before performing side effects.

7.4 External API Calls with Idempotency Keys

If your consumer calls external services, ensure those APIs support idempotency.

For example, payment services like Stripe accept an Idempotency-Key header:

POST /charge
Idempotency-Key: order-123

{ "amount": 100, "currency": "USD" }

If the consumer retries the same request, the service returns the same result instead of charging twice.

7.5 Redis-Based Fast Deduplication

For high-throughput, low-latency consumers, you can store processed message IDs in Redis:

String redisKey = "dedup:" + record.key();
String result = jedis.set(redisKey, "1", SetParams.setParams().nx().ex(3600));
if (result == null) {
 // Duplicate -> skip
} else {
 // Process safely
}

Here, NX ensures the key is only set if it doesn’t exist, and EX adds a TTL so memory isn’t exhausted.

7.6 Kafka Transactions for Exactly-Once Processing

If your consumer is only reading from one Kafka topic and writing to another, use Kafka transactions with offset commits:

producer.initTransactions();
producer.beginTransaction();

// Process input record -> produce output
producer.send(new ProducerRecord<>("output-topic", key, value));

// Atomically commit both messages and offsets
producer.sendOffsetsToTransaction(offsets, consumerGroupId);
producer.commitTransaction();

This ensures:

• Input offset is committed only if the output is written.
• Replays will not reprocess already committed records.

7.7 Best Practices for Consumers

• Always generate a unique message ID (UUID, orderId, transactionId).
• Use database upserts or unique keys for safe deduplication.
• For external APIs, rely on idempotency keys.
• Consider Redis or state stores for fast deduplication with TTL.
• For Kafka-only pipelines, leverage exactly-once transactions.
• Monitor dedup hit rates and alert if duplicates spike.

To wrap up, here’s a quick reference table of best practices for designing idempotent consumers across different scenarios:

7.8 Checklist for Idempotent Consumers

✅ Deduplication mechanism in place (DB, Redis, or Kafka transactions)
✅ Unique identifiers in every message
✅ Manual offset commit only after side effects succeed
✅ DLQ for poison messages
✅ Monitoring for retries and duplicates

Final Thoughts

Event-driven systems are powerful, but ensuring reliability is critical. Combining Kafka’s idempotency, transactions, and the Outbox Pattern allows developers to achieve exactly-once guarantees while keeping services decoupled. These patterns are the foundation for building scalable, consistent, and resilient systems in real-world architectures.

Useful Links:

• Apache Kafka Transactions
• Outbox Pattern Guide
• Kafka Idempotent Producers