Skip to main content
Advertisement

13.3 Apache Kafka Event Publishing/Subscribing for High-Volume Traffic

Kafka is fundamentally different from traditional message queues like RabbitMQ. Messages are permanently retained like a log even after being consumed, and it's designed as a distributed event streaming platform capable of processing tens of millions of events simultaneously.

1. Core Kafka Concepts​

  • Topic: Event category (e.g., order-events, user-signup)
  • Partition: Topic subdivisions. More partitions = higher parallel throughput.
  • Consumer Group: Consumers in the same group divide partitions among themselves for parallel consumption.
  • Offset: A message's position within a partition. Consumers track how far they've read via offsets.

2. Spring Kafka Producer Setup and Publishing​

implementation 'org.springframework.kafka:spring-kafka'
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
@Service
@RequiredArgsConstructor
public class OrderEventProducer {

    private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;

    public void publishOrderEvent(OrderCreatedEvent event) {
        // Using orderId as partition key ensures ordering for the same order's events
        kafkaTemplate.send("order-events", event.getOrderId().toString(), event);
    }
}

3. Consumer and Consumer Groups​

spring:
  kafka:
    consumer:
      group-id: notification-service   # Consumer group ID
      auto-offset-reset: earliest       # earliest=from beginning, latest=from now
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "*"
@Service
@Slf4j
public class NotificationConsumer {

    @KafkaListener(topics = "order-events", groupId = "notification-service")
    public void handleOrderEvent(OrderCreatedEvent event) {
        log.info("Order event received (notification service): {}", event.getOrderId());
        // Send email / SMS / push notification
    }
}
tip

RabbitMQ vs Kafka Decision Guide

  • RabbitMQ: Best for fast task queues or single-service logic where messages disappear after consumption
  • Kafka: Best for MSA event streaming where events need replay and multiple consumer groups independently consume the same events
Advertisement
Advertisement