The rise of Microservice Architecture may help companies build highly scalable products while opening up new points of failure we never had before. This architecture model dictates that systems should be divided into small but independent services, and in the end, you might have tens or hundreds of services. Although each service is disconnected, you might still need integration in the form of data passing between each service. Integrating this many services is a nightmare. If you had ten services, you might end up with up to 45 integrations.
Kafka provides a solution for this particular problem where this tech product decouples each service with the Message Queue model. Kafka is a broker between services, meaning the service sends messages to Kafka without actually know which service is the recipient. With this flow, a service now does not need a response from the recipient, and this led Kafka to enable services to process data in real-time.
This article will introduce you to Kafka components to provide you a basic understanding of what Kafka is so that you can see how simple the system is and help you into the rabbit hole of learning this fantastic tech product.
One of the main components, when we are talking about Kafka, is Kafka Topics. This component labels specific data streams into a group based on the topic this data addresses, hence Kafka Topics. You can have as many Kafka Topics you want but make sure the labels are identifiable.
Data in Kafka Topics are split into several partitions. Each data stored within a partition have incremental identifiers, called offset. Messages stored in Kafka are immutable. Kafka replication.
You can set how many partitions you want but consider the cost of a large number of partitions, such as increased latency, memory consumption, and the number of unavailabilities.
Kafka Broker is a server containing some Kafka Topic partitions. Since Kafka is distributed, the data within Kafka will be spread across multiple servers. Kafka maintains this distribution, so you don't have to implement this feature. Then, how do you access all the information in Kafka if it is distributed? In Kafka, every broker is a bootstrap server, meaning if you had access to one broker, you had access to the whole cluster of Kafka Broker.
In Kafka, a service that produces and sends information is called Producer. Producer no longer sends messages to another service directly within a Kafka-enabled system but with the help of Kafka brokers as the middleman. The Producer sends data along with the messages key to one of the Kafka brokers, and the Kafka broker will pass that information to the recipient.
A service that consumes information from Producer is called Consumer. A consumer reads data from the topic identified by its name. This kind of service will read data in order. In case of consumer failure, it knows where to read from because, in brokers, we had some variable storage called offset.
Kafka fits perfectly in the Microservice Architecture. Its role as a messaging bus helps to integrate and decouple many services. This fantastic product is an essential puzzle for you to migrate your system from the old Monolith structure.
This article is the embodiment of my learning progress in Kampus Merdeka x Gojek internship program. Feel free to follow me to receive updates on my learning journey!