Apache Kafka was born out of a problem. 😉 LinkedIn engineers => faced difficulties in tracking website metrics, activity streams and other operational data. A team of engineers => led by Jay Kreps, Neha Narkhede and Jun Rao started developing a distributed publish-subscribe messaging system that could handle high-throughput, low-latency data streams. This system eventually became Apache Kafka. It was open sourced in early 2011. The name 'Kafka' was chosen by Jay Kreps. He named the system after the famous author 'Franz Kafka'. 😊 Kreps was an admirer of Franz Kafka's writing and found the name fitting for a system that dealt with the flow of information. It's written in Java and Scala. Later they founded => 'Confluent' (a company) in 2014 to provide commercial support and additional tools for Kafka users. 𝐋𝐞𝐭'𝐬 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 𝐭𝐡𝐞 𝐛𝐚𝐬𝐢𝐜 𝐟𝐥𝐨𝐰. [1.] Producer sends a message ◾ An application acts as a producer, creating a message with data (payload) and optional key. ◾ The producer connects to a broker in the Kafka cluster and identifies the target topic. ◾ Kafka uses a partitioner to determine which partition within the topic should receive the message. This ensures load balancing and parallel processing. ◾ The message is delivered to the leader replica of the chosen partition. [2.] Message storage and replication ◾ The leader replica appends the message to its log segment. ◾ The message receives a unique offset, serving as its position within the log. ◾ The leader replicates the message to follower replicas for fault tolerance. [3.] Consumer fetches messages ◾ An application acts as a consumer, joining a consumer group. ◾ Consumers within the same group share offsets and coordinate consumption. ◾ Each consumer fetches messages from its assigned partitions based on its committed offset. ◾ The consumer receives batches of messages and processes them. [4.] Acknowledging consumption ◾ Once processing is complete, the consumer commits its new offset. ◾ This tells Kafka which messages have been successfully consumed. ◾ Kafka tracks committed offsets for each consumer in the group. [*.] Flow continues ◾ Producers continue sending messages and consumers keep fetching and processing them based on their latest offsets. ◾ This cycle ensures ordered delivery and reliable consumption even with failures or restarts. Remember, 👉 Message flow is asynchronous. Producers don't wait for consumers to process messages. 👉 Consumers can lag behind producers if processing is slow. 👉 Kafka offers mechanisms for handling failures and ensuring at-least-once or exactly-once delivery semantics. Topics => Partitions =>Log Segments (Data is actually stored in log segments) Follow @techNmak

@lowkeyloud_ Merged a biggg asss PR with 50+ files. Got it to PROD. Only to realize requirements got changed by PM 💀💀