Hi there👋

This is mingyan. Welcome to my blog. I am a system lover and will document what I learn in my blog.

Kafka Deep Dive: How Fetch Responses Are Built and Zero-Copy Works (Part 4)

This is part 4 of the Kafka series. Part 1: Introduction to Kafka TCP Fragmentation Part 2: Introduction to Reactor Pattern and Kafka’s Reactor Implementation Part 3: An introduction to Kafka’s High‑Performance Binary RPC Protocol (Part 3) The source code referenced in this article uses Kafka’s trunk branch. I’ve pushed it to my personal repo for version alignment with this article: https://github.com/cyrilwongmy/kafka-src-reading In the previous article, we explored the path of a Kafka request from the network layer to the KafkaApis class. However, we didn’t cover much about how Kafka processes and returns a response. This article focuses specifically on how Kafka handles a Fetch request, constructs the response, and sends log data using zero-copy techniques. ...

An introduction to Kafka’s High‑Performance Binary RPC Protocol (Part 3)

This is part 3 of the Kafka series. Part 1: Introduction to Kafka TCP Fragmentation Part 2: Introduction to Reactor Pattern and Kafka’s Reactor Implementation The source code referenced in this article uses Kafka’s trunk branch. I’ve pushed it to my personal repo for version alignment with this article: https://github.com/cyrilwongmy/kafka-src-reading In this part, we will briefly introduce the Kafka’s message protocol. Kafka’s server and client communicate through a custom message protocol. By understanding how Kafka’s RPC protocol is implemented, we can apply similar principles when designing our own protocols. ...

Kafka High-Throughput Architecture: Demystifying the Reactor Pattern (Part 2)

This is part 2 of the Kafka series. In this part, we will dive into the Reactor pattern and how Kafka implements it. In the next following series, we will dive into each component of the Reactor pattern. The source code referenced in this article uses Kafka’s trunk branch. I’ve pushed it to my personal repo for version alignment with this article: https://github.com/cyrilwongmy/kafka-src-reading What is the Reactor Pattern? (Readers familiar with the Reactor pattern can skip to the Kafka’s Reactor Implementation section) ...

Kafka’s TCP Unpacking: A Production-Grade Code Walkthrough (Part 1)

This is part 1 of the Kafka series. The Kafka series is a series of blog posts that will introduce the design and implementation details of Kafka. The source code referenced in this article uses Kafka’s trunk branch. I’ve pushed it to my personal repo for version alignment with this article: https://github.com/cyrilwongmy/kafka-src-reading TCP and Packet Fragmentation Why Does TCP Require Packet Fragmentation? Kafka uses TCP as its underlying transport layer protocol. TCP operates as a continuous byte stream without inherent message boundaries. Therefore, any application using TCP must implement its own mechanism to delineate message boundaries. ...