This is part 4 of the Kafka series.

The source code referenced in this article uses Kafka’s trunk branch. I’ve pushed it to my personal repo for version alignment with this article:

https://github.com/cyrilwongmy/kafka-src-reading

In the previous article, we explored the path of a Kafka request from the network layer to the KafkaApis class. However, we didn’t cover much about how Kafka processes and returns a response. This article focuses specifically on how Kafka handles a Fetch request, constructs the response, and sends log data using zero-copy techniques.

If you’re not familiar zero-copy, there is an introductory video you can have a look.

1. From Fetch Request to FileRecords

Let’s jump straight into the implementation of KafkaApis.handleFetchRequest, which eventually calls ReplicaManager.fetchMessages. While the low-level file lookup logic can be skipped, it’s important to understand that what fetchMessages returns is primarily:

  • A FileRecords object (a wrapper over JDK’s FileChannel)
  • Some metadata

Notably, the log data is not immediately loaded into memory. Instead, Kafka records the file offset range (start and end) that needs to be read later.

The key question now is: how does Kafka serialize and send this data over the network efficiently?

2. Response Construction and Data Wrapping

Inside handleFetchRequest, Kafka defines a callback processResponseCallback which handles the response construction after the fetch data is ready:

replicaManager.fetchMessages(
  params = params,
  fetchInfos = interesting,
  quota = replicationQuota(fetchRequest),
  responseCallback = processResponseCallback,
)

Response Callback Logic

The callback processResponseCallback performs the following:

  • Constructs a map from TopicIdPartition to FetchResponseData.PartitionData
  • For each partition, Kafka builds a PartitionData object and sets its fields (e.g., high watermark, log start offset, records)
  • The records field here is the FileRecords object — a file reference, not in-memory data
.setRecords(data.records)

Final Response

Kafka then calls:

fetchContext.updateAndGenerateResponseData(partitions, nodeEndpoints.values.toSeq.asJava)

This creates the final FetchResponse and sends it using:

requestChannel.sendResponse(request, fetchResponse, None)

3. Serialization and Zero-Copy Packaging

Kafka serializes the response using this method chain:

Request.buildResponseSend()
→ RequestContext.buildResponseSend()
→ AbstractResponse.toSend()
→ SendBuilder.buildSend()

Within SendBuilder.buildSend(), Kafka performs:

  1. Header and payload size calculation
  2. Writes header and FetchResponseData into a buffer
  3. Writes the actual records using:
_writable.writeInt(records.sizeInBytes());
_writable.writeRecords(records); // FileRecords path

Zero-Copy Optimization

When writing records, Kafka distinguishes between:

  • MemoryRecords: in-memory buffers
  • FileRecords: backed by disk files

For FileRecords, Kafka flushes any pending in-memory data to a Send Queue item and then calls: addSend(records.toSend()) wraps the FileRecords into a Send implementation (e.g., DefaultRecordsSend) and append it to the Send queue in SendBuilder.

Summary: Buffer and Send Queue

  • Metadata is serialized into a ByteBuffer , converted to a SendQueue item when a
  • Log data (FileRecords) is added as a Send
  • Both are added into the SendBuilder queue
  • Finally, Kafka returns a MultiRecordsSend object representing the full response

4. Network Sending with Zero-Copy

Once the MultiRecordsSend is ready, it is enqueued into the Processor’s response queue:

processor.enqueueResponse(response)

The processor then sends the response using the selector’s send() method. Kafka wraps the data in a NetworkSend, which sets the send object (MultiRecordsSend) on the corresponding KafkaChannel.

Kafka uses non-blocking I/O (NIO) and epoll to handle write-read events. Each client request is processed sequentially — one must complete before the next begins. This is managed through mute/unmute logic.

Writing to the Socket

When the channel becomes writable, Kafka calls:

send.writeTo(transportLayer)

This in turn invokes:

MultiRecordsSend.writeTo(channel)

Which iterates through its internal queue of Send objects and writes each one to the socket channel.

5. Final Step: FileRecords Zero-Copy Transfer

When the current Send is a DefaultRecordsSend, it delegates to the FileRecords.writeTo() method:

return (int) destChannel.transferFrom(channel, position, count);

Here, destChannel is Kafka’s PlaintextTransportLayer, and it ultimately calls JDK’s:

fileChannel.transferTo(position, count, socketChannel);

This leverages the zero-copy feature of the OS, enabling direct file-to-socket data transfer without copying to user-space memory.

Conclusion

Kafka’s fetch response pipeline — from data preparation to network transmission — is a masterclass in efficient I/O:

  • Uses FileRecords to avoid premature loading into memory
  • Defers serialization until needed
  • Constructs responses with minimal data duplication
  • Leverages JDK’s transferTo for zero-copy network sends

The entire process showcases how to combine Java NIO, efficient design patterns, and clean abstractions to build a high-performance distributed system.

Kafka is not just a messaging system — it’s also an exemplar of elegant and performant I/O engineering.