Kafka: The Definitive Guide: Real-time data and stream by Neha Narkhede

By Neha Narkhede

Methods to take complete benefit of Apache Kafka, the disbursed, publish-subscribe queue for dealing with real-time info feeds. With this finished e-book, you are going to know how Kafka works and the way it really is designed. Authors Neha Narkhede, Gwen Shapira, and Todd Palino make it easier to set up creation Kafka clusters; safe, track, and computer screen them; write rock-solid functions that use Kafka; and construct scalable stream-processing purposes.

Show description

Read Online or Download Kafka: The Definitive Guide: Real-time data and stream processing at scale PDF

Similar data modeling & design books

Modular Ontologies: Concepts, Theories and Techniques for Knowledge Modularization

This e-book constitutes a set of study achievements mature adequate to supply a company and trustworthy foundation on modular ontologies. It provides the reader a close research of the cutting-edge of the study zone and discusses the new recommendations, theories and methods for wisdom modularization.

Advances in Object-Oriented Data Modeling

Till lately, info structures were designed round assorted company capabilities, similar to debts payable and stock keep watch over. Object-oriented modeling, by contrast, constructions platforms round the data--the objects--that make up some of the enterprise services. simply because information regarding a selected functionality is proscribed to 1 place--to the object--the approach is protected against the results of swap.

Introduction To Database Management System

Designed particularly for a unmarried semester, first direction on database platforms, there are four facets that differentiate our ebook from the remaining. simplicity - more often than not, the know-how of database platforms could be very obscure. There are

Extra resources for Kafka: The Definitive Guide: Real-time data and stream processing at scale

Sample text

It is preferable to reduce the size of the page cache rather than swap. Why Not Set Swappiness to Zero? swappiness was always to set it to 0. This value used to have the meaning “do not swap unless there is an out of memory condi‐ tion”. 32-303. This changed the meaning of the value 0 to “never swap under any circumstances”. It is for this reason that a value of 1 is now recommended. There is also a benefit to adjusting how the kernel handles dirty pages that must be flushed to disk. Kafka relies on disk I/O performance to provide a good response time to producers.

Dirty_background_ratio+ value lower than the default of 10. The value is a per‐ centage of the total amount of system memory, and setting this value to 5 is appropri‐ ate in many situations. This setting should not be set to zero, however, as that would cause the kernel to continually flush pages, which would eliminate the ability for the kernel to buffer disk writes against temporary spikes in the underlying device perfor‐ mance. dirty_ratio, increasing it above the default of 20 (also a percentage of total sys‐ tem memory).

CPU Processing power is a lesser concern when compared to disk and memory, but it will affect overall performance of the broker to some extent. Ideally, clients should com‐ press message to optimize network and disk usage. This does require that the Kafka broker decompress every message batch in order to assign offsets, and then recom‐ press the message batch to store it on disk. This is where the majority of Kafka’s requirement for processing power comes from. This should not be the primary factor in selecting hardware, however.

Download PDF sample

Rated 4.95 of 5 – based on 27 votes