Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Servers: Kafka is run as a cluster of one or more servers that can span multiple datacenters or cloud regions. Some of these servers form the storage layer, called the brokers. Other servers run Kafka Connect to continuously import and export data as event streams to integrate Kafka with your existing systems such as relational databases as well as other Kafka clusters. A Kafka cluster is highly scalable and fault-tolerant.

Clients: They allow you to write distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and in a fault-tolerant manner even in the case of network problems or VM failures. Kafka ships with some such clients included, which are augmented by dozens of clients provided by the Kafka community: clients are available for Java and Scala as well as REST APIs.

Key Features of Kafka

Scalability: Handles scalability in all four dimensions, i.e. event producers, event processors, event consumers and event connectors. In other words, Kafka scales easily without downtime.
High-Volume: Can work with huge volumes of data streams.
Data Transformations: Kafka offers provision for deriving new data streams using the data streams from producers.
Fault Tolerance: The Kafka cluster can handle failures with the masters and databases.
Reliability: Distributed, partitioned, replicated and fault tolerant.
Durability: Configurable automatic reply sent out when a new ticket is opened or a message is received.
Replication: By using ingest pipelines, it can replicate the events.
Extensibility: There are as many ways by which applications can plug in and make use of Kafka. In addition, offers ways by which to write new connectors as needed.
Zero Downtime: Very fast and guarantees zero downtime and zero data loss.

CMAK - Cluster Manager for Apache Kafka

Kafka needs a tools for monitoring and managing its services. The developers has built a tools for its purpose called as CMAK (previously known as Kafka Manager) an open source tool for managing Apache Kafka clusters which was developed by Yahoo. Here, we will use this tools too. So we will install it on our VM.

CMAK is installed in the following directory: cd /opt/CMAK. Starting the CMAK services. ~/opt/CMAK/target/universal/cmak-3.0.0.5/bin/cmak The service will run on port 9000.

No contract needed: pay per hour

Apache Kafka with CMAK on Amazon Linux 2Ntegral Inc.

Apache Kafka with CMAK on Amazon Linux 2

Ntegral Inc.

Apache Kafka with CMAK on Amazon Linux 2

Ntegral Inc.

Apache Kafka with CMAK on Amazon Linux 2
Ntegral Inc.