Can anybody explain Apache Kafka for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? What does messaging mean? Messaging between what?!! At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

PS: Please explain as you would explain for a non-technical person

  • $\begingroup$ i think it will help if you can tell bit about your background in technology or no technology at all. That way may be i can or some body can come up with a reference from that domain to explain what kafka or similar things are. $\endgroup$
    – Raj
    Commented Oct 25, 2018 at 14:36

2 Answers 2


Can anybody explain Apache Kafka for me in a plain language?

As the official documentation of Kafka says:

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. It follows the publish-subscribe messaging style, with speed and durability built in.

Now, let me give a dumbed down explanation of the same:

A publish-subscribe(commonly called pub-sub) architecture basically contains a publisher(who sends the messages, for example: A twitter stream) and a subscriber(who receives the messages Ex: the analyst/the user's activity stream)

A clearer explanation would be:

In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead characterize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.

And keep in mind that Kafka is a distributed pub-sub messaging system, designed to scale.

enter image description here


At which stage of a BigData analysis is it used?

It is basically used in the Extraction step of the ETL pipeline. It can contain high volumes and speed of data flow (we call it high throughput in technical terms).

It can store messages also, but it is a persistent store; which means that the data is not stored forever. The stored data has an expiry date.

And what are prerequisites for learning it?

This is a tough question, as everyone who can program can get started with it, but if you really want to implement this in your architecture, then these are the pre-requisites which I can think of:

  • Know what is a persistent data store and where to use them
  • Know what a pub-sub messaging system is
  • Know what an extraction step is, in ETL. And also know how to deal with data with huge volume coming at you at high velocity.

Further reading:

Kafka has an excellent documentation, which can help to get started with it.

  • 1
    $\begingroup$ Thank you very much Dawny for your comprehensive answer. It gave some insight but still too technical for me. E.g. " Extraction step of the ETL pipeline" is a bit heavy for me! I know my knowldege of BigData structures is not enough but this post (along with two others about Flume & ZooKeeper) is supposed to give me a starting point. a VERY starting point! I'll accept your answer for sure if I do not get better (simpler) ones and I'd like to ask you to dig a bit deeper for me IF POSSIBLE! Thanks in advance! $\endgroup$ Commented Jan 11, 2016 at 14:54
  • $\begingroup$ @DanielWelke Welcome to the site :) Yeah, please wait for more better answers. For ETL, please have a look at this Wikipedia article, and for understanding Big Data tools, a clear understanding of what ETL means; is important. And that link would help you with that! $\endgroup$
    – Dawny33
    Commented Jan 11, 2016 at 14:58

It might be easier for you to take a look at the events streaming services that are offered in Amazon Web Services (AWS), mainly Kinesis.

The easiest of them in Kinesis Firehose (http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html), which gives you an endpoint where you can write your events to, and they will appear in Amazon Redshift, which is a data warehouse service that can scale to huge sizes (https://aws.amazon.com/redshift/).

AWS also announced that they will add Kinesis Analytics, which will allow you to write SQL code on top of the events stream of Kinesis. You can calculate different statistics such as moving agerages and other relevant statistic on your incoming events.

These events that are processed in near real time, can be clicks on your web site, purchases in your e-commerce site, security events that should be monitored for potential intrusion, etc.

  • 1
    $\begingroup$ How does this answer the question? $\endgroup$
    – Dawny33
    Commented Jan 12, 2016 at 6:12
  • $\begingroup$ Why did you ask this question? Are you writing an encyclopedia or trying to solve a problem? Kinesis and Kafka solve similar problems. In the context of data science (I think this is the site...), they can be used interchangeably. $\endgroup$
    – Guy
    Commented Jan 12, 2016 at 14:43
  • $\begingroup$ Yes, your answer is correct for What are some nice alternatives to Kafka?, but not for this question :) $\endgroup$
    – Dawny33
    Commented Jan 12, 2016 at 15:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.