Kafka Streams and Kafka Streams – How to Create a Node Application That Consumes Kafka Messages

Kafka Streams and Kafka Streams – How to Create a Node Application That Consumes Kafka Messages

In previous articles on Apache Kafka, I have described how to create a Node.js application that publishes messages to a Kafka Topic, and how to create a simple Java application that does the same thing. In this article, I want to discuss a Node application that consumes the Top-N reports from the Kafka Topic produced to by the Kafka Streams application and periodically reports on the current standing.

The sources of this article are in the Repo.

The interaction with Kafka is done with the npm module kafka-node.

The ZooKeeper connect string is used in the creation of a new client. A consumer is created using the client. The consumer is told to consume from a top how to learn mobile app developmentic. The consumer is associated with a message handler.

The messages are consumed by the consumer.

“Top3CountryPerSizeContinent” is a topic.

The type buffer is the key. We know the key is a String. We can get a key like this.

var continent = new Buffer(countryMessage.key).toString(‘ascii’);

The top3 for the continent is the value property. It can be done quickly.

var top3 = JSON.parse(countryMessage.value);

“BS”, “name”, “population”, “size”, “13880”, “North America”

Each continent has a property in the object countrySize Standings. The property is the same as the top3 that was most recently eaten.

Every reportingIntervalInSecs second the report function is scheduled to be executed. The function writes the data to the console.

The program consumes messages from the topic Top3CountrySizePerContinent to which Running Top3 is produced. The program reports the top 3 largest countries per continent.

The running of the end to end chain requires a running Kafka Cluster and the running of the Node application to produce the country messages from thecsv file.

  • node KafkaCountryProducer.js
  • java -cp target/Kafka-Streams-Country-TopN-1.0-SNAPSHOT.jar;target/dependency/* nl.amis.streams.countries.App
  • node KafkaCountryStreamsConsumer.js

The CountryProducer.js application writes its messages to the console.

The console is also used to write the streaming analytic findings.

The outcome of the stream analysis is continuously reported to the console and updated with the latest findings:

Share this: Print

  • Tweet
  • Email

Leave a Reply