Using a Java Based Kafka Client in a Node.
js ApplicationYoni GoyhmanBlockedUnblockFollowFollowingApr 15A step by step guide for developing a Java based Kafka client in a Node.
js application using GraalVM.
The first time I heard about GraalVM, it totally blew my mind.
Being able to combine multiple languages in a single application or business logic is an incredibly useful and powerful tool.
A real life need for a polyglot application emerged once we decided to switch from RabbitMQ to Kafka as our messaging system.
Most of our RMQ consumers were written in Node.
js, and moving to a different messaging system would force us either use a Node.
js based library, or rewrite our entire business logic.
While there are several Node.
js based Kafka clients, using them poses limitations such as the implemented Kafka API version, or the exposed interfaces and customization options.
Using a Native Kafka client while maintaining the Node.
js business logic would be a real win for us.
This tutorial builds on this awesome medium post on developing with Java and JavaScript together using GraalVM.
We will be using Docker Compose to build and create our images.
A working example can be found here.
Setting up DockerThe minimal needs of our environment are having GraalVM, Zookeeper and Kafka installed.
The quickest way to achieve this is by using Docker and Docker Compose to create a complete running environment:A Docker Compose file containing definitions for zookeeper, Kafka and GraalVM.
Running docker-compose up -d from the containing folder will perform the following:Download a Zookeeper image and run it on port 2181 along with persistent data and log volumes.
Download a Kafka image containing a Kafka broker, and run it.
The broker will connect to a Zookeeper on port 2181, and will allow client connections on ports 9092 and 9093.
Download a GraalVM image.
This image will have GraalVM and Node.
js installed, and will have a shared volume with the host machine in the .
/code folder.
All defined ports will be exposed on the local machine (localhost:port).
Also, services will recognize each other based on their server name.
Accessing Zookeeper from the broker machine will be using zookeeper:2181 as the host name.
Same for kafka-broker:9092 for connecting with the Kafka broker.
Setting up the Java clientWe are going to be using Java 1.
8 and Maven to compile and run our Java client.
Even though the entire Kafka client will reside in a container, it will be helpful to run and debug our code directly from the host machine, using our favorite IDE.
To do that, Maven and Java need to be installed on the host machine.
Connection to other containers will be done using localhost as the host name.
Setting up MavenYou can use this tutorial to start a new Maven based Java project, or just use the following pom file:The above pom file will create the required Java application file structure, along with all the required dependencies.
Notice the ‘maven-shade-plugin’ we are using to compile a single ‘uber-jar’ for the client and all of its dependencies.
This will make it easier for us to add the client to the Node.
js application later.
Make sure to change your.
group.
id to your desired package name.
Creating a Kafka clientNext step is creating our Kafka client (consumer and producer).
We will implement a basic Kafka producer and then a consumer.
Add a Producer.
java file under /src/main/java/my/group/id:The producer in the example above can receive its configuration in a JSON format, and sends a string type message.
The main function in the Producer is an easy way of running the code and sending a test message.
Add a Consumer.
java file in the same folder:Same as the producer, this consumer receives its configuration in a JSON format.
After configuring our consumer, we start a new thread that connects to our Kafka broker and polls for messages.
Each new message is pushed into a queue which will later be used in our Node.
js application.
Compiling the codeRunning mvn package form within the root folder, will compile the code into a single jar file named ‘uber-kafka-client-1.
0.
jar’.
This file contains all required java code and dependencies, and will be used as a java library.
Setting up a Node.
js ApplicationLast but not least is our Node.
js application.
Add an index.
js file under node/services/kafka-user:The code above creates and configures a new Kafka consumer, and then uses node’s experimental workers to create a new thread that listens to messages from that consumer.
The consumer thread notifies the main thread when a new message arrives.
Notice the this.
queue = new java.
util.
concurrent.
LinkedBlockingDeque()on line 4.
This is possible due to using the GraalVM image.
This queue will be a shared instance with the Java consumer we previously defined.
Also, notice the const Consumer = Java.
type('my.
pakcage.
id.
Consumer')in line 20.
Again this is possible due to GraalVM, and will hold a reference to our Java based Kafka consumer.
Running the codeThe previously installed GraalVM image already contains node and GraalVM setup.
If one wishes to run the node application on the host machine instead, installing and configuring GraalVM is required (instructions).
To run our code inside the container, open a terminal from the root folder and type docker-compose run graalvm sh.
This will open a shell within the GraalVM image.
Due to our configuration all of our compiled code and scripts will be located under the .
/code folder.
Run the following command:node –polyglot –jvm –jvm.
cp=code/target/uber-kafka-client-1.
0.
jar — experimental-worker code/node/services/kafka-user/index.
jsThis command will run our node application as a polyglot application in a JVM.
Notice the — jvm.
cp parameter that tells JVM where to find our Java based Kafka client.
Trying it outKeep the terminal open, go back to the Java IDE, and run the Producer.
main procedure.
You should now see the following printed in you terminal:Console printoutSuccess!!SummaryGraalVM makes writing polyglot applications easy.
Adding a docker infrastructure, makes it even easier to develop and run cross-language applications just about anywhere.
The possibilities are virtually endless.
I hope this helps some of you and maybe inspires you to create some cross-language solutions to a real life problem you are facing.
.. More details