Integrating Kafka with Grails
In this case I’m trying to integrate Kafka with Grails. I have a grails application that read tweets from Twitter and I want to increment the number of reads per second. The solution we applied was to have multiple instance of this (Producer), and use a message broker that allow me to handle this multiple instances results using the publish/subscribe pattern.
So the app instance publishes on Kafka and then another app (Customer) receives all the values. So I can add or remove instances for effectively actualising performance without stopping the operation.
We will not enter into how to read tweets, but grails has a plugin that simplify this.
First, we need to add the following libraries, that can be download from Maven:
Then on a grails class you need to create a method like:
def topic = 'topic'
def properties = [:]
KafkaProducer kafkaProducer = new KafkaProducer(properties)
ProducerRecord record = new ProducerRecord(topic, messageToSend)
This will save a message under the topic “Topic”, you can create different topics at the same time, and will work as different threads
Second, we need to create the client.
The client KafkaConsumer object was not working in this version, so we implemented a streaming of the messages instead. Once this is fixed you would be able to retrieve them on demand or even using the data cache.
First, we need to add the following libraries, that can be download from Maven as well:
Then, on grails class you need to create a method like:
Properties props = new Properties();
ConsumerConnector consumer = kafka.consumer.Consumer.createJavaConsumerConnector(new ConsumerConfig(props))
def consumerMap = consumer.createMessageStreams(['topic':1])
def streams = consumerMap.get('topic')
ConsumerIterator iter = m_stream.iterator()
def message = new String(iter.next().message())
As you can see, it is quite simple, you only need to install it. How to do it:
Step 1: Download the code
> tar -xzf kafka_2.11-0.8.2.1.tgz
> cd kafka_2.11-0.8.2.1
Step 2: Start the server
Need to start zookeeper first
> bin/zookeeper-server-start.sh config/zookeeper.properties
Now start kafka
> bin/kafka-server-start.sh config/server.properties
Step 3: Create a topic
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic
For more help about deploying Kafka go here
As you can see it is a very useful tool and can be integrated quite simple with Grails (and probably with java too)
For more info:
Kafka documentation press here
Grails documentation press here
Hope it helps!!!