TransWikia.com

How to fetch data from Kafka in NodeJS

Stack Overflow Asked on January 14, 2021

Currently I am trying to implement Kafka in Node.js. My whole project structure is a react front-end connecting to a Node.js server. While I am have a python based platform for storing sql and fetching data from database. So, I am trying to use Kafka to do the communication between Node.js server <-> Python Platform.

I can able to create the simple API in Express and push the data into Kafka(producer) and then trigger some "Fetch Data Class" in my Python Platform. However, now I stuck with getting the data and shown them in my React front-end. I have no much idea about how to return the data from Python plarform <=> Kafka consumer in Node.js and also my Kafka consumer in Node.js <=> Express. Please kindly help me out about the how to get the data from consumer by Express? Thanks!

React.js:

let url = "localhost/GetData"
axios
    .post(url,params)
    .then(function(response){
    if(response.data.data.length !== 0) {
        'do something
    })
    .catch(err => {
      console.error(err);
    })

Node.js – server.js:

router.post('/GetData', (req, res) => {
    getConnection(function(err, conn){
        if (err) {
            return res.json({ success: false, error: err })
        } else {
            let getData = new getDataObject(id, price, fruit);
            let getDataReq = Object.assign({}, producerRequest);
            let payloads = [];
            getDataReq .messages = [];
            getDataReq .messages.push(JSON.stringify(t.get()));
            payloads.push(getDataReq );
            
            kafkaProducer.send(payloads, function(err, result){
                if(err){
                    console.log(err)
                    return res.json({ success: false, error: err });
                }else{
                    console.log(result)
                    return res.json({ success: true, data: result });
                }
            })          
        }
    });
})

Node.js – kafkaproduced.js:

const kafka = require('kafka-node');
const { NODE_ENV } = require('../../config/env');
const config = require('../../config/kafka.json');
const logger = require('../logger')

const kafkaProducerConfig = '11.11.11.11'; -fake address
const {producer_kafka_server} = kafkaProducerConfig;
const producer_client = new kafka.KafkaClient({kafkaHost: producer_kafka_server});
const producer = new kafka.Producer(producer_client);

producer.on('ready', function(){
    logger.info('kafka producer is ready...')
})

producer.on('error', function(err){
    logger.error(err)
});

exports.producerRequest = {topic: 'get_database_data', messages:[], timestamp: Date.now()};

exports.kafkaProducer = producer;

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP