Kafka consumer problem

Hello,

I am new in forum and I started working with openPDC since few weeks.

I was trying to configure openPdc using kafka but in the consumer side I don’t see data read from kafka queue. In remote system monitor Console I see openPDc connected to kafka broker but it doesnt get any message from queue. I configured custom input with all kafka parameters needed.

On the producer side I was able to configure kafka output writes to the queue.
I am pretty sure something is writing into the queue because with a consumer console I am able to see some data serialized.
Any suggestions?

Thanks

There are two adapters for openPDC and Kafka, one is the TimeSeriesProducer, the other is the TimeSeriesConsumer. Generally these adapters are used to “interact” with Kafka. For example, to get synchrophasor data up into a Kakfa environment you would use the TimeSeriesProducer.

From here you would use the data pushed into Kakfa from within that environment as needed. Note that this includes a data topic, for time-series measurements, and a metadata topic, a data set that describes the measurements. You will need to configure the adapter to push the desired measurements into Kafka.

The TimeSeriesConsumer adapter exists to do just the opposite, i.e., read data positioned in Kafka back into the openPDC for processing. Note that the openPDC is designed to process real-time streaming data, this data would need to positioned in Kafka in a continuous fashion for this to be useful.

Also note that this openPDC adapter uses the Misakai Kafka C# library:

Here is its GitHub site for this library:

It looks like this project hasn’t been updated in about 5 years. As a result, it’s possible newer versions of Kafka might have trouble interacting with this library.

The adapter might need to be revamped with a better library, certainly an opportunity for community contribution :slight_smile:

Note that the code for the Kafka adapter used by the openPDC can be found in the Grid Solutions Framework:

Hope that helps.

Thanks,
Ritchie

Hello,

Thank you for answer.

I was able to configure the openpdc producer to send some measurements to a kakfa topic named test(f.e). It seems to write because with a console consumer I am able to see that something strange is receiving(serialized characters I suppose).

In another machine I installed openPdc with a consumer that has to read from test topic.
I didn’t set any options except the kafka input adapter with queue parameters.
In console I am able to see consumer is connected to the queue but data are not printed out. Basically data are not read from queue, but consumer is connected to it.

I noticed the topic metadata is empty while the real topic has data inside.

Is possible topic metadata is the problem. In that case how to fill it?

*I also using kafka with zookeeper

Certainly the metadata is important when interpreting the data. Note that if the metadata is empty, you may not have the adapter setup to receive any measurements.

On the adapter configuration screen, click on “Input Measurement Keys” and select a few values. Save the adapter changes then click the “Initialize” button.

Now see if there is any data in the metadata topic.

Thanks!
Ritchie

Hello,

I already put some values in input mesaurements in the producer side.
Do I have to put also in consumer side in output mesaurements?
I supposed it didn’t need consumer side. In consumer side I only created a custom input adapter.

Thanks

For clarity:

  • The KafkaAdapters.TimeSeriesProducer adapter is an OutputAdapter from the perspective of the openPDC, so here you would need to define input measurements for the adapter coming from the openPDC that would be published to Kafka.

  • The KafkaAdapters.TimeSeriesConsumer adapter is an InputAdapter from the perspective of the openPDC, so here you need define output measurements for the adapter coming from Kafka that would be published to openPDC - note that these measurements will need to be manually created in the openPDC in order to be available for processing by openPDC.