CSVExport Adapter issue with selected measurements

Hi,

I’ve been trying to configure the adapter to export incoming measurements into a CSV file.
(I need it to further archive averaged measurements into a separate database)

I configured the Adapter to export measurements from 2 distinct PMUs, but it won’t do that.
I tried reloading the OpenPDC Config a couple times, and despite having the correct settings in the configuration files, the adapter would create CSV files with measurements from either of the PMUs.
Also which measurements I got in the CSV file would change depending on how many times I reloaded the config.

I also tried exporting only the frequency on each PMU in the same file and that worked. So I’m a bit puzzled.

Best regards,

I’ll likely need more details about your specific configuration to give you any guidance here. What I will say is, if you are using multiple CsvOutputAdapter instances to export data into a single CSV file, it will not work. Instead, you should create a single CsvOutputAdapter instance and route all your measurements from both devices into that one adapter.

Hi Stephen,

I think you misunderstood my post.
I have OpenPDC installed to which 2 PMUs are connected as input devices.
I tried setting up 1 CSV output adapter including measurements from both devices (F, V mag, V ang, C mag, C ang), but cannot get it to work. It will randomly only write values from either a single PMU, or the other PMU + measurements I didn’t even select.
I tried issuing the ReloadConfig command a couple times, but it will only switch between the 2 status mentioned above (which are not matching the configuration). I’d say that in general, configuration changes seem to be randomly applied to the OpenPDC.

Ironically, I tried a temporary fix that is having 2 separate output adapters, 1 per pmu, writing files in 2 separate folders, and that seems to work fine for now,

Best regards,

Hi Stephen,

As per your last post here, I do use Initialize when performing configuration changes.

I think there’s a really good chance that I am still misunderstanding your post because there are a lot of important details that are still being left out of your explanations. It looks to me as though you’re keen to blame the openPDC’s logic for loading configuration, but from my perspective this is much more likely to be a configuration error, a logic error in the adapter, or user error than it is a defect in the openPDC software. As I indicated in my previous post, I’ll need more information to have the faintest hope of helping you.

  1. Are you using a built-in adapter or a custom adapter to export to CSV?
  2. What does your adapter configuration look like?
  3. What does your output CSV file look like? Also, what do you expect it to look like?
  4. What are these measurements you didn’t select? Or perhaps more to the point, what measurements did you select, how did you select them, and what is showing up in your CSV file that you didn’t expect to see?

All that said, it sounds like you’re fine with your temporary fix for now. If you’re feeling content and would like to simply keep tinkering on your own, don’t let me pressure you into answering these questions.

Hi Stephen,

  1. Are you using a built-in adapter or a custom adapter to export to CSV?
    I am using the built-in CSV Export found under Manage custom Outputs
  2. What does your adapter configuration look like?
    Here is the connection string for the CSV adapter:

ExportPath=PathtoDestination/tmp; InputMeasurementKeys={fd9bd7f1-blabla;0880a7c1-blabla;0318fc6a-blabla;194387d3-blabla;05cc7581-blabla;936ca9e3-blabla;c3799c5a-blabla;62caf16c-blabla;a1b0f69e-blabla;e7642625-blabla;3b9702ac-blabla;9d090545-blabla;77b22ec5-blabla;df2fea3e-blabla;48a707f7-blabla;157a3f6c-blabla;28469ee3-blabla;d689c648-blabla;4812e6b0-blabla;66652897-blabla;70ddae22-blabla;49d92f39-blabla;0b319b1f-blabla;cefcb79c-blabla;a6d76348-blabla;baef7448-blabla}; RolloverSchedule=*/1 * * * *; OffloadPath=PathtoDestination

  1. What does your output CSV file look like? Also, what do you expect it to look like?
    Here is what it could look like:

Timestamp,ID,Value
2019-11-09 00:00:51.4249983,f3ccc120-blabla,0 <—
2019-11-09 00:00:51.4249983,58febc95-blabla,0 <—
2019-11-09 00:00:51.4249983,0318fc6a-blabla,-162.586837513446
2019-11-09 00:00:51.4249983,0880a7c1-blabla,284.739318847656
2019-11-09 00:00:51.4249983,05cc7581-blabla,77.3856881264076
2019-11-09 00:00:51.4249983,194387d3-blabla,285.013061523438
2019-11-09 00:00:51.4249983,c3799c5a-blabla,-42.6271459506236
2019-11-09 00:00:51.4249983,936ca9e3-blabla,284.844573974609
2019-11-09 00:00:51.4249983,a1b0f69e-blabla,179.690447175393
2019-11-09 00:00:51.4249983,62caf16c-blabla,230.60514831543
2019-11-09 00:00:51.4249983,3b9702ac-blabla,63.6789396321069
2019-11-09 00:00:51.4249983,e7642625-blabla,236.001998901367
2019-11-09 00:00:51.4249983,77b22ec5-blabla,-61.2417063824694
2019-11-09 00:00:51.4249983,9d090545-blabla,248.433059692383
2019-11-09 00:00:51.4249983,fd9bd7f1-blabla,59.9873390197754
2019-11-09 00:00:51.4249983,8f43807d-blabla,1.39608759585538E-38 <—

Here is what it could also look like depending on how many times I would issue ReloadConfig and Initialize the adapter.

Timestamp,ID,Value
2019-11-09 04:07:47.6749973,157a3f6c-blabla,-134.691894150629
2019-11-09 04:07:47.6749973,48a707f7-blabla,123.552314758301
2019-11-09 04:07:47.6749973,d689c648-blabla,105.325314586775
2019-11-09 04:07:47.6749973,28469ee3-blabla,123.519721984863
2019-11-09 04:07:47.6749973,66652897-blabla,-14.6399267321193
2019-11-09 04:07:47.6749973,4812e6b0-blabla,123.541618347168
2019-11-09 04:07:47.6749973,49d92f39-blabla,-156.112064067084
2019-11-09 04:07:47.6749973,70ddae22-blabla,44.8208999633789
2019-11-09 04:07:47.6749973,cefcb79c-blabla,76.0225736133698
2019-11-09 04:07:47.6749973,0b319b1f-blabla,29.9991512298584
2019-11-09 04:07:47.6749973,baef7448-blabla,-29.5293839529103
2019-11-09 04:07:47.6749973,a6d76348-blabla,29.2774887084961
2019-11-09 04:07:47.6749973,df2fea3e-blabla,59.9652328491211

I was expecting:

Timestamp,ID,Value
2019-11-09 04:07:47.6749973,157a3f6c-blabla,-134.691894150629
2019-11-09 04:07:47.6749973,48a707f7-blabla,123.552314758301
2019-11-09 04:07:47.6749973,d689c648-blabla,105.325314586775
2019-11-09 04:07:47.6749973,28469ee3-blabla,123.519721984863
2019-11-09 04:07:47.6749973,66652897-blabla,-14.6399267321193
2019-11-09 04:07:47.6749973,4812e6b0-blabla,123.541618347168
2019-11-09 04:07:47.6749973,49d92f39-blabla,-156.112064067084
2019-11-09 04:07:47.6749973,70ddae22-blabla,44.8208999633789
2019-11-09 04:07:47.6749973,cefcb79c-blabla,76.0225736133698
2019-11-09 04:07:47.6749973,0b319b1f-blabla,29.9991512298584
2019-11-09 04:07:47.6749973,baef7448-blabla,-29.5293839529103
2019-11-09 04:07:47.6749973,a6d76348-blabla,29.2774887084961
2019-11-09 04:07:47.6749973,df2fea3e-blabla,59.9652328491211
2019-11-09 04:07:47.6749973,0318fc6a-blabla,-162.586837513446
2019-11-09 04:07:47.6749973,0880a7c1-blabla,284.739318847656
2019-11-09 04:07:47.6749973,05cc7581-blabla,77.3856881264076
2019-11-09 04:07:47.6749973,194387d3-blabla,285.013061523438
2019-11-09 04:07:47.6749973,c3799c5a-blabla,-42.6271459506236
2019-11-09 04:07:47.6749973,936ca9e3-blabla,284.844573974609
2019-11-09 04:07:47.6749973,a1b0f69e-blabla,179.690447175393
2019-11-09 04:07:47.6749973,62caf16c-blabla,230.60514831543
2019-11-09 04:07:47.6749973,3b9702ac-blabla,63.6789396321069
2019-11-09 04:07:47.6749973,e7642625-blabla,236.001998901367
2019-11-09 04:07:47.6749973,77b22ec5-blabla,-61.2417063824694
2019-11-09 04:07:47.6749973,9d090545-blabla,248.433059692383
2019-11-09 04:07:47.6749973,fd9bd7f1-blabla,59.9873390197754

  1. What are these measurements you didn’t select? Or perhaps more to the point, what measurements did you select, how did you select them, and what is showing up in your CSV file that you didn’t expect to see?
    See Above for the results of the files. I put an arrow at the measurements I didn’t want.
    I used the GUI to select measurements, I also tried creating a new adapter and copypasting the connection string directly.

Here is the config as found in SystemConfiguration.xml:

<OutputAdapters><ID>17</ID><AdapterName>TEST3</AdapterName><AssemblyName>CsvAdapters.dll</AssemblyName><TypeName>CsvAdapters.CsvExportAdapter</TypeName> <ConnectionString>ExportPath=PathtoDestination/tmp; RolloverSchedule=*/1 * * * *; OffloadPath=PathtoDestination/; InputMeasurementKeys={fd9bd7f1-blabla;df2fea3e-blabla;48a707f7-blabla;157a3f6c-blabla;28469ee3-blabla;d689c648-blabla;4812e6b0-blabla;66652897-blabla;70ddae22-blabla;49d92f39-blabla;0b319b1f-blabla;cefcb79c-blabla;a6d76348-blabla;baef7448-blabla;0880a7c1-blabla;0318fc6a-blabla;194387d3-blabla;05cc7581-blabla;936ca9e3-blabla;c3799c5a-blabla;62caf16c-blabla;a1b0f69e-blabla;e7642625-blabla;3b9702ac-blabla;9d090545-blabla;77b22ec5-blabla;}</ConnectionString> </OutputAdapters>

Thanks, I was able to find an issue with the CsvExportAdapter. There is an error in the Initialize() function that sets the InputMeasurementKeys property to null, regardless of the adapter’s configuration. This marks the adapter as a “broadcast adapter”, to which the system routes ALL measurements in the system with no filter applied. This explains why your CSV file has measurements you didn’t select.

I’m fixing the error in CsvExportAdapter now. The fix will be available in the nightly builds tomorrow morning.

As for your question about exporting from 2 PMUs into the same file, it should just work out-of-the-box. As a matter of fact, based on the issue I described above, the CsvExportAdapter should be exporting data from both devices no matter how you configure it. Therefore, I’m not sure what’s going on there.

That said, it’s important to note that the data in the CSV file is NOT sorted by time. Data gets dumped to the CSV file as it’s received, in the order it’s received. So if your network traffic is bursting or your PMUs’ timestamps are not aligned, you may only ever see one device’s signals associated with any given timestamp. In other words, it may not be immediately obvious even if both devices’ measurements are being exported to the CSV file.

I’m fixing the error in CsvExportAdapter now. The fix will be available in the nightly builds tomorrow morning.

Great I’ll make sure to update soon !.

Oh that’s great to know there is NO time alignment of the CSV output. It will be useful to debug.
Thanks for your help.