I have question about how to get the CSVExportAdapter to automatically save each hour of the reading from the PMU into a csv with the format of Figure1. Which is a matrix with 12 columns (voltage, its angle, voltage b, its angle, voltage c, its angle, current a, its angle, current b, its angle, current c, its angle) for each data frame instance.
Wit hthe current configuration, I only have the data as a single column and 12 rows for each frame. Beside, the file is only for 5 minutes.
Figure1: the target format
Just a thought, have you considered installing openHistorian (Releases · GridProtectionAlliance/openHistorian · GitHub), then connecting openHistorian to save data from openPDC. Then you could just “export CSV data” as needed in the desired format:
Unless it was a default checked box in the installation stage, I don’t think so.
Without having the openhistorian, we used to get exactly what I want as an output. But after adding the second PMU by another student, the output format changed. If possible I would prefer to have that desire output with the current open pdc interface.
I thought in the “Manage Custon Output Adapter” in the “CSV Export” by setting the right parameters in the “connection string” I can do the task.
For example for saving the csv files as 1 hour file instead of 5 minutes file, I assume setting the right string for RolloverSchedule=*/5 * * * * can be the solution.
Another example, for setting the right columns and rows, maybe “InputMeasurementKeys” , “ColumnMappings” or some other parameters that you may know may do the job.
I got this assumptions from reading the previous posts in the folloing links by Stephen. https://gridbits.gridprotectionalliance.org/t/csvinputadapter-example-or-documentation/35/2
and https://gridbits.gridprotectionalliance.org/t/csvexport-adapter-issue-with-selected-measurements/590/8
Thank you Stephen and James. In the open historian I could export the data in the format that I want. But I need an automatic exporting process to save csv files in constantly for every hour.
Unfortunately, no such process exists. In order to make that happen, the best approach would be to develop a custom adapter based on the CsvExportAdapter in Grid Solutions Framework since that adapter already implements a timer and rollover strategy that can be used for exporting data every hour. You will need to modify the logic to export your data in the format you require. Here is a link to the code for that module.
I use this ‘old’ topic because it’s related to one of my need : automatically producing one csv file per hour ; containing some selected measurements for a filtered list of PMU.
OpenPDC CsvExportAdapter seems to be good, but does not cover 100% of my criteria. I don’t know at the moment if I’m going to need, like Palberz, to group all the measurements associated with a PMU / a timestamp on a single line. However I know that I have to output a CSV file based on a framerate of 10 fps while the original stream is natively 50 fps. I tried using the FramesPerSecond=10 function but it doesn’t seem to work (no error in the console but no effect on the csv output). Is there a parameter for this adapter allowing to export in the output CSV only 10 values of a given measurement out of the 50 received per second ?
No, the CsvExportAdapter can’t change the sampling rate of your data. However, you are correct that this topic is related as the answer is pretty much the same. Concentration can be used to downsample the data you receive in real-time so you should be able to accomplish this by converting the CsvExportAdapter to a custom action adapter.
That said, if you don’t need to change the format of the CSV file, then it may be even easier to simply “abuse” the functionality of an existing action adapter. For example, you can set up a C37.118 output stream at 10 fps to forward downsampled data to another instance of openPDC, then set up CsvExportAdapter in the downstream instance.
I went ahead and added this feature - basically just exporting measurements on a specified interval specified in seconds, for example, 10 frames per second would be 1/10 seconds:
DownsampleInterval = 0.1;
Default value of 0.0 means do full resolution export, as before.
Also added new parameters to handle time reasonability checking, in case this is important for not exporting data with bad time:
EnableTimeReasonabilityValidation = true;
This is not enabled by default to maintain backwards compatibility.
Changes should be available in nightly build within an hour or so of this post.
Hello Stephen and Ritchie,
Thank you very much for your replies ; and many thanks for the improvement of the adapter !
Prior to the Ritchie code evolution I was actually considering creating a new downsampled output stream; but creating a new instance of OpenPDC (and therefore creating a new server) just for this function seems to me a bit excessive for the intended purpose. Another approach would have been to use an output file post-processing script to keep only one line out of 5 for each measurement in the file and for each PMU concerned; but in view of the large volume of each of these files, and the risk of errors that the script does not delete only the desired lines, this might not be optimal. Or how to use a sledgehammer to crack a nut.
So thank you very much for this functionality added to the adapter. I’ll get back to you as soon as I’ve been able to test it.
Have a good day,
Regards
Stephane
Edit : tested and approved !
I’ve put the ‘0 */1 * * *’ CRON parameter to make it cycling every hour at xx:00:00 and it works fine.
Thanks !
Hello Ritchie !
Thank you very much for this improvement.
Another possible one that I would find very useful would be to make it possible to customize the exported ID field in the .csv file produced; with for example the possibility of selecting another field of the Measurement table to identify the exported value. Indeed, with the current operation if you want to share the file produced with another partner who does not have this Measurement ‘reference’ table; it is not able to simply exploit the file to match each ID to its functional value.
This therefore implies having to post-process the file before communicating it; by replacing these SignalID values with a more understandable one (for example the value of the ‘Description’ field of the measurement table); and this for each line of the file which can be voluminous.
It would be interesting to have a parameter that would allow to specify the field value of the measurement table to associate with each exported measurement; for example measurement_label=“Description” to specify the value of the ‘Description’ field corresponding to the measurement, rather than the more abstruse SignalID value.
