How to import custom format CSV

GPA Products openPDC
1

The company I work for has a PMU device that records phasor measurements with outputs over C37.118 as well as a custom binary file that can be converted to CSV format. This CSV output does not contain all of the same data parameters as C37 data stream or any of the other protocols that can be assigned to input devices in the openPDC Manager. (Pasted images A/B at the bottom)

I have tried following discussions for CSV input adapter configurations to stream through a virtual input device at CsvInputAdapter Example or Documentation? but have a couple issues that I am not following with those instructions. First, I would like the measurements to go to my primary phasor archive (PPA), but am not sure how to go about the column mapping. The following is my current connection string on my custom CSV input adapter.

archiveLocation=C:\Program Files\openPDC\Archive; instanceName=PPA; sourceIDs=PPA; publicationInterval=333333; connectOnDemand=true; InputInterval=8.333333; MeasurementsPerInterval=1; SkipRows=3; ColumnMappings={2=Timestamp; 4=PPA:84; 5=PPA:85; 6=PPA:86; 7=PPA:87; 8=PPA:88; 9=PPA:89; 10=PPA:90; 11=PPA91; 12=PPA:92; 13=PPA:93; 14=PPA:94; 15=PPA:95}; FileName=C:\Users\Gabriel\Desktop\Hour_18\P3001760-20210602180000-PSL-Alameda.csv; AutoRepeat=True

I’m not sure what the appropriate measurement IDs are to map, but I tried matching what is shown for another input device connected over C37 (UPMU1760). (Pasted image C at the bottom)

Second, I am not sure how the “connection” is actually made between the virtual input device and the custom CSV input adapter. I have the following virtual input device configuration, but do I need a connection string for the file I want to stream? (Pasted image D at the bottom)

Is it possible to import that CSV data as is? If so, would I be able to get some guidance on how to correct my setup?

Otherwise, what would need to change in my CSV file format in order to import and be recorded to the PPA database?

Feel free to let me know if any clarification or additional information is needed. (Images A-D below)

openPDC_ticket
openPDC_ticket2551×1039 702 KB

Best regards,

Gabriel Gaona

2

Hi Gabriel,

The mapping between the CSV input adapter and the virtual input device is done through the measurements. You need to create new measurements and explicitly associate them with the virtual input device via the Device field. Then, the CSV input adapter will use the ColumnMappings connection string parameter to produce values for those measurements. With your current setup, the best you could hope for is that the CSV input adapter interleaves CSV data with your UPMU1760 input device, which is not what you want.

Having said that, I’m quite certain that the timestamp format is going to cause problems for the CSV input adapter. Your timestamps are split between the date stamp and time stamp columns in the CSV. The adapter was designed to pull all the timestamp information from a single column and parse it. If you can change the format of the CSV files to combine those into a single column, that should help. If not, your only option would be to tweak the CSV input adapter source code to make it work.

The last thing is, you’ve defined a bunch of extra connection string parameters that are useless at best, plus you’re missing the TransverseMode parameter. I would reduce the set of parameters down to what you need.

FileName=C:\Users\Gabriel\Desktop\Hour_18\P3001760-20210602180000-PSL-Alameda.csv; 
SkipRows=3; 
InputInterval=8.333333; 
AutoRepeat=True; 
TransverseMode=True; 
ColumnMappings={ ... }

You may also need ConnectOnDemand=true or SimulateTimestamp=true depending on your requirements. The rest don’t really apply to this adapter.

Thanks,
Stephen

3

Hi Stephen,

Thanks for the clarification. I now see that I missed the UI link to edit measurement mappings per input device. I have mapped the frequency and voltage/current phasor measurements, but I am curious to know more about which measurement types would be appropriate to map for some of the columns in the CSV I am working with.

For example, I have a C37 calculated frequency as well as a frequency calculation over one second in the CSV. Is it appropriate to add two frequency measurement types to one device with a unique point tag, signal reference and description for each?

Also, the status column in my CSV is a bitmap that reads differently from C37 status flags. Would there be any issue mapping that column to a status flags measurement type?

Thanks again for the help.

Gabriel

4

You should be able to map multiple frequency measurements to a device, however it’s important to note that this may break some assumptions for certain features. The best example I can give off the top of my head would be the IEEE C37.118 output streams. The protocol only accepts one frequency measurement per device, so you’d have to map multiple devices to transmit all your frequency signals or else map one of them as an analog value.

FYI, the signal reference of measurements is sometimes expected to be unique. With multiple frequency measurements, you can satisfy the uniqueness expectation by appending a signal index (SHELBY-FQ1, SHELBY-FQ2, etc.), but frequency measurements don’t normally include a signal index so that may also break some assumptions. Like I mentioned above, you can model one of the frequencies as a generic analog value (SHELBY-AV1), but this has some immediately apparent disadvantages because the system uses the signal type as a y-axis hint on charts. Having not tried doing any of this myself, it’s hard for me to say what would be the best option. If I were trying this myself, I would start by simply appending a signal index and see if it causes any problems.

As for the status flags, the first thing to note is that the CSV input adapter will parse all measurement values as base-10 floating point numbers. If your status flags is in hexadecimal, or if it exceeds a certain limit (I think around 40 or 50 bits), then the CSV input adapter will fail to parse it accurately.

Next, nearly all of the protocols we deal with use a 16-bit status word which we map into the lower 16 bits of a 32-bit integer. The protocol parser then maps the protocol-specific flags to a more universally defined set of flags and places them in the upper 16 bits. In other words, if the values in your status column can comfortably fit within a 16-bit integer, then you shouldn’t encounter any problems mapping it as a status flags measurement. However, the CSV input adapter is incapable of mapping your protocol-specific flags to the common set of flags so openPDC won’t be able to tell what types of errors your device is reporting.

If you feel comfortable with C# development, you may want to consider writing a FilterAdapter to map your status flags. The following enum describes the meaning of each of the bits in the upper word of the status flags.

5

Thanks again for the detailed response, Stephen. I was just able to get back to this today and will go over these considerations for my application needs. I appreciate the tip on the FilterAdapter. I’m certainly no expert with C#, but am happy to find opportunities to improve and learn more. I’ll try exploring the source myself some more.

Best,

Gabriel