Unable to make output with CSVInputAdapter

[ali94]

Hi,
OpenPDC works great for reading a CSV file. However, once I use real timestamp of CSV file measurements instead of simulated timestamp (when I put SimulateTimestamp=False in connection string of manage custom input adapters), PMU connection tester is unable to receive data. I am not sure where the mistake occurs. I would appreciate if you could help me. Here are screenshots that might help:

Thanks,
Ali

[Korunekosama]

HEllo,

I guess there is a control on the accordance between your local system time and the timestamps in your input file. If this difference is too large (lag time > few seconds) then they may be rejected by OpenPDC as it may consider them as obsolete.
I’m not sure that it’s the real cause of the problem ; but it could be.

Hope that could help

Stephane

[StephenCWills]

Hi ali94,

Stephane is correct about the time difference between the captured CSV data and your local clock. It’s almost impossible to synchronize these. You should consider checking your output stream configuration. If you are not using simulated timestamps, then you must disable Use Local Clock As Real-Time under Advanced Properties. Also, on the CSV adapter, consider disabling AutoRepeat or else the timestamps will wrap back to the beginning and your output stream will start dropping data.

You may also want to check the openPDC Console and the error log (ErrorLog.txt) when initializing the CSV adapter to check for errors in the CSV adapter. Note that when SimulateTimestamps is set to false, the timestamp column must be specified in ticks, which is a long integer format representing the number of 100-nanosecond intervals since January 1, 0001.

DateTime.Ticks Property (System)

Thanks,
Stephen

[ali94]

Thanks for the reply, now it is possible to receive data with no problem. But there are other problems and I would appreciate if you could help me:

1-Some measurements (Analog Values, Digital Values, Calculated Values) appear to be broken, and the PMU connection tester is unable to receive them in this scenario.

2- Some time stamps and measurements are failed to match correctly in receiving point (PMU connection tester)

3- In the case of a miss data, the PMU connection tester deletes all measurements in a row of input data.

4- Some stats and measurements in Graph Real-time Measurements and Stream Statistics do not work.

Input data in CSV Input Adapter (problem 1)

export data in PMU connection tester (problem 1)

Input data in CSV Input Adapter (problem 2,3)

export data in PMU connection tester (problem 2,3)

Graph Real-time Measurements (problem 4)

Stream Statistics (problem 4)

[StephenCWills]

1. I’m not too sure what you’re referring to here. Which measurements in your screenshots are broken, and how can you tell?

2. The CSV adapter is reading the timestamps and measurement values from the same row of data in the CSV file. It really shouldn’t be possible for there to be a mismatch between them. I can’t tell from your screenshot how you determined that there was mismatch there so I can’t really come up with any theories as to how it might have happened.

3. It looks like the CSV adapter is using double.Parse() on the input data, which will throw an error if you pass it an empty string. As a result, it is skipping those lines of input where you have blank cells in the PPA:194 column. If you don’t have any valid data to enter into those blank cells, you can enter NaN instead to prevent the errors.

4. On the Graph Measurements page, the openPDC Manager is probably using the local clock to filter out stale measurements from the display. There should be a Display Settings link somewhere on that screen that enables you to modify that behavior so you can see the values of those measurements.
   .
   The device statistics are computed by the protocol parsers to determine network performance. The CSV adapter doesn’t produce statistics so that output on the Stream Statistics page is to be expected. I’m not sure whether any of those stats are even applicable to a CSV input or, if they were, what value they would provide.

[ali94]

Thanks for the reply, double.Parse() error is fixed but still the PMU connection tester does not receive analog, digital, or calculated values and the last row of input data in the CSV file(row 32).

Input data in CSV Input Adapter(highlighted columns are analog, digital and calculated values that are not available in PMU connection tester exported data)

export data in PMU connection tester

[StephenCWills]

… still the PMU connection tester does not receive analog, digital, or calculated values and the last row of input data in the CSV file(row 32).

Probably, this has to do with your output stream configuration in openPDC. Note that when you are adding a device to an output stream using the Output Device Wizard, it will not automatically include digitals or analogs. You must explicitly mark the checkboxes to include them.

Also note that this wizard will not include calculated values regardless because there is no corresponding signal type in the IEEE C37.118 standard. If you want to include calculated values, you will have to change the signal type to one that is supported by the standard, such as an analog value. This can be done explicitly for the output stream by mapping the input measurement to an output measurement with an appropriate signal reference, but you may find it easier to simply change the signal type of the input measurement before adding the device to the output stream using the wizard.

[ali94]

Thanks for the reply, Analog and Digital Values are successfully added but once I click on Initialize, these measurements disappear from Measurements in Manage Output Streams.

Input data in CSV Input Adapter(highlighted columns are analog, digital )

export data in PMU connection tester

[StephenCWills]

We do have some code that deletes invalid output stream measurements when reloading configuration from the database.

// Validate measurements associated with this output stream
foreach (DataRow outputStreamMeasurement in database.Connection.RetrieveData(database.AdapterType, $"SELECT * FROM OutputStreamMeasurement WHERE AdapterID = {adapterID} AND NodeID = {nodeIDQueryString}").Rows)
{
    // Parse output stream measurement signal reference
    deviceSignalReference = new SignalReference(outputStreamMeasurement.Field<string>("SignalReference"));

    // Validate that the signal reference is associated with one of the output stream's devices
    if (deviceAcronyms.BinarySearch(deviceSignalReference.Acronym, StringComparer.OrdinalIgnoreCase) < 0)
    {
        // This measurement has a signal reference for a device that is not part of the associated output stream, so we mark it for deletion
        measurementIDsToDelete.Add(outputStreamMeasurement.ConvertField<int>("ID"));
    }

    // Validate that the signal reference type is valid for an output stream
    if (validOutputSignalKinds.All(validSignalKind => deviceSignalReference.Kind != validSignalKind))
    {
        // This measurement has a signal reference type that is not valid for an output stream, so we mark it for deletion
        measurementIDsToDelete.Add(outputStreamMeasurement.ConvertField<int>("ID"));
    }
}

Both of these conditions have to do with the SignalReference field of the output stream measurements. After adding the analogs and digitals, can you check the signal references of your output stream measurements before initializing the output stream?

[ali94]

yes, output stream signal references are visible before initializing, but after initializing they disappear.

( before initializing)

( after initializing)

( Analogs that are generated automatically after adding Device Wizard)

( Digitals that are generated automatically after adding Device Wizard)

[StephenCWills]

Those measurements are being removed because their Signal Reference field isn’t right. It should be SHELBY2-DV1 for the digital value and SHELBY2-AV1 for the analog channel.

[ali94]

Thank you so much, I edited Signal Reference and it worked.