CsvInputAdapter Example or Documentation?

Hello,

I have questions on how to get the CSVInputAdapter to work. I couldn’t find any examples or documentation on it except the source file : https://github.com/GridProtectionAlliance/gsf/blob/master/Source/Libraries/Adapters/CsvAdapters/CsvInputAdapter.cs

Unfortunately, with a quick look I was not able to understand what should I include in my sample CSV file ( should I include the time stamps? If yes, what is the format). Moreover, what should be the connection string? (ColumnMapping and its relation with OutputMeasurement, SimulateTimestamp and its relation with the first string in ColumnMapping)

If there is any sample CSV file and the corresponding connection string, I would appreciate if you could help me find it.

Best regards,

Reza

Hello Reza,

I provided just such an example a couple months ago on the CodePlex discussion boards. Here is the link.


EDIT: Copied from thread 649511 on CodePlex:

This is what the contents of my CSV file look like.

Timestamp,PPA:1,PPA:10,PPA:11,PPA:12,PPA:13,PPA:14,PPA:2,PPA:3,PPA:4,PPA:5,PPA:6,PPA:7,PPA:8,PPA:9
635931310170000000,8228,38.47470822,502.5757239,-131.8987716,214.2654052,178.9717246,59.971,0,-0.42,299567.9782,-132.3800155,298562.0961,-132.3510752,230.1433918
635931310170330000,8228,37.93852206,505.0788209,-132.3994386,213.7387727,178.6746272,59.965,0,0.09,299629.9759,-132.7277883,298623.1939,-132.700026,231.7047453
635931310170660000,8228,36.97593536,512.9061342,-133.4377751,212.1360972,178.2193576,59.971,0,-0.72,299813.5209,-133.0427584,298805.9561,-133.0160747,237.4509706
635931310171000000,8228,36.0862927,510.0617702,-134.0836948,212.1936964,177.7745998,59.969,0,1.73,299856.3056,-133.5532424,298848.435,-133.5215979,235.6415516
635931310171330000,8228,35.67089972,513.5070224,-134.566606,211.166653,177.3162248,59.954,0,-0.76,299822.1794,-133.839839,298795.6906,-133.8124328,238.0153716
...

Here is the connection string I used to configure the adapter in openPDC.

FileName=SHELBY.csv; 
AutoRepeat=True; 
SimulateTimestamp=True; 
TransverseMode=True; 
ColumnMappings={0=Timestamp; 1=PPA:1; 2=PPA:10; 3=PPA:11; 4=PPA:12; 5=PPA:13; 6=PPA:14; 7=PPA:2; 8=PPA:3; 9=PPA:4; 10=PPA:5; 11=PPA:6; 12=PPA:7; 13=PPA:8; 14=PPA:9}

I had to create a virtual device to associate with the measurements. PPA:1 through PPA:14 are the IDs of the measurements associated with the virtual device in my configuration.
http://www.gridprotectionalliance.org/Products/openPDC/Images/CSV/SHELBY-virtual.png
http://www.gridprotectionalliance.org/Products/openPDC/Images/CSV/SHELBY-measurements.png

After everything was set up, I was able to view the data on the Graph Measurements screen.
http://www.gridprotectionalliance.org/Products/openPDC/Images/CSV/SHELBY-graph.png


Thanks,
Stephen

Hi Stephen!

Thanks a lot for the link. It’s a useful example.
Yesterday, I wrote an action adapter that uses a dummy signal with the desired FPS rate as the Input Measurement and reads and publishes each row of the Excel file at each frame. Therefore, I don’t need to include the time stamps column in the Excel file.
It is very simple comparing to CSVInputAdapter but it works. However, it would be much better if I could remove the Input Measurement created itself by an input adapter.

Is there any way that the PublishFrame method is executed based on the defined FramePerSecond, without defining an input measurement?

Thanks,

Reza

There is currently no option in the concentrator to generate frames for missing data. There may be some tricks I haven’t thought of, but I currently know of three options to guarantee that you get a frame for every timestamp at a given framerate.

  1. Generate a dummy input measurement for every frame and send it into the concentrator.
  2. Use some sort of prediction algorithm to fill in data for missing measurements in advance of the frame publication.
  3. Ignore the behavior of the concentrator and fill in missing frames on a timer or when the next frame is received.

The following post contains boilerplate code for writing an adapter using a prediction algorithm.


EDIT: Copied from thread 653551 on CodePlex.

public class MyConcentratedAdapter : ActionAdapterBase
{
    private Ticks[] m_subsecondDistribution;

    public override void Initialize()
    {
        // Call base.Initialize() first to initialize the FramesPerSecond property
        base.Initialize();

        // Force preemptive publishing to false
        // IMPORTANT: This strategy will not work with preemptive publishing because every
        //            call to PublishFrame automatically fills the next frame with measurements
        AllowPreemptivePublishing = false;

        // Force the downsampling method to filtered
        DownsamplingMethod = DownsamplingMethod.Filtered;

        // Get the subsecond distribution for the configured frame rate
        m_subsecondDistribution = Ticks.SubsecondDistribution(FramesPerSecond);
    }

    public override void SortMeasurements(IEnumerable<IMeasurement> measurements)
    {
        base.SortMeasurements(measurements.Select(measurement =>
        {
            // Clone the measurement and set the filter to the priority filter
            IMeasurement clone = Measurement.Clone(measurement);
            clone.MeasurementValueFilter = PriorityFilter;
            return clone;
        }));
    }

    protected override void PublishFrame(IFrame frame, int index)
    {
        // Determine the index of the next frame after this one
        int nextFrameIndex = (index + 1) % m_subsecondDistribution.Length;

        // Calculate the timestamp of the next frame
        Ticks baselinedTimestamp = frame.Timestamp.BaselinedTimestamp(BaselineTimeInterval.Second);
        Ticks nextTimestamp = baselinedTimestamp + m_subsecondDistribution[nextFrameIndex];

        if (nextFrameIndex < index)
            nextTimestamp += Ticks.PerSecond;

        // TODO: Process your measurements here

        // Sort the measurements into the next frame
        base.SortMeasurements(frame.Measurements.Values.Select(measurement =>
        {
            // Get the predicted value for the next measurement
            // TODO: Apply your prediction logic here
            double predictedValue = measurement.Value;

            // Create the predicted measurement
            IMeasurement predictedMeasurement = Measurement.Clone(measurement, predictedValue, nextTimestamp);
            predictedMeasurement.MeasurementValueFilter = PredictedFilter;
            return predictedMeasurement;
        }));
    }

    // Define the filter for incoming measurements which are not predicted.
    private double PriorityFilter(IEnumerable<IMeasurement> measurements)
    {
        return Filter(measurements);
    }

    // Define the filter for predicted measurements.
    private double PredictedFilter(IEnumerable<IMeasurement> measurements)
    {
        return Filter(measurements);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    private double Filter(IEnumerable<IMeasurement> measurements)
    {
        // Get the most recent measurement which is not predicted
        IMeasurement measurement = measurements.LastOrDefault(m => m.MeasurementValueFilter != PredictedFilter);

        // If all we have is predicted measurements, just get the last measurement
        measurement = measurement ?? measurements.Last();

        return measurement.Value;
    }

    public override bool SupportsTemporalProcessing
    {
        get
        {
            return false;
        }
    }
}

Thanks,
Stephen

I just discovered the https link does not work. The http version works.

http://openpdc.codeplex.com/discussions/653551#post1471829

//aj

Hmmm. The https link still works for me. Nonetheless, thanks for providing the http link in case someone else needs it. :slight_smile:

Thank you for following up. The https link works for me Now, so maybe it was a transient issue - who knows if it was my end point or CodePlex.com? … Maybe it’s time to clear my DNS server’s cache.

Hi Stephen,

I have couple questions about the csv export adapter. Here I first start with the issue of creating a matrix with multiple columns per each data frame (time instant):
I followed your example but what I get as a csv result is not 15 columns. Instead all time stamps are in column 0, all IDs are in column 2, and all values are in column 3.
I want to have col0 timestamp, col1 Va, col2 Va angle, and so on.
Would you please help me with this issue?

Hi,Stephen!
I followed the steps that you advice in CodePlex ,but it doesn’t work.It seems that the virtual device is not running normally.So I was unable to view the data on the Graph Measurements screen.Could you give me some suggestions ?
Thanks,
wagebro

I’m not sure what you mean by “not running normally”. A virtual device doesn’t really “run” but rather simply exists in order for you to associate with measurement data that comes from your custom adapters.

I don’t don’t really feel confident providing suggestions here since I have nothing to go by, but here’s some basic troubleshooting advice.

  1. The column headers in the CSV file don’t matter. You can label them however you want. The ColumnMappings parameter maps each column’s position to the ID of the measurement in the database that describes the data in that column. Check the measurement names in this parameter against the ID of the virtual device’s measurements in your openPDC’s configuration.
  2. Make sure the timestamp is defined as a plain old number, not a formatted date/time. If you set SimulateTimestamp=False, the number you enter should be the number in ticks (100-nanosecond intervals since midnight on Jan 1, 0001).
  3. If you use AutoRepeat=True like in my example, the adapter will restart itself every time it reaches the end of the CSV file. If you copied the small portion of the example file I used for testing, you may want to try lengthening the file by adding more rows.

Hi,Stephen
Thank you very much !When I use this input adapter,I have seen a lot of other input adapters.Are there any details about these adapters?I want to know how to use these adapters.image
Thank you!
wagebro

Some of them are described in the documentation pages starting from here.
https://github.com/GridProtectionAlliance/openPDC/wiki/Documentation

Or you can peruse the source code for the adapters starting from here.
https://github.com/GridProtectionAlliance/gsf/tree/master/Source/Libraries/Adapters

I believe those are our best resources.