CsvInputAdapter Example or Documentation?


#1

Hello,

I have questions on how to get the CSVInputAdapter to work. I couldn’t find any examples or documentation on it except the source file : https://github.com/GridProtectionAlliance/gsf/blob/master/Source/Libraries/Adapters/CsvAdapters/CsvInputAdapter.cs

Unfortunately, with a quick look I was not able to understand what should I include in my sample CSV file ( should I include the time stamps? If yes, what is the format). Moreover, what should be the connection string? (ColumnMapping and its relation with OutputMeasurement, SimulateTimestamp and its relation with the first string in ColumnMapping)

If there is any sample CSV file and the corresponding connection string, I would appreciate if you could help me find it.

Best regards,

Reza


CSVAdapter for PMU Data
#2

Hello Reza,

I provided just such an example a couple months ago on the CodePlex discussion boards. Here is the link.


EDIT: Copied from thread 649511 on CodePlex:

This is what the contents of my CSV file look like.

Timestamp,PPA:1,PPA:10,PPA:11,PPA:12,PPA:13,PPA:14,PPA:2,PPA:3,PPA:4,PPA:5,PPA:6,PPA:7,PPA:8,PPA:9
635931310170000000,8228,38.47470822,502.5757239,-131.8987716,214.2654052,178.9717246,59.971,0,-0.42,299567.9782,-132.3800155,298562.0961,-132.3510752,230.1433918
635931310170330000,8228,37.93852206,505.0788209,-132.3994386,213.7387727,178.6746272,59.965,0,0.09,299629.9759,-132.7277883,298623.1939,-132.700026,231.7047453
635931310170660000,8228,36.97593536,512.9061342,-133.4377751,212.1360972,178.2193576,59.971,0,-0.72,299813.5209,-133.0427584,298805.9561,-133.0160747,237.4509706
635931310171000000,8228,36.0862927,510.0617702,-134.0836948,212.1936964,177.7745998,59.969,0,1.73,299856.3056,-133.5532424,298848.435,-133.5215979,235.6415516
635931310171330000,8228,35.67089972,513.5070224,-134.566606,211.166653,177.3162248,59.954,0,-0.76,299822.1794,-133.839839,298795.6906,-133.8124328,238.0153716
...

Here is the connection string I used to configure the adapter in openPDC.

FileName=SHELBY.csv; 
AutoRepeat=True; 
SimulateTimestamp=True; 
TransverseMode=True; 
ColumnMappings={0=Timestamp; 1=PPA:1; 2=PPA:10; 3=PPA:11; 4=PPA:12; 5=PPA:13; 6=PPA:14; 7=PPA:2; 8=PPA:3; 9=PPA:4; 10=PPA:5; 11=PPA:6; 12=PPA:7; 13=PPA:8; 14=PPA:9}

I had to create a virtual device to associate with the measurements. PPA:1 through PPA:14 are the IDs of the measurements associated with the virtual device in my configuration.
http://www.gridprotectionalliance.org/Products/openPDC/Images/CSV/SHELBY-virtual.png
http://www.gridprotectionalliance.org/Products/openPDC/Images/CSV/SHELBY-measurements.png

After everything was set up, I was able to view the data on the Graph Measurements screen.
http://www.gridprotectionalliance.org/Products/openPDC/Images/CSV/SHELBY-graph.png


Thanks,
Stephen


#3

Hi Stephen!

Thanks a lot for the link. It’s a useful example.
Yesterday, I wrote an action adapter that uses a dummy signal with the desired FPS rate as the Input Measurement and reads and publishes each row of the Excel file at each frame. Therefore, I don’t need to include the time stamps column in the Excel file.
It is very simple comparing to CSVInputAdapter but it works. However, it would be much better if I could remove the Input Measurement created itself by an input adapter.

Is there any way that the PublishFrame method is executed based on the defined FramePerSecond, without defining an input measurement?

Thanks,

Reza


#4

There is currently no option in the concentrator to generate frames for missing data. There may be some tricks I haven’t thought of, but I currently know of three options to guarantee that you get a frame for every timestamp at a given framerate.

  1. Generate a dummy input measurement for every frame and send it into the concentrator.
  2. Use some sort of prediction algorithm to fill in data for missing measurements in advance of the frame publication.
  3. Ignore the behavior of the concentrator and fill in missing frames on a timer or when the next frame is received.

The following post contains boilerplate code for writing an adapter using a prediction algorithm.


EDIT: Copied from thread 653551 on CodePlex.

public class MyConcentratedAdapter : ActionAdapterBase
{
    private Ticks[] m_subsecondDistribution;

    public override void Initialize()
    {
        // Call base.Initialize() first to initialize the FramesPerSecond property
        base.Initialize();

        // Force preemptive publishing to false
        // IMPORTANT: This strategy will not work with preemptive publishing because every
        //            call to PublishFrame automatically fills the next frame with measurements
        AllowPreemptivePublishing = false;

        // Force the downsampling method to filtered
        DownsamplingMethod = DownsamplingMethod.Filtered;

        // Get the subsecond distribution for the configured frame rate
        m_subsecondDistribution = Ticks.SubsecondDistribution(FramesPerSecond);
    }

    public override void SortMeasurements(IEnumerable<IMeasurement> measurements)
    {
        base.SortMeasurements(measurements.Select(measurement =>
        {
            // Clone the measurement and set the filter to the priority filter
            IMeasurement clone = Measurement.Clone(measurement);
            clone.MeasurementValueFilter = PriorityFilter;
            return clone;
        }));
    }

    protected override void PublishFrame(IFrame frame, int index)
    {
        // Determine the index of the next frame after this one
        int nextFrameIndex = (index + 1) % m_subsecondDistribution.Length;

        // Calculate the timestamp of the next frame
        Ticks baselinedTimestamp = frame.Timestamp.BaselinedTimestamp(BaselineTimeInterval.Second);
        Ticks nextTimestamp = baselinedTimestamp + m_subsecondDistribution[nextFrameIndex];

        if (nextFrameIndex < index)
            nextTimestamp += Ticks.PerSecond;

        // TODO: Process your measurements here

        // Sort the measurements into the next frame
        base.SortMeasurements(frame.Measurements.Values.Select(measurement =>
        {
            // Get the predicted value for the next measurement
            // TODO: Apply your prediction logic here
            double predictedValue = measurement.Value;

            // Create the predicted measurement
            IMeasurement predictedMeasurement = Measurement.Clone(measurement, predictedValue, nextTimestamp);
            predictedMeasurement.MeasurementValueFilter = PredictedFilter;
            return predictedMeasurement;
        }));
    }

    // Define the filter for incoming measurements which are not predicted.
    private double PriorityFilter(IEnumerable<IMeasurement> measurements)
    {
        return Filter(measurements);
    }

    // Define the filter for predicted measurements.
    private double PredictedFilter(IEnumerable<IMeasurement> measurements)
    {
        return Filter(measurements);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    private double Filter(IEnumerable<IMeasurement> measurements)
    {
        // Get the most recent measurement which is not predicted
        IMeasurement measurement = measurements.LastOrDefault(m => m.MeasurementValueFilter != PredictedFilter);

        // If all we have is predicted measurements, just get the last measurement
        measurement = measurement ?? measurements.Last();

        return measurement.Value;
    }

    public override bool SupportsTemporalProcessing
    {
        get
        {
            return false;
        }
    }
}

Thanks,
Stephen


#5

I just discovered the https link does not work. The http version works.

http://openpdc.codeplex.com/discussions/653551#post1471829

//aj


#6

Hmmm. The https link still works for me. Nonetheless, thanks for providing the http link in case someone else needs it. :slight_smile:


#7

Thank you for following up. The https link works for me Now, so maybe it was a transient issue - who knows if it was my end point or CodePlex.com? … Maybe it’s time to clear my DNS server’s cache.