Create new COMTRADE file


#1

Hi, we are new to COMTRADE file and we are trying to use GSF Comtrade library in order to create a new COMTRADE config (CFG) file starting from another CFG.

Every time we add a new digital channel we find the channel is added 16 times in the schema and we don’t understand what is wrong.

Below the code used to create the file:

List<ChannelMetadata> channelMetadatas = new List<ChannelMetadata>();
foreach (var digital in parser.Schema.DigitalChannels) // parser is the original COMTRADE file
                {
                    channelMetadatas.Add(new ChannelMetadata
                    {
                        CircuitComponent = digital.CircuitComponent,
                        IsDigital = true,
                        Name = digital.Name,
                        SignalType = GSF.Units.EE.SignalType.DIGI
                    });                
}

Schema s = Writer.CreateSchema(channelMetadatas, parser.Schema.StationName, parser.Schema.DeviceID, parser.Schema.StartTime.Value, 1, parser.Schema.Version, FileType.Binary, parser.Schema.TimeFactor, parser.Schema.SampleRates[0].Rate, parser.Schema.NominalFrequency);

Can someone helpus with this topic?
Thanks
Claudio


#2

It probably has something to do with this block of code:

                if (record.IsDigital)
                {
                    // Every synchrophasor digital is 16-bits
                    for (int i = 0; i < 16; i++)
                    {
                        digitalChannels.Add(new DigitalChannel(schema.Version)
                        {
                            Index = digitalIndex++,
                            Name = record.Name,
                            PhaseID = "B" + i.ToString("X")
                        });
                    }
                }

In the past, I’ve called Writer.CreateSchema() with an empty list of ChannelMetadata and added all the analog and digital channels manually. You can see an example of that by following the link below.

You also have the option of using the Schema object’s constructor and writing to its properties manually. That’s basically what the Writer.CreateSchema() function does.

Thanks,
Stephen


#3

Hi Stephen,

thank you very much for your support. Everythings it’s ok now about create ASCII files but we have again a problem with BINARY files.

The schema’s file seem to be correct compared with a one generated by another software but the DAT file is bigger than one create by the other software (26kB vs 46kB) and when we try to read the new binary file an exception is raised:

“Failed to read enough bytes from COMTRADE BINARY file for a record as defined by schema - possible schema/data file mismatch or file corruption.”

Below the code snippet used to create the DAT file:

double[] values = new double[schema.AnalogChannels.Length + schema.DigitalChannels.Length];

int sampleIdx = 0;
while(sampleIdx < schema.SampleRates[0].EndSample)
{
   idx = 0;
   Array.Clear(values, 0, values.Length);

   AnalogExporteds.ToList().ForEach(x =>
                    {
                        values[idx++] = (x.ScalingIdentifier == ScalingIdentifiers.Primary) ? x.PrimaryData[sampleIdx].Y : x.SecondaryData[sampleIdx].Y;
                    });

   DigitalExporteds.ToList().ForEach(x => values[idx++] = x.PrimaryData[sampleIdx].Y);
   sampleIdx++;

   if (IsAsciiSelected)
   {
       Writer.WriteNextRecordAscii(asciiStream, schema, tsData.Value, values, i++);
    }
   else
   {
      Writer.WriteNextRecordBinary(binaryStream, schema, tsData.Value, values, i++);
   }
 }

Thank you very much for you support
Claudio


#4

Hi Claudio,

The first issue, I imagine, is in the method call you used to generate the COMTRADE data file records. Here is the method signature for WriteNextRecordBinary(). Note the injectFracSecValue flag defaults to true, which will add an extra digital value to your COMTRADE records. I suspect that is most likely not what you’re wanting, especially considering you left the fracSecValue field at its default value of zero. This advice also applies to the WriteNextRecordASCII() method.

        /// <summary>
        /// Writes next COMTRADE record in binary format.
        /// </summary>
        /// <param name="output">Destination stream.</param>
        /// <param name="schema">Source schema.</param>
        /// <param name="timestamp">Record timestamp (implicitly castable as <see cref="DateTime"/>).</param>
        /// <param name="values">Values to write - 16-bit digitals should exist as a word in an individual double value.</param>
        /// <param name="sample">User incremented sample index.</param>
        /// <param name="injectFracSecValue">Determines if FRACSEC value should be automatically injected into stream as first digital - defaults to <c>true</c>.</param>
        /// <param name="fracSecValue">FRACSEC value to inject into output stream - defaults to 0x0000.</param>
        /// <remarks>
        /// This function is primarily intended to write COMTRADE binary data records based on synchrophasor data
        /// (see Annex H: Schema for Phasor Data 2150 Using the COMTRADE File Standard in IEEE C37.111-2010),
        /// it may be necessary to manually write records for other COMTRADE needs (e.g., non 16-bit digitals).
        /// </remarks>
        public static void WriteNextRecordBinary(Stream output, Schema schema, Ticks timestamp, double[] values, uint sample, bool injectFracSecValue = true, ushort fracSecValue = 0x0000)

Next, as far as I can tell, you might be adding each individual digital value (1 or 0) to the array of values individually. COMTRADE actually serializes 16 digital values at a time in a single 16-bit value. Therefore, our writer assumes that you’re passing the digital values in as an array of 16-bit “digital words” that were cast to double before entering them into the array. You’ll need to organize your digitals into groups, sixteen digitals at a time, so you can convert each group to a 16-bit integer (ushort) and then add those 16-bit integers to your array of doubles.

As you might expect, I’ve done this before. In this example, I’m operating on a collection DataSeries that contains a colleciton of (time,value) data points where the values are 0 or 1. The code organizes them into groups of 16 consecutive data series and then combines each group into a single data series which represents the collection of values that will be written into the COMTRADE data file. While this code may not be directly applicable, it may give you some ideas on how to handle this. Note that the order of the bits in each digital word is opposite the order of the digital values in the schema file.

Thanks,
Stephen


#5

In response to this conversation, I went ahead and implemented some convenience methods with sane defaults for applications that do not use synchrophasors. If you choose to use these methods, you will need to get tomorrow’s nightly build of the GSF.COMTRADE library. If not, hopefully it will serve as a better example of what is required to successfully write your own COMTRADE data file.

Thanks,
Stephen