Hello!
I have a PMU streaming at 120fps and I’m capturing the data via UDP. I noticed that the historian is only saving 60% of my data. I changed the buffer size as recommended on the documentation, but no success so far. I know I’m receiving 100% of data because I checked on Wireshark. Is there anything I can do to avoid this?
Thanks in advance,
Juliana.
Hello Juliana,
Are you using the default historian in openPDC? How are you measuring the data loss?
Thanks,
Stephen
Hi! Yes, I am using the default historian.
I’m measuring data loss by exporting my data on HistorianView to a csv file. I noticed that after 600ms, it stops saving and only saves data again at the start of a new second, as seen in an example below:
08-Oct-2018 15:00:53.000,128.8942,128.8453,129.0331
08-Oct-2018 15:00:53.008,128.8853,128.8568,129.0379
08-Oct-2018 15:00:53.016,128.8782,128.8503,129.0591
08-Oct-2018 15:00:53.025,128.9199,128.8182,129.0483
08-Oct-2018 15:00:53.033,128.9242,128.8158,129.029
08-Oct-2018 15:00:53.041,128.924,128.8218,129.0215
08-Oct-2018 15:00:53.050,128.9191,128.8407,129.0215
08-Oct-2018 15:00:53.058,128.9129,128.847,129.016
08-Oct-2018 15:00:53.066,128.8969,128.8528,129.0075
08-Oct-2018 15:00:53.075,128.9134,128.8156,129.0002
08-Oct-2018 15:00:53.083,128.9127,128.8042,129.0035
08-Oct-2018 15:00:53.091,128.9182,128.7898,129.0078
08-Oct-2018 15:00:53.100,128.8956,128.7947,128.994
08-Oct-2018 15:00:53.108,128.8757,128.7943,128.9948
08-Oct-2018 15:00:53.116,128.8401,128.7671,128.9667
08-Oct-2018 15:00:53.125,128.8291,128.7343,128.9433
08-Oct-2018 15:00:53.133,128.8285,128.7304,128.9412
08-Oct-2018 15:00:53.141,128.8267,128.721,128.9515
08-Oct-2018 15:00:53.150,128.8284,128.7155,128.9371
08-Oct-2018 15:00:53.158,128.8184,128.7268,128.9386
08-Oct-2018 15:00:53.166,128.8151,128.7549,128.9575
08-Oct-2018 15:00:53.175,128.8336,128.7455,128.9597
08-Oct-2018 15:00:53.183,128.8568,128.7428,128.96
08-Oct-2018 15:00:53.191,128.8597,128.7498,128.9674
08-Oct-2018 15:00:53.200,128.8737,128.7442,128.9762
08-Oct-2018 15:00:53.208,128.8763,128.7555,128.9786
08-Oct-2018 15:00:53.216,128.8649,128.777,128.9686
08-Oct-2018 15:00:53.225,128.8671,128.7722,128.9753
08-Oct-2018 15:00:53.233,128.8741,128.7598,128.9725
08-Oct-2018 15:00:53.241,128.8541,128.7522,128.9545
08-Oct-2018 15:00:53.250,128.8414,128.734,128.9502
08-Oct-2018 15:00:53.258,128.8297,128.7335,128.9546
08-Oct-2018 15:00:53.266,128.8219,128.7473,128.9718
08-Oct-2018 15:00:53.275,128.8231,128.7401,128.9689
08-Oct-2018 15:00:53.283,128.8228,128.7338,128.9607
08-Oct-2018 15:00:53.291,128.8116,128.735,128.9527
08-Oct-2018 15:00:53.300,128.8229,128.7312,128.9612
08-Oct-2018 15:00:53.308,128.8234,128.738,128.9713
08-Oct-2018 15:00:53.316,128.8146,128.7371,128.9787
08-Oct-2018 15:00:53.325,128.8219,128.7153,128.9807
08-Oct-2018 15:00:53.333,128.8312,128.7215,128.9619
08-Oct-2018 15:00:53.341,128.8173,128.7215,128.957
08-Oct-2018 15:00:53.350,128.8144,128.7092,128.9733
08-Oct-2018 15:00:53.358,128.8213,128.708,128.96
08-Oct-2018 15:00:53.366,128.8111,128.7129,128.9596
08-Oct-2018 15:00:53.375,128.8096,128.7026,128.9577
08-Oct-2018 15:00:53.383,128.8191,128.6845,128.9464
08-Oct-2018 15:00:53.391,128.8208,128.6961,128.9542
08-Oct-2018 15:00:53.400,128.8052,128.7061,128.948
08-Oct-2018 15:00:53.408,128.8045,128.6925,128.9411
08-Oct-2018 15:00:53.416,128.8024,128.691,128.9416
08-Oct-2018 15:00:53.425,128.8065,128.6897,128.9421
08-Oct-2018 15:00:53.433,128.82,128.6818,128.9294
08-Oct-2018 15:00:53.441,128.8154,128.6815,128.9392
08-Oct-2018 15:00:53.450,128.7981,128.6962,128.9416
08-Oct-2018 15:00:53.458,128.8015,128.6961,128.9426
08-Oct-2018 15:00:53.466,128.8082,128.6904,128.9382
08-Oct-2018 15:00:53.475,128.8031,128.6975,128.9332
08-Oct-2018 15:00:53.483,128.7902,128.6888,128.9229
08-Oct-2018 15:00:53.491,128.786,128.6875,128.9265
08-Oct-2018 15:00:53.500,128.7823,128.6899,128.9271
08-Oct-2018 15:00:53.508,128.802,128.6911,128.9343
08-Oct-2018 15:00:53.516,128.8071,128.687,128.9331
08-Oct-2018 15:00:53.525,128.7997,128.6761,128.9436
08-Oct-2018 15:00:53.533,128.8177,128.659,128.9522
08-Oct-2018 15:00:53.541,128.8192,128.6648,128.9555
08-Oct-2018 15:00:53.550,128.8058,128.6945,128.9676
08-Oct-2018 15:00:53.558,128.8095,128.6886,128.977
08-Oct-2018 15:00:53.566,128.8238,128.6768,128.9692
08-Oct-2018 15:00:53.575,128.8308,128.6784,128.9529
08-Oct-2018 15:00:53.583,128.8327,128.6637,128.9546
08-Oct-2018 15:00:53.591,128.8368,128.6644,128.9563
08-Oct-2018 15:00:53.600,NaN,NaN,NaN
08-Oct-2018 15:00:53.608,NaN,NaN,NaN
08-Oct-2018 15:00:53.616,NaN,NaN,NaN
08-Oct-2018 15:00:53.625,NaN,NaN,NaN
08-Oct-2018 15:00:53.633,NaN,NaN,NaN
08-Oct-2018 15:00:53.641,NaN,NaN,NaN
08-Oct-2018 15:00:53.650,NaN,NaN,NaN
08-Oct-2018 15:00:53.658,NaN,NaN,NaN
08-Oct-2018 15:00:53.666,NaN,NaN,NaN
08-Oct-2018 15:00:53.675,NaN,NaN,NaN
08-Oct-2018 15:00:53.683,NaN,NaN,NaN
08-Oct-2018 15:00:53.691,NaN,NaN,NaN
08-Oct-2018 15:00:53.700,NaN,NaN,NaN
08-Oct-2018 15:00:53.708,NaN,NaN,NaN
08-Oct-2018 15:00:53.716,NaN,NaN,NaN
08-Oct-2018 15:00:53.725,NaN,NaN,NaN
08-Oct-2018 15:00:53.733,NaN,NaN,NaN
08-Oct-2018 15:00:53.741,NaN,NaN,NaN
08-Oct-2018 15:00:53.750,NaN,NaN,NaN
08-Oct-2018 15:00:53.758,NaN,NaN,NaN
08-Oct-2018 15:00:53.766,NaN,NaN,NaN
08-Oct-2018 15:00:53.775,NaN,NaN,NaN
08-Oct-2018 15:00:53.783,NaN,NaN,NaN
08-Oct-2018 15:00:53.791,NaN,NaN,NaN
08-Oct-2018 15:00:53.800,NaN,NaN,NaN
08-Oct-2018 15:00:53.808,NaN,NaN,NaN
08-Oct-2018 15:00:53.816,NaN,NaN,NaN
08-Oct-2018 15:00:53.825,NaN,NaN,NaN
08-Oct-2018 15:00:53.833,NaN,NaN,NaN
08-Oct-2018 15:00:53.841,NaN,NaN,NaN
08-Oct-2018 15:00:53.850,NaN,NaN,NaN
08-Oct-2018 15:00:53.858,NaN,NaN,NaN
08-Oct-2018 15:00:53.866,NaN,NaN,NaN
08-Oct-2018 15:00:53.875,NaN,NaN,NaN
08-Oct-2018 15:00:53.883,NaN,NaN,NaN
08-Oct-2018 15:00:53.891,NaN,NaN,NaN
08-Oct-2018 15:00:53.900,NaN,NaN,NaN
08-Oct-2018 15:00:53.908,NaN,NaN,NaN
08-Oct-2018 15:00:53.916,NaN,NaN,NaN
08-Oct-2018 15:00:53.925,NaN,NaN,NaN
08-Oct-2018 15:00:53.933,NaN,NaN,NaN
08-Oct-2018 15:00:53.941,NaN,NaN,NaN
08-Oct-2018 15:00:53.950,NaN,NaN,NaN
08-Oct-2018 15:00:53.958,NaN,NaN,NaN
08-Oct-2018 15:00:53.966,NaN,NaN,NaN
08-Oct-2018 15:00:53.975,NaN,NaN,NaN
08-Oct-2018 15:00:53.983,NaN,NaN,NaN
08-Oct-2018 15:00:53.991,NaN,NaN,NaN
08-Oct-2018 15:00:54.000,128.8329,128.7114,128.9592
08-Oct-2018 15:00:54.008,128.823,128.7217,128.9579
08-Oct-2018 15:00:54.016,128.8222,128.708,128.9691
Regards,
Juliana.
In my experience, these types of issues are almost always related to timestamp anomalies. Have you tried exporting the data without aligning timestamps?
I did now, and it still won’t export anything beyond 600 ms. For what it’s worth, I’ve already checked several times on Wireshark if all my data is arriving, which it is.
In the export that is not time aligned, do you see the same number of measurements in each second as in the time-aligned export?
I just checked it. Historian is saving 120fps, but apparently I get a timestamp every 4 ms, instead of 8 ms (which I already checked on Wireshark as the correct interval between phasors).
To clarify, you are saying that Wireshark is displaying 8 ms intervals between frames? I don’t know how much more I will be able to help without seeing some capture files from Wireshark and/or the PMU. If you’d like to, feel free to message me and attach some capture files and I will see what I can find out.
Hi.
Here’s an image from my Wireshark capture (I’m also attaching a capture file).capture.zip (1.7 MB)
As you can see, the interval between packets is around 8 ms.
Hi Juliana,
The timestamp in the Wireshark capture represents the time of receipt of each IP packet. For synchrophasor protocols, the PMU encodes a timestamp into the data frame to indicate exactly when the measurements in that frame were taken. The measurement timestamp is what the openPDC uses when storing the data in the historian.
Wireshark should be able to decode data frame packets so long as the capture also contains a configuration frame packet. This would allow you to check whether the openPDC and Wireshark agree on the 4 ms interval between timestamps.
Thanks,
Stephen
I just noticed you included the capture file in your post. I only see data frames, but I can see that the raw fraction of second (fracsec) value is the actual subsecond value of each frame down to 100 nanoseconds. In order to decode this properly, the timebase value found in the configuration frame should be 10000000. It could be that the timebase is set to 20000000 instead, which would cause the fracsec value to be halved when decoding the timestamp.
EDIT: Fracsec is a 24-bit integer, so the largest it could possibly be is 16777215. That would produce an interval of ~0.004967 seconds between frames.
I checked that my timebase was approximately what you mentioned. I just changed it and that fixed my problem. Thank you very much for your help! 
