PPAREADER for OpenHistorian can't read '.d2' files and replay data in pseudo real-time

Korunekosama · April 20, 2023, 12:40pm

Hello,

I’m having a problem with the ‘.d2’ archive replay mechanism in OpenHistorian. As I indicated in February 2022 (link) I had managed at the time to use OpenHistorian so that it ‘replays’ the data contained in ‘.d2’ files from the same version of OpenHistorian by specifying a time range (start - end).
I’ve tried to reproduce this behaviour with OpenHistorian 2.8.249.
My .d2 files are produced by a 2.8.249 OpenHistorian production server ; and I’ve exported a SQL dump of it’s MySQL database the same day of the .d2 file has been produced (18/04/2023) ; to ensure that the data in the database is consistent with that archived in the .d2 file.
On a second server used by the Research & Developpment Departement, I’ve installed the same version 2.8.249 of OH ; then imported the .sql dump in the local MySQL server DB to connect OpenHistorian on it ; and add the ‘.d2’ file I want to 'replay.

In OpenHistorian Manager, I’ve configured the PDC device connection string like this :

I’ve then configured the ‘PPA’ output historian instance like this :
MaximumArchiveDays=2; WorkingDirectory=D:\openHistorian\Archive_output; ArchiveDirectories={D:\openHistorian\Archive_output\Offload}; DesiredRemainingSpace=10; ArchiveCurtailmentInterval=43200; TargetFileSize=2

And finally create a ‘PPAREADER’ Custom Input like this :

With the following connection string :
InstanceName=PPA; ConnectOnDemand=false; ArchiveDirectories=D:\openHistorian\Datafiles_to_replay_Directory; PublicationInterval=1; ProcessingInterval=0; AutoRepeat=True; sourceIDs=PPA; StartTimeConstraint=04-18-2023 10:00:00; StopTimeConstraint=04-18-2023 10:30:00

The datafile I’m trying to use is named (by the OpenHistorian source on the production server) :
ppa-2023-04-18 07.37.47.000_to_2023-04-18 12.30.08.673-638174178139514799.d2 and is stored in the D:\openHistorian\Datafiles_to_replay_Directory directory.
So it must contain data for the 04/18/2023 from 07:37 UTC to 12:30 UTC.
On the PPAREADER connection string, I’ve specified a StartTimeConstraint of 10:00 to take in account the UTC+2 time difference.

When I’m initializing the Custom input PPAREADER, the following message is shown in the OH Console :

When I remove the start and stop time constraint in the Connection String, then this warning message is not shown ; but the data are not read by the PPAREADER.

This config used to work on 2022/02 (with an older version of OH that I don’t remember).
Would you have an idea of what I’m doing wrong please ?

We do not use anymore Historian 1.0 version from OpenPDC because it’s deprecated in recent version of OpenPDC ; and anyway we very regularly encountered problems with ‘corruption’ of ‘.d’ archiving files: PPA archiving works well for a variable period of time (several days / weeks / months …); then inexplicably generates errors (in yellow in the OpenPDC console) indicating that the measurements can no longer be saved in the files; and are therefore lost.
This is why we now only use the Historian 2.0 archive format with OH; but the loss of this possibility of replaying data archived in this format in pseudo real time would be very problematic for our ex-post analysis.

Any help would be appreciate,

Thanks

Stephane.

ritchiecarroll · April 22, 2023, 4:33pm

Hi Stephane,

I think the parameters for the 2.0 data reader are a little different than those of the 1.0 reader. Technically, a reader should already be automatically created with an openHistorian instance, this is used for temporal (historical) queries when using STTP.

In your case you are using parameters like ArchiveDirectories which were specific the old 1.0 reader.

Here are the default parameters for an openHistorian 2.0 PPA reader:

InstanceName=PPA; HistorianServer=127.0.0.1; ConnectOnDemand=true

In your case, if you are manually modifying the adapter to replay a specific time frame, you can modify the parameters like so:

InstanceName=PPA; HistorianServer=127.0.0.1; ConnectOnDemand=true;
PublicationInterval=1; ProcessingInterval=0; AutoRepeat=True;
StartTimeConstraint=04-18-2023 10:00:00; StopTimeConstraint=04-18-2023 10:30:00

Note that for these specific settings the openHistorian 2.0 reads data way faster than the 1.0 version. Setting the PublicationInterval to 1 is really saying to read the data as fast as possible. When you do that with AutoRepeat set to true, any concentrators that may be concentrating data will start to hold on to data until is time for publication which could cause memory pressure - or - in your case, if you are using historical timestamps, then you will just be injecting the same data with the same timestamps into concentration frames over and over as fast as possible which will cause CPU pressure. I would suggest setting PublicationInterval = 200000 for example, which would read data at closer to 50 frames per second - however you can adjust this as needed if timer should need to run a little quicker, but setting it to 1 seems excessive.

Another thing to consider is that the default PPAREADER adapter is there to provide temporal query support for STTP connections, i.e., to provide historical data to other connections that may be connected to the openHistorian. When you are manually making modifications to this adapter, such as, setting specific start/stop times, this prevents STTP historical queries from operating properly - NOTE: if this is a test instance of the openHistorian, then this does not matter of course, but this would matter if this is your primary openHistorian instance. Instead, I would suggest leaving PPAREADER with is default settings and adding another reader, perhaps ANALYSIS-READER which contains your target reader settings for reading a specific time range - this adapter could then just be enabled as needed to query a data range.

Another option to consider is leaving the PPAREADER to its defaults settings and using an STTP connection to the openHistorian from a test instance of the openPDC. In this case the openPDC could be setup to not query and data in real-time, but rather, change its STTP input adapter to specify a start and stop time, thus causing the openPDC instance to only retrieve historical data. This is just a thought to consider…

Hope that helps, sorry that I don’t fully understand your currently deployed environment, so my guidance my be limited.

Thanks,
Ritchie

Korunekosama · April 24, 2023, 12:53pm

Thank you very much for this very complete and exhaustive answer; thanks to which I actually managed to achieve the desired result. This consisted of being able to play in simulated real time on a non-production, dedicated test server, past time periods stored in .d2 files produced by my ‘production’ openHistorian.
The purpose of the process is to run custom adapters for calculating and detecting oscillations on this ‘test’ openHistorian server. These adapters only work on the ‘real time’ data stream (simulated here) received by OpenPDC (and OpenHistorian since these adapters seem to be compatible with these two products).
The interest is to be able to operate these adapters several times in a row on the same data set by varying the detection parameters to study their influence on the detected oscillatory phenomena and their characteristics.

Since my openHistorian data replay server is out of production and dedicated to this playback function, I can modify the default values of PPAREADER without the risk of disturbing the overall operation of my production openHistorian.

Once again : thank you very for the quick and efficient support provided !

Regards,

Stephane.