Hi,
I have a very old Windows C++ app that collects 18 RS232 binary data streams from a Power Utility. I need to archive the collected data and play it back as a CSV. Openhistorian seems perfect for this task but I’m a bit confused. Do I create an input Adapter and use the OnMeasurement facility for each set of data, or do I use the HistorianInputQue methods?
I have OpenHistorian 2,6,30 running fine with a SQLLite database.
I have created a test application that uses the HistorianInputQue method on a local client connection (127.0.0.1) and and it works to write the data but it appears to loose data in my simulations. I have a very simple test that writes 22 measurements at a time, enques the measurements and waits for the queue to be empty, and starts again. It seems to write 3 entries, loose 4-5, repeat. I am using the DataExtractionUtility to convert the data to a CSV file to verify that what I wrote is in the .csv file extracted.
Is this the proper approach, or do I need to create an actual input adapter to OpenHistorian?
-thanks,
Michael
Ritchie,
This method is in C:\Users\michaelb\source\repos\openHistorian-2.6\Source\Libraries\openHistorian.Core\Queues\HistorianInputQueue.cs
I believe there was a demo program that I found that used it. I believe the starting point was
C:\Users\michaelb\source\repos\openHistorian-2.6\Source\Libraries\Adapters\openHistorian.Adapters\RemoteOutputAdapter.cs
Which had a pProcessMeasurements method. I took this and created a simple test application in the OpenHistorian solution.
-thanks,
Michael
Ritchie,
Also to add a bit. This application will end up being stand alone on a Windows box, it may have local web access.
It needs the historian archive and the ability to extract to CSV. It would be nice to have the graphing capabilities of the OpenHistorian Manager and ability to use a browser to access the Archive/Extract and Graph functions, but all the other adapters and ability to concentrate data are not needed at this time.
This is for a major Utility vendor, that’s why we thought using OpenHistorian would be useful, and extensions may be added over time.
-thanks,
Michael
If the ultimate goal is simply importing CSV files, there are many adapters already built-in that do this task. Always the challenge here is setting up the measurements prior to import, then mapping the measurements to columns in the CSV.
Note that the openHistorian web page includes a COMTRADE import feature that will automatically define measurements for source data and then import the data. If accessed locally: http://localhost:8180/ImportCOMTRADE.cshtml
CSV Adapters that Operate with Pre-defined Measurements:
Ritchie,
Yes, we need to extract CSV. I have done that feature using the OpenHistorian GUI and the web app. What I need to really get correct is the data collection. Is the best approach and highest speed using the OnMeasurement function and a custom Adapter? Currently I am using the HistorianInputQueue method which may be dropping data. At least the CSV files I’m extracting appear to have it dropping data.
I am really struggling with the custom adapter. I can create one, but I can’t figure out how to get it into the openHistorian with the manager application. My current method is a stand alone application using the HistorianInputQueue. Eventually I need to turn the whole thing into a continually running windows service.
In your opinion shoult it be able to handle 18 rs232 connections at 19200. Each connection generates approximately 30 16 bit words every 50 milliseconds.
-thanks,
Michael
Just my two cents on the data dropping problem. I wouldn’t expect that the HistorianInputQueue would be dropping data due to performance reasons. If data is truly being dropped from the archive, it would almost certainly be because the SignalID and Timestamp of each individual data point is not unique.
Stephen,
Yes you are correct. I suspected the time issue. Since my test is not really doing I/O it was very fast. I modified it so the point timestamp changes and I am not loosing data anymore.
I still need to understand whether I should use the HistorianQue method from a standalone application or the OnMeasurement from a device adapter dll.
-thanks,
Michael
First, you mentioned you have 18 connections with 30 16-bit words each at 50 milliseconds (20 samples per second). We’ve used the openHistorian in systems that archive data from hundreds of devices that provide ~20 signals at 30 samples per second. Matter of fact, openHistorian benchmarks have indicated that it is capable of archiving on the scale of millions of data points per second. The historian should be able to handle your data volume with no problem.
In regards to the HistorianInputQueue approach versus the OnNewMeasurements approach, I don’t think performance should be your primary concern. The difference between these two approaches would be that the HistorianInputQueue uses the SnapNetworkClient to send data over a socket whereas the input adapter is hosted within the openHistorian application itself. Therefore, the HistorianInputQueue has the obvious advantage of enabling you to host your custom code on another machine and send the data to the openHistorian over the network, and the input adapter has the advantage of bypassing the overhead of the socket connection, although I suspect the difference in performance is likely negligible. What may be less obvious is that the input adapter solution will provide the APIs to interact with the openHistorian configuration so you can apply the appropriate signal IDs to your measurements and then properly visualize them using the openHistorian web UI that Ritchie mentioned. The SnapNetworkClient bypasses the measurement routing mechanisms in openHistorian and writes measurements directly to the archive, which means that if you want to use the openHistorian web tools to access your data, you will have to synchronize configuration between your HistorianInputQueue app and the openHistorian database.
If your question was about whether to use the HistorianInputQueue within an input adapter to write measurements to the archive, I would obviously recommend against doing that since all it does is add the overhead of dealing with a socket-based connection. Instead, you should just call OnNewMeasurements. The measurements can make their way to the archive via the routing mechanisms in the host environment.
Steven and Ritchie,
Thank-you. The explanations above were very helpful. The difference between the HistorianInputQueue methods and OnNewMeasurements, was the piece I missed. I will create an Adapter and try that.
Is there a specific description on how to actually add a new adapter and points into the OpenHistorian?
I did try using the OpenHistorian manager and was not successful. Is it just adding the adapter info to the CustomInputAdapter table? It seems like there is more involved that I’m missing.
-thanks again,
Michael
Nearly all the tools in openHistorian Manager and the web UI expect the measurements you create to be associated with a Device record. So while it may not seem absolutely necessary in order to make your custom input adapter to function, I recommend creating a virtual input device to associate with your measurements. Here are the configuration steps.
Create a virtual input device by simply adding a new device and selecting Virtual Device from the protocol list. Also, make sure the device is enabled.
Create the input measurements and associate each one with the virtual device by selecting the device’s name from the Device drop-down list.
Create the CustomInputAdapter, but instead of explicitly specifying output measurements, use a filter expression: OutputMeasurements={FILTER ActiveMeasurements WHERE Device = 'MY_VIRTUAL_DEVICE'}
If you follow these steps, you can create new measurements by cloning the appropriate measurement from the OutputMeasurements property in your adapter code. You’ll need to determine which measurement is which through your program’s logic (which can be rather nontrivial), but here is a trivial example that assumes the adapter has just one output measurement.
