openHistorian Service stopping frequently

GPA Products openHistorian
1

Hi Ritchie,
Of late one of our openHistorian has been behaving strangely. When the service is initiated the CPU usage spikes to 100%, then stabilizes to about 65% and then suddenly the openHistorian service stops on its own with the CPU usage going down to 2-3%. After 3-4 iterations the service becomes stable with the CPU usage around 55%. In this regard I went through the error logs and have gathered the following findings:

***[9/27/2021 9:23:44 AM] (Inner Exception)***
Date and Time:         9/27/2021 9:23:44 AM
Machine Name:          Ausnet-OpenPDC-
Machine IP:            fe80::7de0:e9b8:e017:9b43%7
Machine OS:            Microsoft Windows NT 6.2.9200.0

Application Domain:    openHistorian.exe
Assembly Codebase:     C:/Program Files/openHistorian/openHistorian.exe
Assembly Full Name:    openHistorian, Version=2.6.30.2, Culture=neutral, PublicKeyToken=null
Assembly Version:      2.6.30.2
Assembly Build Date:   9/16/2019 1:14:56 PM
.Net Runtime Version:  4.0.30319.42000

Exception Source:      mscorlib
Exception Type:        System.Collections.Generic.KeyNotFoundException
Exception Message:     The given key was not present in the dictionary.
Exception Target Site: ThrowKeyNotFoundException

---- Stack Trace ----
   System.ThrowHelper.ThrowKeyNotFoundException()
       openHistorian.exe: N 00036
   System.Collections.Generic.Dictionary`2.get_Item(key As TKey)
       openHistorian.exe: N 00052
   GSF.Snap.Services.Net.SnapStreamingClient.GetDatabase(databaseName As String, encodingMethod As EncodingDefinition)
       openHistorian.exe: N 00081
   openHistorian.Adapters.<GetHistorianData>d__2.MoveNext()
       openHistorian.exe: N 00387
   openHistorian.Adapters.LocalInputAdapter.StartDataReader(state As Object)
       openHistorian.exe: N 00706
   System.Threading.ExecutionContext.RunInternal(executionContext As ExecutionContext, callback As ContextCallback, state As Object, preserveSyncCtx As Boolean)
       openHistorian.exe: N 00355
   System.Threading.ExecutionContext.Run(executionContext As ExecutionContext, callback As ContextCallback, state As Object, preserveSyncCtx As Boolean)
       openHistorian.exe: N 00020
   System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       openHistorian.exe: N 00115
   System.Threading.ThreadPoolWorkQueue.Dispatch()
       openHistorian.exe: N 00338


(Outer Exception)
**Date and Time:         9/27/2021 9:23:44 AM**
Machine Name:          Ausnet-OpenPDC-
Machine IP:            fe80::7de0:e9b8:e017:9b43%7
Machine OS:            Microsoft Windows NT 6.2.9200.0

Application Domain:    openHistorian.exe
Assembly Codebase:     C:/Program Files/openHistorian/openHistorian.exe
Assembly Full Name:    openHistorian, Version=2.6.30.2, Culture=neutral, PublicKeyToken=null
Assembly Version:      2.6.30.2
Assembly Build Date:   9/16/2019 1:14:56 PM
.Net Runtime Version:  4.0.30319.42000

Exception Source:      
Exception Type:        System.Exception
Exception Message:     UnhandledException

---- Stack Trace ----


**[9/27/2021 9:31:44 AM] Date and Time:         9/27/2021 9:31:44 AM**
Machine Name:          Ausnet-OpenPDC-
Machine IP:            fe80::7de0:e9b8:e017:9b43%7
Machine OS:            Microsoft Windows NT 6.2.9200.0

Application Domain:    openHistorian.exe
Assembly Codebase:     C:/Program Files/openHistorian/openHistorian.exe
Assembly Full Name:    openHistorian, Version=2.6.30.2, Culture=neutral, PublicKeyToken=null
Assembly Version:      2.6.30.2
Assembly Build Date:   9/16/2019 1:14:56 PM
.Net Runtime Version:  4.0.30319.42000

Exception Source:      GSF.PhasorProtocols
Exception Type:        System.IndexOutOfRangeException
Exception Message:     Index was outside the bounds of the array.
Exception Target Site: ParseCommonHeader

---- Stack Trace ----
   GSF.PhasorProtocols.IEC61850_90_5.FrameParser.ParseCommonHeader(buffer As Byte[], offset As Int32, length As Int32)
       openHistorian.exe: N 00808
   GSF.Parsing.FrameImageParserBase`2.ParseFrame(buffer As Byte[], offset As Int32, length As Int32)
       openHistorian.exe: N 00055
   GSF.Parsing.BinaryImageParserBase.ParseBuffer(buffer As Byte[], offset As Int32, count As Int32)
       openHistorian.exe: N 00453

It would be great if you could throw light on the issue.
Thanks,
Datta

2

The first error represents a failure to find the historian instance when attempting to read data from the openHistorian archive. Perhaps the archive was renamed and the reader’s instance name isn’t configured correctly.

The second error is a just an IEC 61850-90-5 parsing error. We could be receiving bad data from a device or handling an edge case in the protocol in an unexpected way.

Neither of these errors really points to a reason why you’re having those strange performance issues. This is pure speculation, but perhaps the parsing errors are causing your input adapter to restart periodically, causing the drop in CPU and eventual recovery.