I think this is the connection string you’re referring to - let me know if you had something else in mind. I uploaded the ErrorLog file to the link you provided.
That helps - thanks - looks like connection string is blank, which means everything is using the default settings.
While I look at your error log, I would recommend adding the following to the connection string:
desiredRemainingSpace=10; maximumArchiveDays=100
Then click Save and Initialize. You can adjust these settings later if you find you can archive more than 100 days of data.
Thanks,
Ritchie
So the error log is just the same message over and over, out of space.
I think the next best step is to apply the changes to connection string I mentioned above and report back.
Thanks,
Ritchie
Thanks for looking at this! I’ve updated the connection string as suggested, and I’ll report back with results after it’s been running long enough to see if behavior has changed.
After updating the connection string as noted, I’m still having server stability problems, but I have not seen any more error logs about being out of space. For the most recent freeze, I have some additional detail which may help.
I was viewing data in Grafana when it became unresponsive (around 9:25am) and it displayed several alerts saying “Annotation Query Failed / Internal Server Error”. However, the DNP3 adapter was apparently still working at that point because I logged into the connected RTAC and I could see that it was still getting polled by the OH computer.
After restarting the OH machine, the Grafana graph of the test data showed that OH was still receiving and recording DNP data for about 10 minutes after the web server became unresponsive:
I checked the error log for entries from today, and this was the first:
[6/11/2020 9:22:37 AM] Date and Time: 6/11/2020 9:22:37 AM
Machine Name: SEL
Machine IP: xxxx
Machine OS: Microsoft Windows NT 6.2.9200.0
Application Domain: openHistorian.exe
Assembly Codebase: C:/Program Files/openHistorian/openHistorian.exe
Assembly Full Name: openHistorian, Version=2.7.195.0, Culture=neutral, PublicKeyToken=null
Assembly Version: 2.7.195.0
Assembly Build Date: 5/5/2020 6:22:56 PM
.Net Runtime Version: 4.0.30319.42000
Exception Source: System
Exception Type: System.InvalidOperationException
Exception Message: There were not enough free threads in the ThreadPool to complete the operation.
Exception Target Site: BeginGetResponse
---- Stack Trace ----
System.Net.HttpWebRequest.BeginGetResponse(callback As AsyncCallback, state As Object)
openHistorian.exe: N 00514
System.Net.Http.HttpClientHandler.StartGettingResponse(state As RequestState)
openHistorian.exe: N 00082
System.Net.Http.HttpClientHandler.StartRequest(obj As Object)
openHistorian.exe: N 00084
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
openHistorian.Adapters.<ProxyPage>d__9.MoveNext()
openHistorian.exe: N 01090
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Threading.Tasks.<CastToObject>d__1`1.MoveNext()
openHistorian.exe: N 00234
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Web.Http.Controllers.<InvokeActionAsyncCore>d__1.MoveNext()
openHistorian.exe: N 00300
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Web.Http.Filters.<CallOnActionExecutedAsync>d__6.MoveNext()
openHistorian.exe: N 00307
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Web.Http.Filters.<CallOnActionExecutedAsync>d__6.MoveNext()
openHistorian.exe: N 00729
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Web.Http.Filters.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
openHistorian.exe: N 00435
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Web.Http.Controllers.<ExecuteAsync>d__5.MoveNext()
openHistorian.exe: N 00881
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Web.Http.Dispatcher.<SendAsync>d__15.MoveNext()
openHistorian.exe: N 00657
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Web.Http.Dispatcher.<SendAsync>d__15.MoveNext()
openHistorian.exe: N 01169
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
GSF.Web.Security.<SendAsync>d__13.MoveNext()
openHistorian.exe: N 00553
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
openHistorian.exe: N 00032
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(task As Task)
openHistorian.exe: N 00062
System.Web.Http.<SendAsync>d__24.MoveNext()
openHistorian.exe: N 00554
There were several more errors recorded subsequent to this, but all the same error.
Any ideas?
Thanks,
Mark
What are the specs on the system you are running this on? Interesting error about not enough free threads - haven’t seen that one before.
Thanks,
Ritchie
Here is a clipping from system info:
Is the one RTAC the only device providing data to the historian?
Yes, that’s correct.
About how long was the system running before it became unresponsive?
Roughly 40 minutes. If you look at the grafana chart above, the initial spike ramping up to 400A was the first test data I sent after restarting the machine.
We are looking into this, we don’t test the DNP3 adapter as much as other tools - so we are looking here first.
Thanks! I appreciate your assistance!
Because of the error that was generated, I was curious about the number of threads OpenHistorian is using on this system. So I checked this morning, and windows task manager shows openhistorian.exe (openHistorian Iaon Host) using 2,121 threads. In comparison, I’ve seen the openHistorian installed on my home laptop using only 68 threads at most, usually hovering between 60 and 65. I don’t know if this is useful info or not, but thought I would pass it on.
That is helpful - could be a thread leak in the DNP3 adapter.
Yes, that is helpful. It further confirms my suspicion that thread pool threads are being leaked. On top of the fact that the error message explicitly states there aren’t enough thread pool threads, the setting for MaxThreadPoolWorkerThreads defaults to 2048, which is suspiciously close to that count of total threads used.
If the problem is related to a thread leak in the DNP3 adapter, what would you say our best bet is moving forward? Is it possible there are settings on our end that could be changed to remedy the situation, or is likely something internal that would require a patch / update of some sort?
Alternatively, could it be something machine-specific creating unforeseen circumstances to account for thread accumulation?
I suspect it will require a patch to fix. The next step is for me to attempt to reproduce in a test environment so I can attach a debugger and inspect the active threads. I haven’t had the time away from my active projects to get the test environment set up, but it’s on my to-do list.
I can certainly understand that, and I appreciate the time you guys have already invested in helping me with this. Is there anything I can do on this end (possibly attaching a debugger here) that may facilitate in isolating the problem? I’m not familiar enough to know if that’s something that has to be done in the development environment, or if it is relatively easy implement on a system like ours. Let me know if there’s any way I can assist, and thank you for your time!
As an open source project, the source code is available to you so it’s entirely up to your ability. You’d need to download the Grid Solutions Framework source code, build it with Visual Studio 2017/2019 Community in Debug mode, then copy the dlls from the debug build into an openHistorian instance that’s using DNP3 (replacing the existing dlls). With the openHistorian running, you should be able to use Visual Studio to attach a debugger to the process. The process can then be paused, and the threads can be inspected using the Threads view.
If Visual Studio is installed on a different system from openHistorian, I believe there are also options for remote debugging. However, I’ve never done that myself, and I’m not sure if there are any limitations.