That is helpful - could be a thread leak in the DNP3 adapter.
Yes, that is helpful. It further confirms my suspicion that thread pool threads are being leaked. On top of the fact that the error message explicitly states there aren’t enough thread pool threads, the setting for MaxThreadPoolWorkerThreads defaults to 2048, which is suspiciously close to that count of total threads used.
If the problem is related to a thread leak in the DNP3 adapter, what would you say our best bet is moving forward? Is it possible there are settings on our end that could be changed to remedy the situation, or is likely something internal that would require a patch / update of some sort?
Alternatively, could it be something machine-specific creating unforeseen circumstances to account for thread accumulation?
I suspect it will require a patch to fix. The next step is for me to attempt to reproduce in a test environment so I can attach a debugger and inspect the active threads. I haven’t had the time away from my active projects to get the test environment set up, but it’s on my to-do list.
I can certainly understand that, and I appreciate the time you guys have already invested in helping me with this. Is there anything I can do on this end (possibly attaching a debugger here) that may facilitate in isolating the problem? I’m not familiar enough to know if that’s something that has to be done in the development environment, or if it is relatively easy implement on a system like ours. Let me know if there’s any way I can assist, and thank you for your time!
As an open source project, the source code is available to you so it’s entirely up to your ability. You’d need to download the Grid Solutions Framework source code, build it with Visual Studio 2017/2019 Community in Debug mode, then copy the dlls from the debug build into an openHistorian instance that’s using DNP3 (replacing the existing dlls). With the openHistorian running, you should be able to use Visual Studio to attach a debugger to the process. The process can then be paused, and the threads can be inspected using the Threads view.
If Visual Studio is installed on a different system from openHistorian, I believe there are also options for remote debugging. However, I’ve never done that myself, and I’m not sure if there are any limitations.
We’ve updated our DNP3 engine to the latest version of OpenDNP3. When installing the nightly build (as of 9/30/2020 or later), you should now run the DNP3ConfigGenerator.exe from the install folder. This tool will extract the needed OpenSSL DLL’s and the required Visual C++ Redistributable Installer, i.e., VC_redist.x64.exe. The Visual C++ Redistributable Installer will need to be run before attempting to use the DNP3 adapter, otherwise the adapter may not show up as an available input adapter in the openHistorian manager. The DNP3 config generation tool will also create a sample device.xml comm file and mappings.xml mappings file in the openHistorian installation folder. If you already have files with these names in your openHistorian installation folder that are important, you should back these up before running the DNP3ConfigGenerator.exe.
Thanks,
Ritchie