Hello,
I’m looking at the last nightly build of openHistorian version, as it use Grafana 9.1.5. I’m surprised because since the beginning of my work with openPDC / openHistorian, I 've always used the LocalSystem account to run the service as it has been much more stable than the NT SERVICE\openHistorian one. I’ve tried this last version with both NT SERVICE\openHistorian and LocalSystem account for running openHistorian service :
-
NT SERVICE\openHistorian is working fine for the couple of hours I’ve tested it. But I don’t have rune it on a long period of time to make sure it’s stable.
-
LocalSystem account can make the service entering the ‘running’ state, but I have the following error in the openPDCConsole window ; and the openPDC Manager can’t connect to the service :
I report this as I don’t know if it’s an expected behavior or not ; which would mean that the LocalSystem account wouldn’t be a valid account for the service anymore ?
Regard
Stephane.
Hello Stephane,
The main difference between LocalSystem and NT SERVICE\openHistorian is the level of permission each one has to the resources on the local system. Any perception you have of the service being more stable when using one account or the other can likely be more appropriately attributed to Windows permissions rather than stability of the software or validity of the account. With that said, LocalSystem is still a perfectly valid account to use for the service.
The error you’re seeing on the console appears to indicate that the service encountered an error when attempting to authenticate the user of the console. This error may have nothing to do with the LocalSystem account. I recommend looking through the ErrorLog.txt file for any errors that might seem relevant. In particular, you should look for any errors that occur around the time that the authentication failure occurs.
Thanks,
Stephen
Hello Stephen,
Thank you for your quick reply.
On the past (my first openPDC installations when I started to work with it 3 years ago), I used NT SERVICE\openPDC account as it was the default choice proposed on a ‘from scratch’ installation. It used to work fine for a couple of days, then starting to make some errors when using the Console or Manager. Using the LocalSystem account solved that so I didn’t report it ; using this workaround 'till today. On the same way I use to deal with the default ‘password encrypted = true’ option in the openHistorian.console.exe.config and openPDC.console.exe.config because after some time the console stop working due to a encryption issue. The error is illustrated bellow :
Putting ‘false’ on this option in the console config file make it working great again (it sometimes come back to ‘true’ by itself, so I have to change it to ‘false’ again …).
Anyway for the present suspected issue : I’ve set up a configuration with NT SERVICE\openHistorian account to start the service and launch the Console : everything is OK.
Then I stop the service and close the Console ; changed NT SERVICE\openHistorian to LocalSystem and launch the service again : the service seems to work fine (‘running’ state) but the console can’t connect to the service, nor the Manager.
This is what I’ve got in the ‘error.log’ file :
[10/11/2022 8:50:01 PM] (Inner Exception)
Date and Time: 10/11/2022 8:50:01 PM
Machine Name: XXXXXXXXXXX
Machine IP: XXX.XXX.XXX.XXX
Machine OS: Microsoft Windows NT 6.2.9200.0
Application Domain: openHistorian.exe
Assembly Codebase: E:/openHistorian/openHistorian.exe
Assembly Full Name: openHistorian, Version=2.8.203.0, Culture=neutral, PublicKeyToken=null
Assembly Version: 2.8.203.0
Assembly Build Date: 10/6/2022 12:26:32 AM
.Net Runtime Version: 4.0.30319.42000
Exception Source: System
Exception Type: System.NotSupportedException
Exception Message: The server mode SSL must use a certificate with the associated private key.
Exception Target Site: InternalEndProcessAuthentication
---- Stack Trace ----
System.Net.Security.SslState.InternalEndProcessAuthentication(lazyResult As LazyAsyncResult)
openHistorian.exe: N 8026946
System.Net.Security.SslState.EndProcessAuthentication(result As IAsyncResult)
openHistorian.exe: N 00078
GSF.Communication.TlsServer.ProcessTlsAuthentication(asyncResult As IAsyncResult)
openHistorian.exe: N 00171
(Outer Exception)
Date and Time: 10/11/2022 8:50:01 PM
Machine Name: XXXXXXXXXXX
Machine IP: XXX.XXX.XXX.XXX
Machine OS: Microsoft Windows NT 6.2.9200.0
Application Domain: openHistorian.exe
Assembly Codebase: E:/openHistorian/openHistorian.exe
Assembly Full Name: openHistorian, Version=2.8.203.0, Culture=neutral, PublicKeyToken=null
Assembly Version: 2.8.203.0
Assembly Build Date: 10/6/2022 12:26:32 AM
.Net Runtime Version: 4.0.30319.42000
Exception Source:
Exception Type: System.Exception
Exception Message: Unable to authenticate connection to client [127.0.0.1]: The server mode SSL must use a certificate with the associated private key.
Note that on the same server, I’ve also an ‘old’ version of OpenPDC (2.9.43) that is working fine with the LocalSystem account :
OpenPDC 2.9.43 service :
OpenHistorian 2.8.203 (on the same server)
Hope this can help ?
Regards,
Stephane
The error seems to refer to the openHistorian service not having access to the private key for the openHistorian.cer file. If that file was generated by the NT SERVICE\openHistorian account, then there is indeed a good chance that the private key is stored in that account’s personal certificate store, meaning that you’ll have no real way of accessing it without changing the account back and using a special openHistorian Console command.
This shouldn’t really be an issue, however, because openHistorian is supposed to simply generate a new self-signed certificate if it can’t access the private key for the old one. We recently made a change to generate new certificates using PowerShell as opposed to the deprecated makecert utility. Perhaps this new process does not work as expected when running the service as LocalSystem.
If the system fails to generate a certificate at startup, it will log details about the error to the Windows Event Log. To find it, open Event Viewer and navigate to Windows Logs > Application, then search for messages from the openHistorian with type Error that occur around the time when the openHistorian service was started.
Thanks,
Stephen
Thank you for the suggestion. I’ve taked a look on the Windows events viewer and it seems that the problem is due to MacAffee : "*NT AUTHORITY\SYSTEM ran E:\openHistorian\openHistorian.exe, which attempted to access the process powershell.exe, violating the rule “T1055.003 - Creating a thread in another process”, and was blocked. That could explain why the PowerShell command did not go to the end.
For information about how to respond to this event, see KB85494.*"
Note that since we have migrated from MacAffee to Trellix/MacAffee I have a significative number of warning/errors on our servers with recent versions of OpenPDC/openHistorian :
As examples :
- The application E:\openHistorian\openHistorianConsole.exe was contained at the request of Adaptive Threat Protection.
- Adaptive Threat Protection ran the openHistorianConsole.exe application in a container because its reputation (Unknown) is below the configured containment threshold.
- NT SERVICE\openHistorian ran E:\openHistorian\openHistorianConsole.exe, which accessed the file C:\Windows\assembly\NativeImages_v4.0.30319_64\openHistorianConsole\f7616bac53879cf2d9e34358fdb7d944\openHistorianConsole.ni.exe, violating the rule “T1059 - Executing any child process”. Access was allowed because the rule wasn’t configured to block.
- NT SERVICE\openHistorian ran E:\openHistorian\openHistorianConsole.exe, which accessed the file E:\openHistorian\DynamicAssemblies\RazorEngine_0rpmd32d.hnz\CompiledRazorTemplates.Dynamic.RazorEngine_c5d02dc42ac945ed9cddd69175accb40.dll, violating the rule “T1554 - Modifying portable executable files”. Access was allowed because the rule wasn’t configured to block.
- The application E:\openHistorian\openHistorianConsole.exe was released from containment at the request of .
- The application E:\openHistorian\openHistorian.exe was contained at the request of Adaptive Threat Protection.
** NT AUTHORITY\SYSTEM ran E:\openHistorian\openHistorian.exe, which accessed the file E:\openHistorian\openHistorian.cer, violating the rule “T1005 - Reading files commonly targeted by ransomware-class malware”. Access was allowed because the rule wasn’t configured to block*
And so on …
I’m on discussion with my IT security department on this issues ; but without any solution for the moment … Have you any other customers / users of this products that are having the same issues ?
Regards,
Stephane.
Yes, I’ve seen plenty of issues caused by virus scanners before. Honestly, we generally expect our products will always have an “Unknown” reputation with virus scanners because the use-case for our software doesn’t reach that many people. I can understand why IT security might apply these rules to their servers, but this isn’t really any cause for alarm as far as we are concerned so we’ll typically push it back on IT security departments to get the issue resolved through configuration of the virus scanning software.
If IT security is concerned about enabling our software to run a PowerShell script, there is a process you can go through to work around it. Basically, you would have to generate your own certificate, install it in the Personal certificate store for the local machine, adjust permissions to give openHistorian access to that certificate’s private key, place the .cer file in the appropriate location, and then restart the openHistorian service.
Of course, you can also simply disable TLS. This is a whole lot simpler, but the console interface would no longer be encrypted and the TLS data publisher would be inoperable.
Thanks for your reply Stephen,
I will see to manage the exceptions at the level of the AV rules (and/or asking the AV editor in case they could manage this at their level (AV engine for example) …).
Anyway thank you very much for the efficient and fast support.
Have a good day,
Stephane.
