Hello,
Openpdc crash every 10-15 days, openpdc.exe use all memory and then crash.
We have :
- 176 PMUs connected
- CPU 2x8 core 2.4 Ghz
- 32 GB of RAM
- In the same server openpdc and openhistorian together
Any ideas what is wrong?
Best Regards,
Pietro
Hello Pietro,
The best thing to do when openPDC is running out of memory is to grab a copy of the Status.txt file and search for the word “queued”. Try to find one with a really large number next to it. When troubleshooting memory buildup, there is nearly always a single queue in the system that is backed up with too many objects due to some sort of bottleneck in the communications pipeline.
Thanks,
Stephen
Hello Stephen,
I found this line for one adapter:
Status of IActionAdapter component 5, OUTPUTSTREAM:
Output protocol: IEEE C37.118
Configured time base: 16777215
Validating ID code: False
Configuration frame size: 13174 bytes
Total configured devices: 91
Total device measurements: 1057
Auto-publish config frame: False
Auto-start data channel: True
Data stream ID code: 235
Nominal frequency: 50Hz
Data format: FloatingPoint
Coordinate format: Polar
Minimum output latency: 5092ms over 220 tests
Maximum output latency: 5097ms over 220 tests
Average output latency: 5093ms over 220 tests
Digital normal mask: 0000000000000000 (big-endian)
Digital valid inputs mask: 1111111111111111 (big-endian)
Command Channel Status
----------------------
Server state: Running
Server runtime: 56 minutes 55 seconds
Connected clients: 1
Maximum clients: Infinite
Receive buffer: 32768
Transport protocol: Tcp
Text encoding used: US-ASCII
Queued payloads: 102997 for client 1
Data source defined: True
Referenced data source: Iaon, 20 tables
Initialization timeout: 15000 milliseconds
Adapter initialized: True
Operational state: Running
Connect on demand: False
Respecting input demands: False
Respecting output demands: True
Processed measurements: 158685575
Total adapter run time: 56 minutes 54.89 seconds
Temporal processing: Unsupported
Adapter ID: 4
Connection string: 22 key/value pairs
DataChannel =
CommandChannel = port=4725; maxSendQueueSize=-1; interface=0.0.0.0
IdCode = 235
AutoPublishConfigFrame = 0
AutoStartDataChannel = 1
NominalFrequency = 50
LagTime = 5
LeadTime = 2
FramesPerSecond = 50
UseLocalClockAsRealTime = 0
AllowSortsByArrival = 1
IgnoreBadTimestamps = 0
TimeResolution = 200000
AllowPreemptivePublishing = 1
DownsamplingMethod = LastReceived
DataFormat = FloatingPoint
CoordinateFormat = Polar
CurrentScalingValue = 2423
VoltageScalingValue = 2725785
AnalogScalingValue = 1373291
PerformTimestampReasonabi = 1
DigitalMaskValue = -65536
It’s an output stream with 91 PMUS.
Best Regards,
Pietro
If the command channel is showing that there are queued payloads, it means that the data is getting bottlenecked at the socket layer. I would recommend checking with the remote end of that connection and seeing if the system is running okay.
Another option to help with the buildup on your end would be to modify the maxSendQueueSize property of the command channel. I notice the current value is -1 which means there is no upper limit and the queue will build up infinitely. If you set it to something like 30000 (ten minutes of data at 50 fps), the system will dump the whole queue whenever it hits that threshold in the hopes of clearing out memory and staying close to real-time.
Thanks,
Stephen