Openpdc full memory


#1

Hello,
Openpdc crash every 10-15 days, openpdc.exe use all memory and then crash.

We have :

  • 176 PMUs connected
  • CPU 2x8 core 2.4 Ghz
  • 32 GB of RAM
  • In the same server openpdc and openhistorian together

Any ideas what is wrong?

Best Regards,
Pietro


#2

Hello Pietro,

The best thing to do when openPDC is running out of memory is to grab a copy of the Status.txt file and search for the word “queued”. Try to find one with a really large number next to it. When troubleshooting memory buildup, there is nearly always a single queue in the system that is backed up with too many objects due to some sort of bottleneck in the communications pipeline.

Thanks,
Stephen


#3

Hello Stephen,
I found this line for one adapter:

  Status of IActionAdapter component 5, OUTPUTSTREAM:
           Output protocol: IEEE C37.118
      Configured time base: 16777215
        Validating ID code: False
  Configuration frame size: 13174 bytes
  Total configured devices: 91
 Total device measurements: 1057
 Auto-publish config frame: False
   Auto-start data channel: True
       Data stream ID code: 235
         Nominal frequency: 50Hz
               Data format: FloatingPoint
         Coordinate format: Polar
    Minimum output latency: 5092ms over 220 tests
    Maximum output latency: 5097ms over 220 tests
    Average output latency: 5093ms over 220 tests
       Digital normal mask: 0000000000000000 (big-endian)
 Digital valid inputs mask: 1111111111111111 (big-endian)

              Command Channel Status              
              ----------------------              
              Server state: Running
            Server runtime: 56 minutes 55 seconds
         Connected clients: 1
           Maximum clients: Infinite
            Receive buffer: 32768
        Transport protocol: Tcp
        Text encoding used: US-ASCII
       Queued payloads: 102997 for client 1

Data source defined: True
    Referenced data source: Iaon, 20 tables
    Initialization timeout: 15000 milliseconds
       Adapter initialized: True
         Operational state: Running
         Connect on demand: False
  Respecting input demands: False
 Respecting output demands: True
    Processed measurements: 158685575
    Total adapter run time: 56 minutes 54.89 seconds
       Temporal processing: Unsupported
                Adapter ID: 4
         Connection string: 22 key/value pairs

              DataChannel =                                                   
           CommandChannel = port=4725; maxSendQueueSize=-1; interface=0.0.0.0 
                   IdCode = 235                                               
   AutoPublishConfigFrame = 0                                                 
     AutoStartDataChannel = 1                                                 
         NominalFrequency = 50                                                
                  LagTime = 5                                                 
                 LeadTime = 2                                                 
          FramesPerSecond = 50                                                
  UseLocalClockAsRealTime = 0                                                 
      AllowSortsByArrival = 1                                                 
      IgnoreBadTimestamps = 0                                                 
           TimeResolution = 200000                                            
AllowPreemptivePublishing = 1                                                 
       DownsamplingMethod = LastReceived                                      
               DataFormat = FloatingPoint                                     
         CoordinateFormat = Polar                                             
      CurrentScalingValue = 2423                                              
      VoltageScalingValue = 2725785                                           
       AnalogScalingValue = 1373291                                           
PerformTimestampReasonabi = 1                                                 
         DigitalMaskValue = -65536 

It’s an output stream with 91 PMUS.

Best Regards,
Pietro


#4

If the command channel is showing that there are queued payloads, it means that the data is getting bottlenecked at the socket layer. I would recommend checking with the remote end of that connection and seeing if the system is running okay.

Another option to help with the buildup on your end would be to modify the maxSendQueueSize property of the command channel. I notice the current value is -1 which means there is no upper limit and the queue will build up infinitely. If you set it to something like 30000 (ten minutes of data at 50 fps), the system will dump the whole queue whenever it hits that threshold in the hopes of clearing out memory and staying close to real-time.

Thanks,
Stephen