Reply

ISSUE : System swap space usage exceeds the critical threshold value

Authority
Posts: 29
1266     0

Hi Team,

 

Recently we have started observing 'System swap space usage exceeds the critical threshold value'. after checking the infoblox.log in Support Bundle I see the issue started few month back. The system started to log ***Process Memory Usage List Start***  every min. after the first time the SWAP sapce alert was triggered. From the Process memory usage list I can see only clusterd process used memory has been increasing over the time almost increased with 2GB there no change in used memory of other process.

 

Could someone help to resolve this issue. Also i would like to know:

1) I'm looking at the right logs(infoblox.log) to identify the route cause of SWAP memory issuye?

2) Any idea what is clusterd process and its Used Memory normaly increases over the time?

 

Note: We have enabled Bloxtool on the device

 

Error log message : "one_send_snmp_trap(): Sending threshold trap for XX.XX.XX.XX - swap_usage (System swap space usage exceeds the critical threshold value.) - value: 51 - high value: 50 - low value:30"

 

Firt occurance of process memory usage log output

 

(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:172 LogProcessMemory(): ***** Process Memory Usage List Start *****
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 1 Process Name: clusterd, PPid: 1, Pid: 3607, Memory Used: 10839532 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 2 Process Name: kworker/2:1, PPid: 2, Pid: 21249, Memory Used: 4459736 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 3 Process Name: qemu-kvm, PPid: 1, Pid: 20641, Memory Used: 4459736 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 4 Process Name: kvm-pit/20641, PPid: 2, Pid: 20642, Memory Used: 4459736 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 5 Process Name: dhcpd, PPid: 1, Pid: 28125, Memory Used: 3235000 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 6 Process Name: kworker/3:1H, PPid: 2, Pid: 28657, Memory Used: 3235000 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 7 Process Name: kworker/0:1H, PPid: 2, Pid: 29662, Memory Used: 3235000 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 8 Process Name: scsi_eh_1, PPid: 2, Pid: 1324, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 9 Process Name: scsi_tmf_1, PPid: 2, Pid: 1325, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 10 Process Name: scsi_eh_2, PPid: 2, Pid: 1328, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 11 Process Name: scsi_tmf_2, PPid: 2, Pid: 1329, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 12 Process Name: scsi_eh_3, PPid: 2, Pid: 1332, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 13 Process Name: scsi_tmf_3, PPid: 2, Pid: 1333, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 14 Process Name: scsi_eh_4, PPid: 2, Pid: 1336, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 15 Process Name: scsi_tmf_4, PPid: 2, Pid: 1337, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 16 Process Name: scsi_eh_5, PPid: 2, Pid: 1340, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 17 Process Name: scsi_tmf_5, PPid: 2, Pid: 1341, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 18 Process Name: ipv6_addrconf, PPid: 2, Pid: 1393, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 19 Process Name: deferwq, PPid: 2, Pid: 1405, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 20 Process Name: kjournald, PPid: 2, Pid: 1522, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:180 LogProcessMemory(): ***** Process Memory Usage List End *****

 

 

RECENT process memory usage log output

 

(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:172 LogProcessMemory(): ***** Process Memory Usage List Start *****
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 1 Process Name: clusterd, PPid: 1, Pid: 3607, Memory Used: 12150252 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 2 Process Name: kworker/2:1, PPid: 2, Pid: 21249, Memory Used: 4459736 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 3 Process Name: qemu-kvm, PPid: 1, Pid: 20641, Memory Used: 4459736 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 4 Process Name: kvm-pit/20641, PPid: 2, Pid: 20642, Memory Used: 4459736 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 5 Process Name: dhcpd, PPid: 1, Pid: 29305, Memory Used: 3239068 kB, Thread Count: 1
[5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 6 Process Name: kworker/0:1H, PPid: 2, Pid: 29662, Memory Used: 3239068 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 7 Process Name: scsi_tmf_0, PPid: 2, Pid: 1321, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 8 Process Name: scsi_eh_1, PPid: 2, Pid: 1324, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 9 Process Name: scsi_tmf_1, PPid: 2, Pid: 1325, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 10 Process Name: scsi_eh_2, PPid: 2, Pid: 1328, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 11 Process Name: scsi_tmf_2, PPid: 2, Pid: 1329, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 12 Process Name: scsi_eh_3, PPid: 2, Pid: 1332, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 13 Process Name: scsi_tmf_3, PPid: 2, Pid: 1333, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 14 Process Name: scsi_eh_4, PPid: 2, Pid: 1336, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 15 Process Name: scsi_tmf_4, PPid: 2, Pid: 1337, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 16 Process Name: scsi_eh_5, PPid: 2, Pid: 1340, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 17 Process Name: scsi_tmf_5, PPid: 2, Pid: 1341, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 18 Process Name: ipv6_addrconf, PPid: 2, Pid: 1393, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 19 Process Name: deferwq, PPid: 2, Pid: 1405, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:178 LogProcessMemory(): 20 Process Name: kjournald, PPid: 2, Pid: 1522, Memory Used: 3084408 kB, Thread Count: 1
(5914 /infoblox/one/bin/monitor) CMemoryMonitor.c:180 LogProcessMemory(): ***** Process Memory Usage List End *****

Re: ISSUE : System swap space usage exceeds the critical threshold value

Authority
Posts: 19
1266     0

Hi Sachin,

 

Since there can be multpe factors behind the clusterd process utilizing more swap, this looks like something Support can help you with. I have linked KB below from the support site that you may find relevant:

 

System swap space usage exceeds the critical threshold value" error, things to know and data to be c...

 

Hope this helps.

 

 

Highlighted

Re: ISSUE : System swap space usage exceeds the critical threshold value

Authority
Posts: 29
1266     0

Thanks Nedward for your input.

 

I will go through the KB article and see if that helps.

 

Thanks,

 

Sachin

Showing results for 
Search instead for 
Do you mean 

Recommended for You