How to find what is using linux swap or what is in the swap?
I have virtual linux (Fedora 17) server with 28GB RAM and 2GB swap. The server is running mysql db that is set up to use most of the RAM.
After some time running the server starts to use swap to swap out unsued pages. That is fine as my swappiness is at default 60 and it is the expected behaviour.
The strange thing is that the number in top/meminfo does not correspond with info from processes. I.e. the server is reporting these numbers:
If I use the script from http://serverfault.com/a/423603/98204 it reports reasonable numbers (few MBs swapped by bash'es, systemd, etc) and one big allocation from mysql: (I ommited a lot of output lines)
So if I get the script output right the total swap usage should be 449264K=cca 440MB with mysql using cca 90% of the swap.
The question is why this differs so much from the top and meminfo numbers? Is there any way how to "dump" swap info to see what is actually in it instead of summing the swap usages from all processes?
When analyzing the issue I came up with different ideas but they all seems to be wrong:
- The script output is not in KB. Even if it would be in 512 or 4KB units it won't match. Actually the ratio (1200:440)
is about 3:1 which is "strange" number.
- There are some pages in swap that are somehow shared between processes as mentioned in http://serverfault.com/a/477664/98204. If this is true how can I find the actual number of memory used like this? I mean it would need to make cca 800MB difference. And that does not sound right in this scenario.
- There are some "old" pages in swap used by processes that already finished. I would not mind that if I were able to find out how much is this "freeable" swap.
- There are pages in swap that has been swapped back to memory and are in swap just in case they did not change in RAM and need to be swapped out again as mentioned in http://serverfault.com/a/100636/98204. But the SwapCached value is only 24MB.
The strange thing is that the swap usage is slowly increasing while the sum output from the script is roughly the same. In last 3 days the swap used increased from 1100MB to current 1230MB while the sum increased from 430MB to current 449MB (cca).
The server has enoough free(able) RAM so I could just turn off the swap and turn it back on.Or I could probably set swappiness to 0 so the swap would get used only if the is no other way. But I would like to solve the issue or at least find out what is the cause of this.