slabtop command (part of the package procps) shows top memory objects used by the kernel.
dstat can help you figure out what is happening. dstat -cdnpmgs --top-bio --top-cpu --top-mem
Also have a look at smem ("smem -kt"), it can show you nicely what is in your swap.
A. Panagiotidis, S. Frey, and T. Ertl. Eurographics Conference on Visualization (EuroVis) - Short Papers, page 13-17. The Eurographics Association, (2015)
A. Cheik Ahamed, and F. Magoulès. High Performance Computing and Communication 2012 IEEE 9th International Conference on Embedded Software and Systems (HPCC-ICESS), 2012 IEEE 14th International Conference on, page 1307-1314. (June 2012)