Adding Live Memory Load

So I said I wanted to add the number of requests per hour next to the top of the site so people could see its load, but Gavin pinged me on Twitter and asked about showing memory usage. I had thought about memory, so I decided to tackle it. In addition to getting the amount of used system memory, I decided to also show heap allocation and usage as well.

Getting System Memory

I added the data collection into the same cron job that's collecting the CPU usage every minute. This bit was a little trickier since I couldn't just do it all in a one-liner. The first step was getting the system memory, which was pretty straightforward. Using the "free -m" command, I grep out the line with the memory details and then use gawk to extract the two columns I want and print them separately. Since I'm not doing a percentage-based chart, I need both values. The total system RAM will be the max value of the Y axis and the used amount will be the data points for that series. These get set into a variable.

# Get current system memory max and usage, in that order
mem_system=`free -m | grep Mem: | gawk '{print $2,$3}'`

Getting JVM Memory

The first step is to get the pid of the java process. Now, I could have more than one java process running (or none at all), so I sort the batch mode output of "top" desc by memory, filter out the java ones, grab the first one with "head" and gawk out the first column which is pid. It looks ugly, but it works.

# Get current JVM heap max and usage, in that order
# Start by getting the pid of the largest java process (memory-wise)
java_pid=`top -b -n 1|awk '{if ($1~/[0-9]+/) {if ($5~/m/) {$5=int($5*1024*1024)};print}}'|sort -k5rn,5 | grep java | head -n1 | gawk '{print $1}'`

Now, check and see if I got a pid. If there was no process running called "java", the $java_pid variable will be empty. To get the JVM heap stats, I'm using jstat. "tail" strips off the headers and "gawk' adds together all of the "allocated" and "used" memory spaces to give me two totalled numbers.

# Check and make sure we got a pid
if [ "$java_pid" = "" ]
then
        mem_heap="0 0"
else
        # Get the actual heap usage
        # http://docs.oracle.com/javase/6/docs/technotes/tools/share/jstat.html
        mem_heap=`jstat -gc $java_pid | tail -n 1 | gawk '{print ($1+$2+$5+$7+$9)/1000,($3+$4+$6+$8+$10)/1000}'`
fi

Storing The Data

Now we have all the pieces, so concat the variables and append them onto our log file with the usual 60-line rotation.

# Output them into the file.
# systemMax systemUsed JVMAllocated JVMUsed
echo "$mem_system $mem_heap" >> ~/mem_stats.txt

# Keep last 60 lines
cat ~/mem_stats.txt > ~/_mem_stats.txt
cat ~/_mem_stats.txt | tail -n 60 > ~/mem_stats.txt
rm ~/_mem_stats.txt

This data looks something like this:

927 738 277.504 220.125
927 738 277.504 178.277
927 739 277.504 178.388
927 739 277.504 178.463
927 739 277.504 198.341

Charting It

I used another Flot chart, this time with the "stacked" plugin which required a new js file be added to my site's header. I created a second ContentBox widget to display this chart. I love how ContentBox lets me create widgets on-the-fly and actually edit the CFML code from the admin interface. I can create widgets in no time and drop them right into the site. The preparation is similar to the CPU, but I have to parse all 4 data points out of the sample and build separate chart series. I removed the formatting options to reduce the amount of code here.

var stats = fileRead( '/root/mem_stats.txt' );
stats = listToArray( stats, chr(10) );
var system_max = 1000;
var system_used = [];
var heap_max = [];
var heap_used = [];	
var series = [];
var i = 0;
for ( var sample in stats ) {
	i++;
	sample = listToArray( sample, ' ' );
	system_max = sample[1];
	system_used.append( [ i, sample[2]-sample[3] ]  );
	heap_max.append( [ i, sample[3]-sample[4] ]  );
	heap_used.append( [ i, sample[4] ]  );
}
series.append( { 'label': 'Heap Used', 'data': heap_used, 'color': 'white' } );
series.append( { 'label': 'Heap Allocated', 'data': heap_max, 'color': 'red' } );
series.append( { 'label': 'System Used', 'data': system_used, 'color': 'rgba(0, 255, 0, 1)' } );

savecontent variable="local.results" {

writeOutput( "
		
	<div style='display:inline-block'>
		<div style='width:300px;text-align:center'>Memory Usage for last hour</div>
		<div id='memplaceholder' style='width:300px;height:75px'></div>
		<div id='placeholder_legend' style='display:inline-block;float: right;'></div>
	</div>
	
	<script language='javascript'>
		$( document ).ready(function() {
			$.plot(
				$('##memplaceholder'),
				#serializeJSON( series )#,
				{} // options removed
			);
		});
	</script>
		
");

}
return local.results;

I left the legend in and needed some extra formatting to keep it all lined up. Displaying the widget just looks like this:

#cb.widget("mem-load")#

And produces a nice chart like so:

You can see the live memory chart at the top of this page next to the CPU usage chart. Like before, data is updated every minute, but ContentBox's content caching may show the same data for up to 5 minutes.