These days, most companies have at least begun to virtualize their systems. One of the big advantages of virtualizing a server is that we can take a “snapshot” of the box, attempt some questionable update, and if anything goes wrong, we can just “revert to snapshot”. Lots of administrators end up with snapshot-sprawl, with every snapshot causing them to go deeper and deeper in time and space – time it will take to collapse the snapshots back, and space consumed by the deltas until you do the collapse.

And when we snapshot we get not only the server drives, but there is a checkbox to “snapshot the virtual machine’s memory”, too, and by default it’s checked.

I almost always de-select the checkbox, partly because of all the space it wastes, but also because of a little security hole that may be a serious risk.

In 2005, an obscure paper was published by some Stanford University researchers, claiming that a server’s certificate and private key – something we pay a thousand dollars a year for, and needs to be kept private – sit in clear text in the RAM of the server. I found out about it when another bunch of researchers published an article in the New York Times in 2007. They had a photo of a frozen memory DIMM, and said that if they could get the memory out of the box, they’d have the private key in clear text 1’s and 0’s, but if they stole the memory chip of course the electricity would be disconnected, and the 1’s and 0’s would fall away. So they tried spraying the memory DIMM first with a can of compressed air, to freeze it, then walked away with the chip. They found they had several hours to get to the data, so they had the time they needed, to break out their pattern-recognition algorithm, to figure out which 1’s and 0’s they needed, and they had their private key.

But while reading that, I was just learning the benefits of virtualization with VMWare ESX and my ability to do snapshots, and I knew that if I snapshot the system memory too, I get a separate file on the VMFS-3 partition. Though most administrators don’t have the pattern-recognition algorithm from the article, I’ve always thought of that memory snapshot file as a potential risk.

Charlie Messemer
CNS