VMmark – comments

VMware recently announced availability of VMmark – the first virtualization benchmark that measures… well, some characteristics of workloads running in virtual machines.

Overall, VMmark is a step in the right direction. However, it has a long way to become a solid benchmark. Here are some of the issues:

• Virtualization platform scalability is not measured – The largest VM in the tile is 2CPU/2GB SQL Server VM, which is much smaller than the physical server itself. VMware ESX supports a VM with 4 virtual CPUs and 16GB of memory, yet – why??? – these capabilities are not used in the test.
  
• Workload mix is not realistic – There will always be different virtualized workloads running on the same physical machine, but all VMmark workloads are quite different. Such a mix could be more realistic for small/medium business consolidation scenario, but not the typical enterprise datacenter deployment.
  
• Performance of individual workloads is not measured – with such a wild workload mix, it is difficult to find out how well a server is suited for consolidating several instances of a certain workload – such as Exchange Server, SQL Server or file server.
  
• Mixing platforms is unnecessary – Mixing platforms on the same machine prevents running VMmark using Virtuozzo – not that I suggest it was done on purpose J In real life, enterprises typically run Windows and Linux on different boxes, and majority of the consolidated workloads are Windows-based.
  
• Aggregate score has no physical meaning - All workloads in the tile are throttled and never run at the full capacity. And, if the server is powerful enough that a single tile cannot load it fully, you are supposed to add another, which would double the number of VMs and probably skew the results.
  

A better approach would be a more traditional methodology that runs workloads serially:

For each workload, measure maximum aggregate performance with 1, 2, 4, 8, 16 and 32 virtual environments. Normalize against the same workload on the same hardware without virtualization and average the results.

The benefits of this approach:

• A single VE test will measure scalability and performance of the virtualization platform – ability to use multiple CPUs, large memory, fast storage subsystems and network interfaces.
  
• 16/32 VE tests will measure realistic density limits for consolidating specific workloads on the tested virtualization platform.
  
• It will measure for each workload how virtualization overhead grows with density.
  
• Finally, Virtuozzo will be able to run the benchmark because only one OS is used for a single benchmark run.
  

Let me know what you think about pros and cons of each approach.