h1

In-the-Lab: NexentaStor vs ESXi, Redux

February 24, 2012

In my last post, I mentioned a much complained about “idle” CPU utilization quirk with NexentaStor when running as a virtual machine. After reading many supposed remedies on forum postings (some reference in the last blog, none worked) I went pit-bull on the problem… and got lucky.

As an avid (er, frequent) NexentaStor user, the luster of the NMV (Nexenta’s Web GUI) has worn off. Nearly 100% of my day-to-day operations are on the command line and/or Nexenta’s CLI (dubbed NMC). This process includes power-off events (from NMC, issue “setup appliance power-off” or “setup appliance reboot”).

For me, the problem cropped-up while running storage benchmarks on some virtual storage appliances for a client. These VSA’s are bound to a dedicated LSI 9211-8i SAS/6G controller using VMware’s PCI pass-through (Host Configuration, Hardware, Advanced Settings). The VSA uses the LSI controller to access SAS/6G disks and SSDs in a connected JBOD – this approach allows for many permutations on storage HA and avoids physical RDMs and VMDKs. Using a JBOD allows for attachments to PCIe-equipped blades, dense rack servers, etc. and has little impact on VM CPU utilization (in theory).

So I was very surprised to find idle CPU utilization (according to ESXi’s performance charting) hovering around 50% from a fresh installation. This runs contrary to my experience with NexentaStor, but I’ve seen reports of such issues on the forums and even on my own blog. I’ve never been able to reproduce more than a 15-20% per vCPU bias between what’s reported in the VM and what ESXi/vCenter sees. I’ve always attributed this difference to vSMP and virtual hardware (disk activity) which is not seen by the OS but is handled by the VMM.

CPU record of idle and IOzone testing of SAS-attached VSA

During the testing phase, I’m primarily looking at the disk throughput, but I notice a persistent CPU utilization of 50% – even when idle. Regardless, the 4 vCPU VSA appears to perform well (about 725MB/sec 2-process throughput on initial write) despite the CPU deficit (3 vCPU test pictured above, about 600MB/sec write). However, after writing my last blog entry, the 50% CPU leach just kept bothering me.

After wasting several hours researching and tweaking with very little (positive) effect, a client e-mail prompted a NMV walk through with resulted in an unexpected consequence: the act of powering-off the VSA from web GUI (NMV) resulted is significantly reduced idle CPU utilization.

Getting lucky: noticing a trend after using NMV to reboot for a client walk-through of the GUI.

Working with the 3 vCPU VSA over the previous several hours, I had consistently used the NMC (CLI) to reboot and power-off the VM. The fact of simply using the NMV to shutdown the VSA couldn’t have anything to do with idle CPU consumption, could it? Remembering that these were fresh installations I wondered if this was specific to a fresh installation or could it show up in an upgrade. According to the forums, this only hampered VMs, not hardware.

I grabbed a NexentaStor 3.1.0 VM out of the library (one that had been progressively upgraded from 3.0.1) and set about the upgrade process. The result was unexpected: no difference in idle CPU from the previous version; this problem was NOT specific to 3.1.2, but specific to the installation/setup process itself (at least that was the prevailing hypothesis.)

Regression into my legacy VSA library, upgrading from 3.1.1 to 3.1.2 to check if the problem follows the NexentaStor version.

If anything, the upgraded VSA exhibited slightly less idle CPU utilization than its previous version. Noteworthy, however, was the extremely high CPU utilization as the VSA sat waiting for a yes/no response (NMC/CLI) to the “would you like to reboot now” question at the end of the upgrade process (see chart above). Once “no” was selected, CPU dropped immediately to normal levels.

Now it seemed apparent that perhaps an vestige of the web-based setup process (completed by a series of “wizard” pages) must be lingering around (much like the yes/no CPU glutton.) Fortunately, I had another freshly installed VSA to test with – exactly configured and processed as the first one. I fired-up the NMV and shutdown the VSA…

Confirming the impact of the "fix" on a second fresh installed NexentaStor VSA

After powering-on the VM from the vSphere Client it was obvious. This VSA had been running idle for some time, so it’s idle performance baseline – established prior across several reboots from CLI – was well recorded by the ESXi host (see above.) The resulting drop in idle CPU was nothing short of astounding: the 3 vCPU configuration has dropped from a 50% average utilization to 23% idle utilization. Naturally, these findings (still anecdotal) have been forwarded on to engineers at Nexenta. Unfortunately, now I have to go back and re-run my storage benchmarks; hopefully clearing the underlying bug has reduced the needed vCPU count…

7 comments

  1. This is very interesting work Collin. For what it’s worth, with 1 vCPU (as per previous discussions on here) my Nexenta CE 3.0.4 VSA on ESXi 4.1.0 is idling at ~4% CPU with little spikes every 5-10 minutes, so we can see what should be possible.

    I do worry the VSA is neglected by Nexenta – whilst the VSA does present a chicken/egg issue, it is a neat way of making better use of cores in a small cluster (rather than dedicated hardware).

    Like


    • The chicken/egg dilemma would be a good subject for refinement.

      The current crop of VSA’s I’ve been working with are “glued” to hardware, if you will, by the presence of a pass-through LSI SAS controller (either on motherboard or PCIe card). I make use of the Host Virtual Machine Startup/Shutdown feature of ESXi to insure that the VSA is the first to power-on and delay any dependent resources. This works extremely well provided NFS is used as the storage type provided by the VSA.

      Where chicken/egg issues abound is in the attempt to use iSCSI as the storage type. ESXi is not very forgiving during VSA reboots or it’s delayed presence at host boot time. This makes for either a tremendous amount of tuning of the ESXi host or scripting from the VSA to force the host to “rescan” its iSCSI storage once it’s available. Again, no big deal, but something to consider.

      I find that VSAs with short reboot times while providing NFS storage can do so while most dependent virtual workloads are still running. ESXi is forgiving for the most parts – in some cases, IO timers in the VM’s OS may need to be extended (depending on the workload). All of this is turning into “secret sauce” stuff for a lot of “vendors” right now, but it should be more openly discussed and appreciated.

      I too worry about Nexenta’s neglect of the VSA, although it’s nice to see a refreshed VSA with their 3.1.2 release. It’s a shame that I missed it or I’d have tested it along with the “install from disc” version. The historical lack of VMXNET3 is both a short sighted one and a critical misstep on Nexenta’s part (read: eschewing free 10G-like performance in embedded VSA applications).

      Imagine a single 2U 4-node, 2P box (i.e. Supermicro/MDS) in a SME with 2+ VSAs supporting a 24-disk mix of SSDs and “cheap rust” capable of handling 150-300 virtual desktops, 75-100 virtual servers (HA) and primary storage (I heard you cringe) too – and JBOD expansion options. That’s a decent ROI lead, if not a good idea 😉

      Like


      • That’s an interesting comparison of iSCSI vs NFS which I hadn’t considered.

        What I was thinking about with chicken/egg was that ideally I’d have a compute layer and a storage layer. My hosts would boot directly from the storage layer (i.e. no local storage) or else a local SDcard (e.g. very common these days for vSphere). Trouble is the VSA needs its system image somewhere that isn’t in the storage layer, e.g. separate local disks.

        That said, in a non-SAN environment the VSA is going to have to be connected to host-specific PCI cards (& direct attached array(s)), or RAID controllers (& and on-board disks) so maybe it can never be part of my normal compute layer (as you can’t take advantage of being able to easily move it between hosts). Which sort of implies that, providing you’re not licensed by processor core (i.e. like Nexenta), maybe you should only have a physical storage appliance anyway.

        In my case I use a Nexenta VSA to provide NFS shares to run shared Oracle homes which does seem to work quite nicely.

        Like


  2. […] has been debated for a while – certainly a single vCPU worked better for me. There is also a long running issue with idle vCPU usage which is hopefully going to be resolved soon (this bugmay be the same […]

    Like


  3. Interesting post. It would be nice to see a parallel post using EON
    http://sites.google.com/site/eonstorage/begin
    http://eonstorage.blogspot.com/

    Like


  4. Pardon me for adding “useless” information to a long posted topic, but I couldn’t help but chuckle when I read this post.

    Let me explain:

    Long ago (in IT terms, so really not that long..) I was setting up multiple Windows 2003 VM’s in a cluster, and flipping from one to the other in succession modifying options as needed after the Kickstart deploy of the base OS and settings.

    I was also training a customer at the same time about the ins/outs of VirtualCenter, and we flipped to the “Performance” tab and I started to explain….

    Imagine my surprise when 3 of the hosts in the cluster had close to 70% CPU Utilization, while the 4th was hovering around 3-5%!! I looked around, all hosts had the same number of VM’s (and essentially the exact same type/settings/OS at this point) but the cluster was getting beat up CPU wise.

    What I didn’t know is: on each Host that had high CPU utilization, there were at least 2 VM’s that had “Yes/No/Continue” prompts for the software package the customer was deploying!!

    Once we answered the question…WALA! CPU dropped and everything went on as expected.

    Hilarious how many places I looked to try to figure this out, and how simple the solution ended up being!

    Thanks!

    smooter

    Like


    • Smooter,

      Certainly another example of userland code making for unusually high host CPU utilization, thanks. I would expect to see high guest CPU utilization in such cases, and you would most likely have seen the same guest utilization during that application install in a direct-to-hardware install if live monitoring we typical in legacy environments (otherwise, it goes unnoticed).

      The key issue in this article is the gross disparity between the guest and host utilization. In these cases, the guest shows 0-2% CPU utilization where the host shows 30-50% CPU utilization. While it appears that some of the disparity can be apportioned to virtual device poling/activity, such a wide differential is not common among typical virtual workloads.

      Cheers,

      Collin

      Like



Comments are closed.

%d bloggers like this: