h1

In-the-Lab: Install VMware Tools on NexentaStor VSA

June 17, 2010

Physical lab resources can be a challenge to “free-up” just to test a potential storage appliance. With NexentaStor, you can download a pre-configured VMware (or Xen) appliance from NexentaStor.Org, but what if you want to build your own? Here’s a little help on the subject:

  1. Download the ISO from NexentaStor.Org (see link above);
  2. Create a VMware virtual machine:
    1. 2 vCPU
    2. 4GB RAM (leaves about 3GB for ARC);
    3. CD-ROM (mapped to the ISO image);
    4. One (optionally two if you want to simulate the OS mirror) 4GB, thin provisioned SCSI disks (LSI Logic Parallel);
    5. Guest Operating System type: Sun Solaris 10 (64-bit)
    6. One E1000 for Management/NAS
    7. (optional) One E1000 for iSCSI
  3. Streamline the guest by disabling unnecessary components:
    1. floppy disk
    2. floppy controller (remove from BIOS)
    3. primary IDE controller (remove from BIOS)
    4. COM ports (remove from BIOS)
    5. Parallel ports (remove from BIOS)
  4. Boot to ISO and install NexentaStor CE
    1. (optionally) choose second disk as OS mirror during install
  5. Register your installation with Nexenta
    1. http://www.nexenta.com/register-eval
    2. (optional) Select “Solori” as the partner
  6. Complete initial WebGUI configuration wizard
    1. If you will join it to a domain, use the domain FQDN (i.e. microsoft.msft)
    2. If you choose “Optimize I/O performance…” remember to re-enable ZFS intent logging under Settings>Preferences>System
      1. Sys_zil_disable = No
  7. Shutdown the VSA
    1. Settings>Appliance>PowerOff
  8. Re-direcect the CD-ROM
    1. Connect to Client Device
  9. Power-on the VSA and install VMware Tools
    1. login as admin
      1. assume root shell with “su” and root password
    2. From vSphere Client, initiate the VMware Tools install
    3. cd /tmp
      1. untar VMware Tools with “tar zxvf  /media/VMware\ Tools/vmware-solaris-tools.tar.gz”
    4. cd to /tmp/vmware-tools-distrib
      1. install VMware Tools with “./vmware-install.pl”
      2. Answer with defaults during install
    5. Check that VMware Tools shows and OK status
      1. IP address(es) of interfaces should now be registered

        VMware Tools are registered.

  10. Perform a test “Shutdown” of your VSA
    1. From the vSphere Client, issue VM>Power>Shutdown Guest

      System shutting down from VMware Tools request.

    2. Restart the VSA…

      VSA restarting in vSphere

Now VMware Tools has been installed and you’re ready to add more virtual disks and build ZFS storage pools. If you get a warning about HGFS not loading properly at boot time:

HGFS module mismatch warning.

it is not usually a big deal, but the VMware Host-Guest File System (HGFS) has been known to cause issues in some installations. SInce the NexentaStor appliance is not a general purpose operating system, you should customize the install to not use HGFS at all. To disable it, perform the following:

  1. Edit “/kernel/drv/vmhgfs.conf”
    1. Change:     name=”vmhgfs” parent=”pseudo” instance=0;
    2. To:     #name=”vmhgfs” parent=”pseudo” instance=0;
  2. Re-boot the VSA

Upon reboot, there will be no complaint about the offending HGFS module. Remember that, after updating VMware Tools at a future date, the HGFS configuration file will need to be adjusted again. By the way, this process works just as well on the NexentaStor Commercial edition, however you might want to check with technical support prior to making such changes to a licensed/supported deployment.

25 comments

  1. Thanks for excellent instructions!

    Like


  2. TIP: In ESXi 4.0 and I’m pretty sure in almost every VMware product, do NOT disable the IDE controller (just yet). Wait until you’re done installing NS en VM Tools. Since the VM BIOS is programmed by VMware there’s a slight glitch. If you disable the IDE controllers with an IDE CD-ROM drive connected to it. You’ll still be able to select ‘CD-DRIVE’ at the boot-device-selector thingie (ESC while booting). This is misleading since it’ll actually never work because you disabled IDE. In a actual machine the option wouldn’t show up. VM forgot to implement this.

    Like


  3. before you can untar vmtools, you need to enable expert mode and drop into a real bash shell (step 9)

    # option expert_mode=1 -s
    # !bash

    Like


    • Not necessary. In step (9) I suggest logging-in as “admin” – this user is not a restricted “nmc” shell like the “root” user – so there is no “option expert_mode” nor need for additional “bash-like” shell.

    • 9. Power-on the VSA and install VMware Tools
      1. login as admin
        1. assume root shell with “su” and root password
      2. From vSphere Client, initiate the VMware Tools install
      3. cd /tmp
        1. untar VMware Tools with “tar zxvf  /media/VMware\ Tools/vmware-solaris-tools.tar.gz”
    • If you follow my suggestion in step 9.1, you’ll see for yourself…

      Like


  4. Hey,
    I was wondering if you have had any problems running NexentaStor on ESXi ?
    I’m struggling with high cpu usage on the host. If I go to the ESXi performance tab for my nexenta vm it shows my cpus hovering from 20-40% usage. If I ssh into the vm and do a prstat I don’t see anything using any cpu at all. I have tried installing the vmware-tools as you described in the post, but it doesn’t seem to change anything. So, I was just wondering if you have ran into the same problems ?

    Thanks,
    Raul

    Like


    • Raul:

      Your mileage will vary depending on the processor. Anything Nehalem/i7, Opteron (Shanghai or better), Phenom II/X4/X6 with RVI (or equivalent) should be fine. Older Intel CPU’s are not really usable outside the lab space.

      If you’re using compression and/or deduplication expect to see higher average CPU utilization. Without L2ARC I wouldn’t recommend the use of deduplication. Also, the zpool processes will consume more CPU with these features enabled. In VMware the disk emulation type will have some affect as well, plus the ZFS tuning parameters will impact CPU utilization.

      With AMD 4/6-core Opteron, NexentaStor running in a 64-bit VM with 4GB RAM and RVI enabled, virtual quad-core (1 vCPU, 4 vCores) and VMware tools, I see accurate CPU utilization in ESXi. If you’re using the VM as a “production” VSA, make sure you have the correct resource reservations in VMware to reduce/eliminate VM swap and CPU contention, etc.

      Like


      • I’m running a Xeon X3440 which has Vt-x and EPT. My nexenta Vm isn’t using compression or de-dup. I just have it sitting at ~99% idle. I’ve even tried destroying and recreating the VM but the host still reports ~30 cpu usage for the VM. My Linux/Windows VMs all idle < 50Mhz. I dunno if it's something on ESX that's configured wrong, or if something on the VM is using CPU that's not reported by mpstat.

        Any ideas ?

        Thanks 🙂

        Like


      • Raul:

        Could it be the IDE interface polling for your CD-ROM? Just to check, make sure your NexentaStor VM’s CD-ROM is configured in vSphere as “Device Type = Client Device” and see if that has any effect. I’ve seen reports that CPU utilization in VMware is abnormally high with “Device Type = Host Device.”. Also, this article indicates – in step 2 – that streamlining the VM is an important step, and with reason, so reduce that virtual hardware footprint as well. My lab VM shows about 8 percent CPU utilization at idle viewed from vCenter’s Performance tab.

        Like


      • Also noteworthy, is whether or not services are in demand from the VSA. For instance, with a web browser open to the NMV (web GUI) General Status page, the average utilization is about 3% higher than “true idle” – i.e. 5 percent versus 8 percent. With active shares, remember that the Ethernet port(s) and SCSI interface(s) are emulated in software and this emulation shows-up in the vSphere Client’s CPU “Performance” statistics but not within the VM’s OS metering. If your VSA is “busy” moving data across your network, this could represent a portion of your “hidden” CPU utilization.

        Like


      • I tried everything you suggested but nothing made a difference. I decided to boot up my VM in single user mode and monitor the CPU usage. To my surprise it was idling at < 1% CPU usage. So then I put it into init 2. It still was idling at < 1% CPU. Then I put it into init 3 and the CPU usage on the host shot up to 40%. So I did a svcs -a in init 3 and init 2 to see which services got started when I switched to init 3. I switched back init 3 and started to disable services one at a time and monitor the CPU usage on the host. Once I finally disabled the nmdtrace service my CPU usage shot down to < 1%!! I started up all my services except nmdtrace, nmv and nms and my box is idling like it should. It seems to be that something in nmdtrace is probing something that the host doesn't like. I might just leave it disabled for a while because I don't actually need to use the web gui for anything.

        Thanks for the help!
        Raul

        Like


  5. Thanks for this post Collin – the NexentaStor VA is a little older (3.0.0) so I installed my own (3.0.4) and it was handy to have your notes for reassurance. Note for others: installing VMtools from ESXi 4.1U1 (build 348481) I didn’t get any HGFS complaints.

    Raul: do you still have nmdtrace, nmv and nms disabled? My 2 vCPU VM (NMV HTTPS on) idles at 40% busy too so I think I need to some something (either switch to 1 vCPU or switch off nm…).

    Like


    • The actual culprit is nmv. I’m idling around 3% cpu now with 4 vCPUs. If i kill nmdtrace and nms I’m down to <1% but I figure 3% is way better then 40%.

      Like


  6. I’ve just switched my Nexenta CE VM from 2 vCPUs to 1 (Intel 5400 series) and according to vSphere Client the CPU when idling has dropped from about 40% of 2 vCPUs to about 13-15% of 1 vCPU (or 1700MHz down to 250MHz if you prefer):

    Like


    • I’ve seen the same behaviour when experimenten with CPU count. I have a dual dual opteron and for me 2CPUs worked best although 1 CPU gave better idle consumption. I’ve tried anything from 4 (don’t do this, pispoor ESXi performance) to 1.

      Like


      • You have a point depending on the workload profile. Regression testing shows a disproportionate amount of CPU at idle when going from one to two vCPUs. When going from 2 to 4 vCPUs, it’s necessary to do some workload testing to see if the additional CPU utilization cannibalized from the system results in a commensurate increase in storage IO and bandwidth performance. Since ZFS is a heavily threaded system, the additional vCores CAN improve overall storage performance depending on the application. As always, test the use case – there are no “generic” configurations here…

        Like


  7. After seeing the above post I thought I’d post my findings for what it’s worth.
    Installed V3.1.0 community edition within Sun (oracle) VirtualBox to play with. Experiencing 25-35% cpu usage while idle no iscsi or volumes configured ie its a clean install.
    Disabled svc nmdtrace and cpu dropped to around 6% idle.
    This VM is running as single cpu and not using https.
    So it appears this issue is not only manifesting itself within vmware.

    Like


    • @SRG: thanks for posting – what is interesting with your experience is that using a single vCPU and not HTTPS didn’t help.

      I worry that the VSA isn’t a common deployment approach (compared to a low cost server with big JBOD) so maybe it’s not a priority for the development team. That said HP are doing interesting stuff with their LeftHand acquisition / P4000 stuff and given that even the smallest servers are getting very fast maybe virtual appliances have yet to get the limelight.

      Like


    • That’s the storage emulation layer rearing it’s ugly head. Still, 30% is high; is that a SATA storage controller? I never see more than 15% with SAS or iSCSI.

      Like


      • I have mine using the PV RDM and I still get 40% cpu usage. I didn’t think the RDM stuff went through the storage emulation layer. I wonder if I was to do a pass through SATA controller if the cpu usage is still high at that point. Any one want to try it ?

        Like


      • Just thought I would update. I just did a fresh install of 3.1.2 and the CPU usage is good. Once you open up NMV and it shows the usage monitors the CPU spikes up to 50% and stays there. Seems to be like dtrace is causing ESXi to get angry. Just thought I would share.

        Like


  8. […] This establishes the VM’s relationship to a specific VMware Tools load. Installation of VMware Tools in NexentaStor is covered in detail in an earlier blog entry. VMware Tools bombs-out at SUNWuiu8 package failure. Illumos-based NexentaStor has no such […]

    Like


  9. […] Install VMware Tools on NexentaStor VSA ; […]

    Like


  10. […] network configuration so you can get NexentaStor set up, then add the better vmxnet3 NICs later.  Here’s a good how-to about installing the tools on NexentaStor, and also a workaround for a problem you’ll run into regarding running […]

    Like


  11. […] network configuration so you can get NexentaStor set up, then add the better vmxnet3 NICs later.  Here’s a good how-to about installing the tools on NexentaStor, and also a workaround for a problem you’ll run into regarding running […]

    Like



Comments are closed.

%d bloggers like this: