Posts Tagged ‘esxi’

h1

New Security Patches for ESX/ESXi 3.5, 4.0, 4.1 and 5.0 (6/14/2012)

June 14, 2012

New patches are available today for ESX/ESXi 3.5, 4.0, 4.1 and 5.0 to resolve a few known security vulnerabilities. Here’s the run down if you’re running ESXi 5.0 standard image:

VMware Host Checkpoint File Memory Corruption

Certain input data is not properly validated when loading checkpoint files. This might allow an attacker with the ability to load a specially crafted checkpoint file to execute arbitrary code on the host. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3288 to this issue.

The following workarounds and mitigating controls might be available to remove the potential for exploiting the issue and to reduce the exposure that the issue poses.

Workaround: None identified.

Mitigation: Do not import virtual machines from untrusted sources.

VMware Virtual Machine Remote Device Denial of Service

A device (for example CD-ROM or keyboard) that is available to a virtual machine while physically connected to a system that does not run the virtual machine is referred to as a remote device. Traffic coming from remote virtual devices is incorrectly handled. This might allow an attacker who is capable of manipulating the traffic from a remote virtual device to crash the virtual machine.
The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3289 to this issue.

The following workarounds and mitigating controls might be available to remove the potential for exploiting the issue and to reduce the exposure that the issue poses.

Workaround: None identified.

Mitigation: Users need administrative privileges on the virtual machine in order to attach remote devices. Do not attach untrusted remote devices to a virtual machine.

Deployment Considerations

None beyond the required patch bundles and reboot information listed in the table above.

5.0 Patch Release Notes:
ESXi: http://kb.vmware.com/kb/2021031

Patch Release Information for Other Versions

3.5 Patch Release Notes:
ESX: http://kb.vmware.com/kb/2021020
ESXi: http://kb.vmware.com/kb/2021021

4.0 Patch Release Notes:
ESX: http://kb.vmware.com/kb/2021025
ESXi: http://kb.vmware.com/kb/2021027

4.1 Patch Release Notes:
ESX: http://kb.vmware.com/kb/2019065
ESXi: http://kb.vmware.com/kb/2019243

h1

Quick Take: Syslog Stops Working after Upgrade to ESXi 5.0 Update 1

March 24, 2012

If you’ve recently upgraded your ESXi from 5.0 build 456551 and were logging to syslog, it’s possible that your events are no longer being received by your syslog server. It seems that there was a “feature” in ESXi 5.0 build 456551 that allowed syslog to escape the ESXi firewall regardless of the firewall setting. This could be especially problematic if your upgraded from ESXi 4.x where there was no firewall configuration needed for syslog traffic.

VMware notes that syslog traffic was not affected by the ESXi firewall in v5 build 456551. See KB2003322 for details.

However, in ESXi 5.0 Update 1, the firewall rules definitely applies and if you were “grandfathered-in” during the upgrade to build 456551: check your syslog for your ESXi 5 servers. If your no longer getting syslog entries, either set the policy in the host’s Configuration->Security Profile->Properties… control panel:

Enabling syslog traffic in the ESXi firewall within the vSphere Client interface.

 

Or use ESXCLI to do the work (especially with multiple hosts):

esxcli network firewall ruleset set –ruleset-id=syslog –enable=true

esxcli network firewall refresh

That will take care of the “absent” syslog entries.

SOLORI’s Take: Gotcha! As ESXi becomes more like ESX in terms of provisioning, old-school ESXiers (like me) need to make sure they’re up-to-speed on the latest changes in ESXi. Ashamed to admit it, but this exact scenario got me in my home lab… Until I stumbled onto KB2003322 I didn’t think to go back and check the ESXi firewall settings – after all, it was previously working 😉

h1

In-the-Lab: NexentaStor vs ESXi, Redux

February 24, 2012

In my last post, I mentioned a much complained about “idle” CPU utilization quirk with NexentaStor when running as a virtual machine. After reading many supposed remedies on forum postings (some reference in the last blog, none worked) I went pit-bull on the problem… and got lucky.

As an avid (er, frequent) NexentaStor user, the luster of the NMV (Nexenta’s Web GUI) has worn off. Nearly 100% of my day-to-day operations are on the command line and/or Nexenta’s CLI (dubbed NMC). This process includes power-off events (from NMC, issue “setup appliance power-off” or “setup appliance reboot”).

For me, the problem cropped-up while running storage benchmarks on some virtual storage appliances for a client. These VSA’s are bound to a dedicated LSI 9211-8i SAS/6G controller using VMware’s PCI pass-through (Host Configuration, Hardware, Advanced Settings). The VSA uses the LSI controller to access SAS/6G disks and SSDs in a connected JBOD – this approach allows for many permutations on storage HA and avoids physical RDMs and VMDKs. Using a JBOD allows for attachments to PCIe-equipped blades, dense rack servers, etc. and has little impact on VM CPU utilization (in theory).

So I was very surprised to find idle CPU utilization (according to ESXi’s performance charting) hovering around 50% from a fresh installation. This runs contrary to my experience with NexentaStor, but I’ve seen reports of such issues on the forums and even on my own blog. I’ve never been able to reproduce more than a 15-20% per vCPU bias between what’s reported in the VM and what ESXi/vCenter sees. I’ve always attributed this difference to vSMP and virtual hardware (disk activity) which is not seen by the OS but is handled by the VMM.

CPU record of idle and IOzone testing of SAS-attached VSA

During the testing phase, I’m primarily looking at the disk throughput, but I notice a persistent CPU utilization of 50% – even when idle. Regardless, the 4 vCPU VSA appears to perform well (about 725MB/sec 2-process throughput on initial write) despite the CPU deficit (3 vCPU test pictured above, about 600MB/sec write). However, after writing my last blog entry, the 50% CPU leach just kept bothering me.

After wasting several hours researching and tweaking with very little (positive) effect, a client e-mail prompted a NMV walk through with resulted in an unexpected consequence: the act of powering-off the VSA from web GUI (NMV) resulted is significantly reduced idle CPU utilization.

Getting lucky: noticing a trend after using NMV to reboot for a client walk-through of the GUI.

Working with the 3 vCPU VSA over the previous several hours, I had consistently used the NMC (CLI) to reboot and power-off the VM. The fact of simply using the NMV to shutdown the VSA couldn’t have anything to do with idle CPU consumption, could it? Remembering that these were fresh installations I wondered if this was specific to a fresh installation or could it show up in an upgrade. According to the forums, this only hampered VMs, not hardware.

I grabbed a NexentaStor 3.1.0 VM out of the library (one that had been progressively upgraded from 3.0.1) and set about the upgrade process. The result was unexpected: no difference in idle CPU from the previous version; this problem was NOT specific to 3.1.2, but specific to the installation/setup process itself (at least that was the prevailing hypothesis.)

Regression into my legacy VSA library, upgrading from 3.1.1 to 3.1.2 to check if the problem follows the NexentaStor version.

If anything, the upgraded VSA exhibited slightly less idle CPU utilization than its previous version. Noteworthy, however, was the extremely high CPU utilization as the VSA sat waiting for a yes/no response (NMC/CLI) to the “would you like to reboot now” question at the end of the upgrade process (see chart above). Once “no” was selected, CPU dropped immediately to normal levels.

Now it seemed apparent that perhaps an vestige of the web-based setup process (completed by a series of “wizard” pages) must be lingering around (much like the yes/no CPU glutton.) Fortunately, I had another freshly installed VSA to test with – exactly configured and processed as the first one. I fired-up the NMV and shutdown the VSA…

Confirming the impact of the "fix" on a second fresh installed NexentaStor VSA

After powering-on the VM from the vSphere Client it was obvious. This VSA had been running idle for some time, so it’s idle performance baseline – established prior across several reboots from CLI – was well recorded by the ESXi host (see above.) The resulting drop in idle CPU was nothing short of astounding: the 3 vCPU configuration has dropped from a 50% average utilization to 23% idle utilization. Naturally, these findings (still anecdotal) have been forwarded on to engineers at Nexenta. Unfortunately, now I have to go back and re-run my storage benchmarks; hopefully clearing the underlying bug has reduced the needed vCPU count…

h1

In-the-Lab: NexentaStor and VMware Tools, You Need to Tweak It…

February 24, 2012

While working on an article on complex VSA’s (i.e. a virtual storage appliance with PCIe pass-through SAS controllers) an old issue came back up again: NexentaStor virtual machines still have a problem installing VMware Tools since it branched from Open Solaris and began using Illumos. While this isn’t totally Nexenta’s fault – there is no “Nexenta” OS type in VMware to choose from – it would be nice if a dummy package was present to allow a smooth installation of VMware Tools; this is even the case with the latest NexentaStor release: 3.1.2.

I could not find where I had documented the fix in SOLORI’s blog, so here it is… Note, the NexentaStor VM is configured as an Oracle Solaris 11 (64-bit) virtual machine for the purpose of vCenter/ESXi. This establishes the VM’s relationship to a specific VMware Tools load. Installation of VMware Tools in NexentaStor is covered in detail in an earlier blog entry.

VMware Tools bombs-out at SUNWuiu8 package failure. Illumos-based NexentaStor has no such package.

Instead, we need to modify the vmware-config-tools.pl script directly to compensate for the loss of the SUNWuiu8 package that is explicitly required in the installation script.

Commenting out the SUNWuiu8 related section allows the tools to install with no harm to the system or functionality.

Note the full “if” stanza for where the VMware Tools installer checks for ‘tools-for-solaris’ must be commented out. Since the SUNWuiu8 package does not exist – and more importantly is not needed for Illumos/Nexenta – removing a reference to it is a good thing. Now the installation can proceed as normal.

After the changes, installation completes as normal.

That’s all there is to getting the “Oracle Solaris” version of VMware Tools to work in newer NexentaStor virtual machines – now back to really fast VSA’s with JBOD-attached storage…

SOLORI’s Note: There is currently a long-standing bug that affects NexentaStor 3.1.x running as a virtual machine. Currently there is no known workaround to keep NexentaStor from running up a 50% cpu utilization from ESXi’s perspective. Inside the NexentaStor VM we see very little CPU utilization, but from the performance tab, we see 50% utilization on every configured vCPU allocated to the VM. Nexenta is reportedly looking into the cause of the problem.

I looked through this and there is nothing that stands out other that a huge number of interrupts while idle. I am not sure where those interrupts are coming from. I see something occasionally called volume-check and nmdtrace which could be causing the interrupts.

Nexenta Support

A bug report was reportedly filed a couple of days ago to investigate the issue further.

h1

Short-Take: vSphere 4.0 U3 Out Today, Drivers & Fixes

May 6, 2011

Today, VMware announces the release of vSphere 4.0 Update 3 (U3). Many, many fixes and enhancements – some rolled-in from (or influenced by) vSphere 4.1. Updates to ESX, ESXi, vCenter and vCenter Update Manager are available now (see below for links).

Don't forget to click the "View History" link to expose the vCenter and ESX updates available for older versions...

VMware ESX/ESXi

  • Enhanced APD handling with automatic failover
  • Inclusion of additional drivers
  • Bug and security fixes
  • Additional guest operating system support
  • Updated VM Tools (WDDM, XPDM and PVSCSI drivers)

Patch Release ESX400-Update03 contains the following individual bulletins:

ESX400-201105201-UG: Updates the VMware ESX 4.0 Core and CIM components
ESX400-201105202-UG: Updates the VMware-esx-remove-rpms
ESX400-201105203-UG: Updates the VMware ESX 4.0 EHCI HCD device driver
ESX400-201105204-UG: Updates the VMware ESX 4.0 USB core component
ESX400-201105205-UG: Updates the VMware ESX 4.0 SCSI qla4xxx driver
ESX400-201105206-UG: Updates the VMware ESX 4.0 USB storage component
ESX400-201105207-UG: Updates the VMware ESX 4.0 SCSI qla2xxx driver
ESX400-201105209-UG: Updates the VMware ESX 4.0 e1000e driver
ESX400-201105210-UG: Updates the VMware ESX 4.0 mptsas, mptspi device drivers
ESX400-201105212-UG: Updates the VMware ESX 4.0 nx-nic device driver
ESX400-201105215-UG: Updates the VMware ESX 4.0 scsi hpsa device driver
ESX400-201105216-UG: Updates the VMware ESX 4.0 bnx2x device driver
ESX400-201105217-UG: Updates the VMware ESX 4.0 Megaraid SAS device driver
ESX400-201105218-UG: Updates the VMware ESX 4.0 bnx2 device driver
ESX400-201105219-UG: Updates the VMware ESX 4.0 SCSI-AIC 79xx device driver

Patch Release ESXi400-Update03 contains the following individual bulletins:

ESXi400-201105201-UG: Updates Firmware
ESXi400-201105202-UG: Updates Tools
ESXi400-201105203-UG: Updates VI Client

VMware vCenter™

  • Windows 2008 R2 is now a supported platform
  • Additional guest operating system customization support
  • Additional vCenter Server database support
  • Bug and security fixes

VMware vCenter Server 4.0 Update 3 offers the following improvements:

Guest Operating System Customization Improvements: vCenter Server adds support for customization of the following guest operating systems:

  •         RHEL 6.0 (32-bit and 64-bit)
  •         SLES 11 SP1 (32-bit and 64-bit)
  •         Windows 7 SP1 (32-bit and 64-bit)
  •         Windows Server 2008 R2 SP1

Additional vCenter Server Database Support: vCenter Server now supports the following databases:

  •         Microsoft SQL Server 2008 R2 (32-bit and 64-bit)
  •         Oracle 11g R2 (32-bit and 64-bit)
  •         IBM DB2 9.7.2 (32-bit and 64-bit)

For more information about using IBM DB2 – 9.7.2 database with vCenter Server 4.0 Update 3, see KB 1037354.

Additional vCenter Server Operating System Support: You can install vCenter Server on Windows Server 2008 R2.
Resolved Issues:In addition, this release delivers a number of bug fixes that have been documented in the Resolved Issues section.

VMware vCenter Update Manager

  • Windows 2008 R2 is now a supported platform
  • Additional vCenter Server database support
h1

ESXi Patches fix VSS, NTP and VMkernel

June 3, 2010

Two patches were made available on May 27, 2010 for ESXi 4.0 to fix certain bugs and security vulnerabilities in the platform. These patches are identified as ESXi400-201005401-SG and ESXi400-201005402-BG.

The first patch is security related and requires a reboot of the ESXi host:

This patch fixes a security issue. The updated NTP daemon fixes a flaw in the way it handled certain malformed NTP packets. The NTP daemon logged information about all such packets and replied with a NTP packet that was treated as malformed when received by another ntpd. A remote attacker could use this flaw to create an NTP packet reply loop between two ntpd servers through a malformed packet with a spoofed source IP address and port, causing ntpd on those servers to use excessive amounts of CPU time and fill disk space with log messages. The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2009-3563 to this issue.

ESXi 4.0 hosts might stop responding when interrupts are shared between VMkernel and service console. You might also observe the following additional symptoms:

  • Network pings to the ESXi hosts might fail.
  • Baseboard management controllers (BMC) such as HP Integrated Lights-Out (iLO) console might appear to be in a non-responsive state.

– VMware KB Article, 1021041

An interesting note is the reference to the service console in ESXi, however the sharing of interrupts between ESX drivers and the service console has long been a problem in ESX (not ESXi since there is no service console)… The second patch does not require a reboot, although it includes an update to VMware Tools which could impact uptime on affected virtual machines (Windows Server 2008 R2 and Windows 7). The KB article says this about the patch:

The VMware Snapshot Provider service is not listed in the Services panel. The quiesced snapshots do not use VMware Tools VSS components in Windows Server 2008 R2 or Windows 7 operating systems. This issue is seen when the user or backup software performs a quiesced snapshot on virtual machines on ESXi 4.0 hosts. This patch fixes the issue.

– VMware KB Article, 1021042

Since VSS quiescence is at issue here, DR snapshots and backups relying on VMware Data Recovery may be unreliable without the new VMware Tools installed. If your systems rely on VMware Data Recovery APIs for backup, this patch should be considered mandatory.

h1

Quick-Take: It’s Time to Upgrade from ESX to ESXi

January 15, 2010

That’s right, I said upgrade from ESX to ESXi – not ESX 3.x to vSphere, but ESX (any version) to vSphere’s ESXi! Ever since VMware PartnerExchange 2009 (April), SOLORI has been advising clients and prospects to focus on ESXi and move away from ESX-based hosts – you know, the one with the “Linux” service console. Personally, I’m glad VMware has strengthened their message about the virtues of ESXi and the direction of their flagship product.

ESXi has a superior architecture and we encourage customers to deploy ESXi as part of any new vSphere deployment. Our future posts will compare ESX 4 and ESXi 4 in detail on topics like hardware compatibility list, performance, and management to demonstrate that ESXi is either on par with or superior than ESX. But for now, here are some key points you should know about ESXi vs. ESX:

  1. The functionality and performance of VMware ESX and ESXi are the same; the difference between the two hypervisors resides in their packaging architecture and operational management. VMware ESXi is the latest hypervisor architecture from VMware. It has an ultra thin footprint with no reliance on a general-purpose OS, setting a new bar for security and reliability (learn more).
  2. In the future, ESXi’s superior architecture will be the exclusive focus of VMware’s development efforts.
  3. New and existing customers are highly encouraged to deploy ESXi. Many Fortune 100 companies have already standardized on the ESXi platform.

VMware Blog, June, 2009

Not unfamiliar with the VI3 version of ESXi, its ease of installation, configuration and management and smaller footprint, I was one of about 10 participants in an “ESXi BoF Breakout Session” with Charu Chaubal of VMware. While discussing vSphere’s ESXi with Charu, I never heard the words “ESXi is a superior architecture,” but I did get a clear message that ESXi was the way of the future. From that point on, it seemed as though any efforts concentrated on (net new) ESX deployments was going to be “time wasted.”

However, it was clear by the whispered tone about ESXi’s virtues that the timing was not right for the real message to be spoken aloud. Remember, this was the launch of vSphere and “ESXi” was strongly associated with “ESXi Free” – not a clear marketing message when license sales and adoption curves are on the line. Perhaps that’s why the “talking points” at PEX2009 always suggested that ESX was the “flagship” hypervisor and ESXi was “targeted for embedded and OEM” deployments.

In practical terms, migration from ESX 3.x to vSphere/ESXi didn’t make a lot of sense for many large or institutional customers at the time due to the lack of third-party driver parity between ESX and ESXi in vSphere. However, for net new installations where thrid-party drivers and service console agents were not a concern, the message about ESXi’s superiority was getting lost. Thomas Paine once said “what we obtain too cheap, we esteem too lightly” and I’d attribute the slow uptake on ESXi to the perception that it was somehow inferior to ESX – a misconception owed to it being offered in a “free” version.

Price is what you pay; value is what you get.

– Warren Buffet

By Warren Buffet’s standards, ESXi is a tremendous value. For the novice entering bare-metal virtualization for the first time, the relative ease with which ESXi installs to a USB flash drive allows for a trial experience that is both quick and reassuring. I’d argue that ESXi’s tiny memory footprint and generous hardware support make it the easiest platform for testing bare-metal virtualization. As I’ve blogged about, ESXi takes about 14 minutes to install to USB flash – from initial CD-ROM boot to configuration console.

Anyone experienced with ESX will have encountered service and performance affecting IRQ sharing issues with the service console. The hours discovering, tracking and mitigating these kinds of issues after a BIOS update or hardware refresh can never be reclaimed. The fact is these problems simply do not exist for ESXi – something many ESXers will appreciate.

I’ve attended a great deal of WebEx sessions on VMware products over the last year and I’m still hearing hushed tones and uncertainty about the role of ESXi versus ESX. To be clear, ESXi is being talked about as “enterprise ready,” but much of the focus is still on ESX. These overtones were still present in an “vSphere: Install, Configure and Manage” course I recently attended to qualify for VCP410. While our instructors were very knowledgeable and experienced, there seemed to be much less confidence when discussing ESXi versus ESX. In fact, the lab guide clearly states:

If you are new to VMware vSphere and you do not have any special needs for more advanced features, use ESXi.

– Page 599, Module 13, Installing VMware ESX and ESXi

The manual – and VMware’s choice of message here – seems to indicate that ESX has “more advanced features” than ESXi. While the “advanced features” VMware is talking about are service console related, it leaves many regarding ESXi as the inferior product in sharp contrast to today’s message. If the statement “ESXi’s superior architecture will be the exclusive focus of VMware’s development efforts” isn’t writing on the wall for the rest of you, here’s a list of VMware’s new talking points on ESXi:

  • Improved Reliability and Security
  • Fewer Patches
  • Less Disk Space
  • Streamlined Deployment and Configuration
  • Reduced Management Overhead
  • Next Generation Hypervisor
  • Superior Architecture
  • Drastically Reduced Hypervisor Footprint
  • Smaller Code Base, Smaller Attach Surface
  • Certified on over 1,000 Server Systems – including USB keys
  • New, Operating System Independent

In contrast, the ESX platform is being re-imaged. Here are some new talking points about ESX:

  • The Older Architecture
  • Relies on Linux OS for Resource Management
  • 20x Larger on-disk Footprint
  • More Complex to Configure
  • Console OS Administration for Configuration and Diagnostics
  • Prone to Arbitrary Code Execution (Console OS)

For many of us familiar with both ESXi and ESX, nothing here is really new. The only real change is the message: build your eco-system around ESXi…

SOLORI’s Take: It’s clear to me that VMware took inventory of its customers and chose to lead with ESX when vSphere was released. I suspect this was a practical decision due to the overwhelming numbers of ESX hosts already installed. However, the change in marketing and positioning we’re witnessing signals that we’re moving toward a time when ESX will be openly considered a dead-end variant.

When will ESX be phased-out? That’s up to market forces and VMware, but the cloud loves efficiency and ESXi is certainly more resource efficient and compartmentalized than its brother ESX. Furthermore, VMware has to maintain two development and support chains with ESX and ESXi and Darwin likes ESXi. If I had to bet, I wouldn’t put my money on seeing an ESX version of the next major release. In any case, when ESX is gone VMware can stop having to make excuses for the “linux console” and the implications that VMware is somehow “based on Linux.”

h1

vSphere 4, Update 1 and ESXi

November 30, 2009

On November 19, 2009 VMware released Update 1 for vSphere 4.0 which, among other bug fixes and errata, adds the following new features:

  • ESX/ESXi
    • VMware View 4.0 support (previously unsupported)
    • Windows 7 and Windows 2008 R2 support (previously “experimental”) – guest customizations now supported
    • Enhanced Clustering support for Microsoft Windows – adds support for VMware HA and DRS by allowing HA/DRS to be disabled per MSCS VM instead of per ESX host
    • Enhanced VMware Paravirtualized SCSI support (pvSCSI boot disks now supported in Windows 2003/2008)
    • Improved vNetwork Distributed Switch performance
    • Increased vCPU per Core limit (raised from 20 to 25)
    • Intel Xeon 3400 (uni-processor variant of the Nehalem)
  • vCenter Server
    • Support for IBM DB2 (Enterprise, Workgroup and Express 9, Express C)
    • Windows 7 and Windows 2008 R2 support (previously “experimental”) – guest customizations now supported
  • vCenter Update Manager
    • Does not support IBM DB2
    • Still no scan or remediate for Windows Server 2003 SP2/R2 64-bit, Windows Server 2008 or Windows 7
  • vSphere Client
  • vSphere Command-Line Interface
    • allows the use of comma-separated bulletins with –bundle option in “vihostupdate”

Authorized VMware users can download the necessary updates for vSphere Update 1 directly from VMware. For ESX and ESXi, updates can be managed and installed from the vCenter Update Manager within the vSphere Client. In addition to the normal backup procedure and those steps recommend by VMware, the following observations may be helpful to you:

  • DRS/HA cluster members CAN be updated auto-magically, however we observed very low end-to-end success rates in our testing lab. We recommend the following:
    • Manually enter maintenance mode for the ESXi server
    • Manually stage/remediate the patches to avoid conflicts
    • Manually re-boot ESXi servers if they do not reboot on their own
    • Re-scan patches when re-boot is complete, to check/confirm upgrade success
    • Manually recover from maintenance mode and confirm HA host configuration
  • For the vSphere Client on Windows 7, completely remove the “hacked” version and clean-install the latest version (download from the updated ESX/ESXi server(s))

SOLORI’s Notes: When upgrading ESXi “auto-magically” we experienced the following failures and unwanted behavior:

  • Update manager failure to stage pending updates and upgrades correctly, resulting in a “time-out” failure. However, updates are/were applied properly after manual reboot.
  • DRS/DPM conflicts with upgrade process:
    • inadequate time given to servers for servers to recover from sleep mode
    • Hosts sent to sleep while updates being evaluated, causing DRS to hold maintenance mode while sleeping hosts awakened, resulting in failed/time-out update process
  • Off-line VMs and templates not automatically migrated during update (maintenance mode) process causing extended unavailability of these assets during update

Additional Notes: Directed to the question of which updates/patches may or may not be “rolled-up” into this update, the release notes are very clear. However, for the sake of convenience, we repeat them here:

Patch Release ESX400-Update01 contains the following individual bulletins:

ESXi 4.0 Update 1 also contains all fixes in the following previously released bundles:

Patch Release ESX400-200906001
Patch Release ESX400-200907001
Patch Release ESX400-200909001

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 5

September 28, 2009

In Part 4 of this series we created two vSphere virtual machines – one running ESX and one running ESXi – from a set of master images we can use for rapid deployment in case we want to expand the number of ESX servers in our lab. We showed you how to use NexentaStor to create snapshots of NFS and iSCSI volumes and create ZFS clone images from them. We then showed you how to stage the startup of the VSA and ESX hosts to “auto-start” the lab on boot-up.

In this segment, Part 5, we will create a VMware Virtual Center (vCenter) virtual machine and place the ESX and ESXi machines under management. Using this vCenter instance, we will complete the configuration of ESX and ESXi using some of the new features available in vCenter.

Part 5, Managing our ESX Cluster-in-a-Box

With our VSA and ESX servers purring along in the virtual lab, the only thing stopping us from moving forward with vMotion is the absence of a working vCenter to control the process. Once we have vCenter installed, we have 60-days to evaluate and test vSphere before the trial license expires.

Prepping for vCenter Server for vSphere

We are going to install Microsoft Windows Server 2003 STD for the vCenter Server operating system. We chose Server 2003 STD since we have limited CPU and memory resources to commit to the management of the lab and because our vCenter has no need of 64-bit resources in this use case.

Since one of our goals is to have a fully functional vMotion lab with reasonable performance, we want to create a vCenter virtual machine with at least the minimum requirements satisfied. In our 24GB lab server, we have committed 20GB to ESX, ESXi and the VSA (8GB, 8GB and 4GB, respectively). Our base ESXi instance consumes 2GB, leaving only 2GB for vCenter – or does it?

Memory Use in ESXi

VMware ESX (and ESXi) does a good job of conserving resources by limiting commitments for memory and CPU. This is not unlike any virtual memory capable system that puts a premium on “real” memory by moving less frequently used pages to disk. With a lot of idle virtual machines, this ability alone can create significant over-subscription possibilities for VMware; this is why it could be possible to run 32GB worth of VM’s to run on a 16-24GB host.

Do we really want this memory paging to take place? The answer – for the consolidation use cases – is usually “yes.” This is because consolidation is born out of the need to aggregate underutilized systems in a more resource efficient way. Put another way, administrators tend to provision systems based on worst case versus average use, leaving 70-80% of those resources idle in off-peak times. Under ESX’s control those underutilized resources can be re-tasked to another VM without impacting the performance of either one.

On the other hand, our ESX and VSA virtual machines are not the typical use case. We intend to fully utilized their resources and let them determine how to share them in turn. Imagine a good number of virtual machines running on our virtualized ESX hosts: will they perform well with the added hardship of memory paging? Also, when begin to use vMotion those CPU and memory resources will appear on BOTH virtualized ESX servers at the same time.

It is pretty clear that if all of our lab storage is committed to the VSA, we do not want to page its memory. Remember that any additional memory not in use by the SAN OS in our VSA is employed as ARC cache for ZFS to increase read performance. Paging memory that is assumed to be “high performance” by NexentaStor would result in poor storage throughput. The key to “recursive computing” is knowing how to anticipate resource bottlenecks and deploy around them.

This brings the question: how much memory is left after reserving 4GB for the VSA? To figure that out, let’s look at what NexentaStor uses at idle with 4GB provisioned:

NexentaStor's RAM footprint with 4GB provisioned, at idle.

NexentaStor's RAM footprint with 4GB provisioned, at idle.

As you can see, we have specified a 4GB reservation which appears as “4233 MB” of Host Memory consumed (4096MB+137MB). Looking at the “Active” memory we see that – at idle – the NexentaStor is using about 2GB of host RAM for OS and to support the couple of file systems mounted on the host ESXi server (recursively).

Additionally, we need to remember that each VM has a memory overhead to consider that increases with the vCPU count. For the four vCPU ESX/ESXi servers, the overhead is about 220MB each; the NexentaStor VSA consumes an additional 140MB with its two vCPU’s. Totaling-up the memory plus overhead identifies a commitment of at least 21,828MB of memory to run the VSA and both ESX guests – that leaves a little under 1.5GB for vCenter if we used a 100% reservation model.

Memory Over Commitment

The same concerns about memory hold true for our ESX and ESXi hosts – albeit in a less obvious way. We obviously want to “reserve” memory for required by the VMM – about 2.8GB and 2GB for ESX and ESXi respectively. Additionally, we want to avoid over subscription of memory on the host ESXi instance – if at all possible – since it will already be working running our virtual ESX and ESXi machines.

Read the rest of this entry ?

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 4

August 26, 2009

In Part 3 of this series we showed how to install and configure a basic NexentaStor VSA using iSCSI and NFS storage. We also created a CIFS bridge for managing ISO images that are available to our ESX servers using NFS. We now have a fully functional VSA with working iSCSI target (unmounted as of yet) and read-only NFS export mounted to the hardware host.

In this segment, Part 4, we will create an ESXi instance on NFS along with an ESX instance on iSCSI, and, using writable snapshots, turn both of these installations into quick-deploy templates. We’ll then mount our large iSCSI target (created in Part 3) and NFS-based ISO images to all ESX/ESXi hosts (physical and virtual), and get ready to install our vCenter virtual machine.

Part 4, Making an ESX Cluster-in-a-Box

With a lot of things behind us in Parts 1 through 3, we are going to pick-up the pace a bit. Although ZFS snapshots are immediately available in a hidden “.zfs” folder for each snapshotted file system, we are going to use cloning and mount the cloned file systems instead.

Cloning allows us to re-use a file system as a template for a copy-on-write variant of the source. By using the clone instead of the original, we can conserve storage because only the differences between the two file systems (the clone and the source) are stored to disk. This process allows us to save time as well, leveraging “clean installations” as starting points (templates) along with their associate storage (much like VMware’s linked-clone technology for VDI.) While VMware’s “template” capability allows us save time by using a VM as a “starting point” it does so by copying storage, not cloning it, and therefore conserves no storage.

Using clones in NexentaStor to aid rapid deployment and testing.

Using clones in NexentaStor to conserve storage and aid rapid deployment and testing. Only the differences between the source and the clone require additional storage on the NexentaStor appliance.

While the ESX and ESXi use cases might not seem the “perfect candidates” for cloning in a “production” environment, in the lab it allows for an abundance of possibilities in regression and isolation testing. In production you might find that NFS and iSCSI boot capabilities could make cloned hosts just as effective for deployment and backup as they are in the lab (but that’s another blog).

Here’s the process we will continue with for this part in the lab series:

  1. Create NFS folder in NexentaStor for the ESXi template and share via NFS;
  2. Modify the NFS folder properties in NexentaStor to:
    1. limit access to the hardware ESXi host only;
    2. grant the hardware ESXi host “root” access;
  3. Create a folder in NexentaStor for the ESX template and create a Zvol;
  4. From VI Client’s “Add Storage…” function, we’ll add the new NFS and iSCSI volumes to the Datastore;
  5. Create ESX and ESXi clean installations in these “template” volumes as a cloning source;
  6. Unmount the “template” volumes using the VI Client and unshare them in NexentaStore;
  7. Clone the “template” Zvol and NFS file systems using NexentaStore;
  8. Mount the clones with VI Client and complete the ESX and ESXi installations;
  9. Mount the main Zvol and ISO storage to ESX and ESXi as primary shared storage;
Basic storage architecture for the ESX-on-ESX lab.

Basic storage architecture for the ESX-on-ESX lab.

Read the rest of this entry ?