Posts Tagged ‘esxi’

h1

New Security Patches for ESX/ESXi 3.5, 4.0, 4.1 and 5.0 (6/14/2012)

June 14, 2012

New patches are available today for ESX/ESXi 3.5, 4.0, 4.1 and 5.0 to resolve a few known security vulnerabilities. Here’s the run down if you’re running ESXi 5.0 standard image:

VMware Host Checkpoint File Memory Corruption

Certain input data is not properly validated when loading checkpoint files. This might allow an attacker with the ability to load a specially crafted checkpoint file to execute arbitrary code on the host. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3288 to this issue.

The following workarounds and mitigating controls might be available to remove the potential for exploiting the issue and to reduce the exposure that the issue poses.

Workaround: None identified.

Mitigation: Do not import virtual machines from untrusted sources.

VMware Virtual Machine Remote Device Denial of Service

A device (for example CD-ROM or keyboard) that is available to a virtual machine while physically connected to a system that does not run the virtual machine is referred to as a remote device. Traffic coming from remote virtual devices is incorrectly handled. This might allow an attacker who is capable of manipulating the traffic from a remote virtual device to crash the virtual machine.
The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3289 to this issue.

The following workarounds and mitigating controls might be available to remove the potential for exploiting the issue and to reduce the exposure that the issue poses.

Workaround: None identified.

Mitigation: Users need administrative privileges on the virtual machine in order to attach remote devices. Do not attach untrusted remote devices to a virtual machine.

Deployment Considerations

None beyond the required patch bundles and reboot information listed in the table above.

5.0 Patch Release Notes:
ESXi: http://kb.vmware.com/kb/2021031

Patch Release Information for Other Versions

3.5 Patch Release Notes:
ESX: http://kb.vmware.com/kb/2021020
ESXi: http://kb.vmware.com/kb/2021021

4.0 Patch Release Notes:
ESX: http://kb.vmware.com/kb/2021025
ESXi: http://kb.vmware.com/kb/2021027

4.1 Patch Release Notes:
ESX: http://kb.vmware.com/kb/2019065
ESXi: http://kb.vmware.com/kb/2019243

h1

Quick Take: Syslog Stops Working after Upgrade to ESXi 5.0 Update 1

March 24, 2012

If you’ve recently upgraded your ESXi from 5.0 build 456551 and were logging to syslog, it’s possible that your events are no longer being received by your syslog server. It seems that there was a “feature” in ESXi 5.0 build 456551 that allowed syslog to escape the ESXi firewall regardless of the firewall setting. This could be especially problematic if your upgraded from ESXi 4.x where there was no firewall configuration needed for syslog traffic.

VMware notes that syslog traffic was not affected by the ESXi firewall in v5 build 456551. See KB2003322 for details.

However, in ESXi 5.0 Update 1, the firewall rules definitely applies and if you were “grandfathered-in” during the upgrade to build 456551: check your syslog for your ESXi 5 servers. If your no longer getting syslog entries, either set the policy in the host’s Configuration->Security Profile->Properties… control panel:

Enabling syslog traffic in the ESXi firewall within the vSphere Client interface.

 

Or use ESXCLI to do the work (especially with multiple hosts):

esxcli network firewall ruleset set –ruleset-id=syslog –enable=true

esxcli network firewall refresh

That will take care of the “absent” syslog entries.

SOLORI’s Take: Gotcha! As ESXi becomes more like ESX in terms of provisioning, old-school ESXiers (like me) need to make sure they’re up-to-speed on the latest changes in ESXi. Ashamed to admit it, but this exact scenario got me in my home lab… Until I stumbled onto KB2003322 I didn’t think to go back and check the ESXi firewall settings – after all, it was previously working 😉

h1

In-the-Lab: NexentaStor vs ESXi, Redux

February 24, 2012

In my last post, I mentioned a much complained about “idle” CPU utilization quirk with NexentaStor when running as a virtual machine. After reading many supposed remedies on forum postings (some reference in the last blog, none worked) I went pit-bull on the problem… and got lucky.

As an avid (er, frequent) NexentaStor user, the luster of the NMV (Nexenta’s Web GUI) has worn off. Nearly 100% of my day-to-day operations are on the command line and/or Nexenta’s CLI (dubbed NMC). This process includes power-off events (from NMC, issue “setup appliance power-off” or “setup appliance reboot”).

For me, the problem cropped-up while running storage benchmarks on some virtual storage appliances for a client. These VSA’s are bound to a dedicated LSI 9211-8i SAS/6G controller using VMware’s PCI pass-through (Host Configuration, Hardware, Advanced Settings). The VSA uses the LSI controller to access SAS/6G disks and SSDs in a connected JBOD – this approach allows for many permutations on storage HA and avoids physical RDMs and VMDKs. Using a JBOD allows for attachments to PCIe-equipped blades, dense rack servers, etc. and has little impact on VM CPU utilization (in theory).

So I was very surprised to find idle CPU utilization (according to ESXi’s performance charting) hovering around 50% from a fresh installation. This runs contrary to my experience with NexentaStor, but I’ve seen reports of such issues on the forums and even on my own blog. I’ve never been able to reproduce more than a 15-20% per vCPU bias between what’s reported in the VM and what ESXi/vCenter sees. I’ve always attributed this difference to vSMP and virtual hardware (disk activity) which is not seen by the OS but is handled by the VMM.

CPU record of idle and IOzone testing of SAS-attached VSA

During the testing phase, I’m primarily looking at the disk throughput, but I notice a persistent CPU utilization of 50% – even when idle. Regardless, the 4 vCPU VSA appears to perform well (about 725MB/sec 2-process throughput on initial write) despite the CPU deficit (3 vCPU test pictured above, about 600MB/sec write). However, after writing my last blog entry, the 50% CPU leach just kept bothering me.

After wasting several hours researching and tweaking with very little (positive) effect, a client e-mail prompted a NMV walk through with resulted in an unexpected consequence: the act of powering-off the VSA from web GUI (NMV) resulted is significantly reduced idle CPU utilization.

Getting lucky: noticing a trend after using NMV to reboot for a client walk-through of the GUI.

Working with the 3 vCPU VSA over the previous several hours, I had consistently used the NMC (CLI) to reboot and power-off the VM. The fact of simply using the NMV to shutdown the VSA couldn’t have anything to do with idle CPU consumption, could it? Remembering that these were fresh installations I wondered if this was specific to a fresh installation or could it show up in an upgrade. According to the forums, this only hampered VMs, not hardware.

I grabbed a NexentaStor 3.1.0 VM out of the library (one that had been progressively upgraded from 3.0.1) and set about the upgrade process. The result was unexpected: no difference in idle CPU from the previous version; this problem was NOT specific to 3.1.2, but specific to the installation/setup process itself (at least that was the prevailing hypothesis.)

Regression into my legacy VSA library, upgrading from 3.1.1 to 3.1.2 to check if the problem follows the NexentaStor version.

If anything, the upgraded VSA exhibited slightly less idle CPU utilization than its previous version. Noteworthy, however, was the extremely high CPU utilization as the VSA sat waiting for a yes/no response (NMC/CLI) to the “would you like to reboot now” question at the end of the upgrade process (see chart above). Once “no” was selected, CPU dropped immediately to normal levels.

Now it seemed apparent that perhaps an vestige of the web-based setup process (completed by a series of “wizard” pages) must be lingering around (much like the yes/no CPU glutton.) Fortunately, I had another freshly installed VSA to test with – exactly configured and processed as the first one. I fired-up the NMV and shutdown the VSA…

Confirming the impact of the "fix" on a second fresh installed NexentaStor VSA

After powering-on the VM from the vSphere Client it was obvious. This VSA had been running idle for some time, so it’s idle performance baseline – established prior across several reboots from CLI – was well recorded by the ESXi host (see above.) The resulting drop in idle CPU was nothing short of astounding: the 3 vCPU configuration has dropped from a 50% average utilization to 23% idle utilization. Naturally, these findings (still anecdotal) have been forwarded on to engineers at Nexenta. Unfortunately, now I have to go back and re-run my storage benchmarks; hopefully clearing the underlying bug has reduced the needed vCPU count…

h1

In-the-Lab: NexentaStor and VMware Tools, You Need to Tweak It…

February 24, 2012

While working on an article on complex VSA’s (i.e. a virtual storage appliance with PCIe pass-through SAS controllers) an old issue came back up again: NexentaStor virtual machines still have a problem installing VMware Tools since it branched from Open Solaris and began using Illumos. While this isn’t totally Nexenta’s fault – there is no “Nexenta” OS type in VMware to choose from – it would be nice if a dummy package was present to allow a smooth installation of VMware Tools; this is even the case with the latest NexentaStor release: 3.1.2.

I could not find where I had documented the fix in SOLORI’s blog, so here it is… Note, the NexentaStor VM is configured as an Oracle Solaris 11 (64-bit) virtual machine for the purpose of vCenter/ESXi. This establishes the VM’s relationship to a specific VMware Tools load. Installation of VMware Tools in NexentaStor is covered in detail in an earlier blog entry.

VMware Tools bombs-out at SUNWuiu8 package failure. Illumos-based NexentaStor has no such package.

Instead, we need to modify the vmware-config-tools.pl script directly to compensate for the loss of the SUNWuiu8 package that is explicitly required in the installation script.

Commenting out the SUNWuiu8 related section allows the tools to install with no harm to the system or functionality.

Note the full “if” stanza for where the VMware Tools installer checks for ‘tools-for-solaris’ must be commented out. Since the SUNWuiu8 package does not exist – and more importantly is not needed for Illumos/Nexenta – removing a reference to it is a good thing. Now the installation can proceed as normal.

After the changes, installation completes as normal.

That’s all there is to getting the “Oracle Solaris” version of VMware Tools to work in newer NexentaStor virtual machines – now back to really fast VSA’s with JBOD-attached storage…

SOLORI’s Note: There is currently a long-standing bug that affects NexentaStor 3.1.x running as a virtual machine. Currently there is no known workaround to keep NexentaStor from running up a 50% cpu utilization from ESXi’s perspective. Inside the NexentaStor VM we see very little CPU utilization, but from the performance tab, we see 50% utilization on every configured vCPU allocated to the VM. Nexenta is reportedly looking into the cause of the problem.

I looked through this and there is nothing that stands out other that a huge number of interrupts while idle. I am not sure where those interrupts are coming from. I see something occasionally called volume-check and nmdtrace which could be causing the interrupts.

Nexenta Support

A bug report was reportedly filed a couple of days ago to investigate the issue further.

h1

Short-Take: vSphere 4.0 U3 Out Today, Drivers & Fixes

May 6, 2011

Today, VMware announces the release of vSphere 4.0 Update 3 (U3). Many, many fixes and enhancements – some rolled-in from (or influenced by) vSphere 4.1. Updates to ESX, ESXi, vCenter and vCenter Update Manager are available now (see below for links).

Don't forget to click the "View History" link to expose the vCenter and ESX updates available for older versions...

VMware ESX/ESXi

  • Enhanced APD handling with automatic failover
  • Inclusion of additional drivers
  • Bug and security fixes
  • Additional guest operating system support
  • Updated VM Tools (WDDM, XPDM and PVSCSI drivers)

Patch Release ESX400-Update03 contains the following individual bulletins:

ESX400-201105201-UG: Updates the VMware ESX 4.0 Core and CIM components
ESX400-201105202-UG: Updates the VMware-esx-remove-rpms
ESX400-201105203-UG: Updates the VMware ESX 4.0 EHCI HCD device driver
ESX400-201105204-UG: Updates the VMware ESX 4.0 USB core component
ESX400-201105205-UG: Updates the VMware ESX 4.0 SCSI qla4xxx driver
ESX400-201105206-UG: Updates the VMware ESX 4.0 USB storage component
ESX400-201105207-UG: Updates the VMware ESX 4.0 SCSI qla2xxx driver
ESX400-201105209-UG: Updates the VMware ESX 4.0 e1000e driver
ESX400-201105210-UG: Updates the VMware ESX 4.0 mptsas, mptspi device drivers
ESX400-201105212-UG: Updates the VMware ESX 4.0 nx-nic device driver
ESX400-201105215-UG: Updates the VMware ESX 4.0 scsi hpsa device driver
ESX400-201105216-UG: Updates the VMware ESX 4.0 bnx2x device driver
ESX400-201105217-UG: Updates the VMware ESX 4.0 Megaraid SAS device driver
ESX400-201105218-UG: Updates the VMware ESX 4.0 bnx2 device driver
ESX400-201105219-UG: Updates the VMware ESX 4.0 SCSI-AIC 79xx device driver

Patch Release ESXi400-Update03 contains the following individual bulletins:

ESXi400-201105201-UG: Updates Firmware
ESXi400-201105202-UG: Updates Tools
ESXi400-201105203-UG: Updates VI Client

VMware vCenter™

  • Windows 2008 R2 is now a supported platform
  • Additional guest operating system customization support
  • Additional vCenter Server database support
  • Bug and security fixes

VMware vCenter Server 4.0 Update 3 offers the following improvements:

Guest Operating System Customization Improvements: vCenter Server adds support for customization of the following guest operating systems:

  •         RHEL 6.0 (32-bit and 64-bit)
  •         SLES 11 SP1 (32-bit and 64-bit)
  •         Windows 7 SP1 (32-bit and 64-bit)
  •         Windows Server 2008 R2 SP1

Additional vCenter Server Database Support: vCenter Server now supports the following databases:

  •         Microsoft SQL Server 2008 R2 (32-bit and 64-bit)
  •         Oracle 11g R2 (32-bit and 64-bit)
  •         IBM DB2 9.7.2 (32-bit and 64-bit)

For more information about using IBM DB2 – 9.7.2 database with vCenter Server 4.0 Update 3, see KB 1037354.

Additional vCenter Server Operating System Support: You can install vCenter Server on Windows Server 2008 R2.
Resolved Issues:In addition, this release delivers a number of bug fixes that have been documented in the Resolved Issues section.

VMware vCenter Update Manager

  • Windows 2008 R2 is now a supported platform
  • Additional vCenter Server database support
h1

ESXi Patches fix VSS, NTP and VMkernel

June 3, 2010

Two patches were made available on May 27, 2010 for ESXi 4.0 to fix certain bugs and security vulnerabilities in the platform. These patches are identified as ESXi400-201005401-SG and ESXi400-201005402-BG.

The first patch is security related and requires a reboot of the ESXi host:

This patch fixes a security issue. The updated NTP daemon fixes a flaw in the way it handled certain malformed NTP packets. The NTP daemon logged information about all such packets and replied with a NTP packet that was treated as malformed when received by another ntpd. A remote attacker could use this flaw to create an NTP packet reply loop between two ntpd servers through a malformed packet with a spoofed source IP address and port, causing ntpd on those servers to use excessive amounts of CPU time and fill disk space with log messages. The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2009-3563 to this issue.

ESXi 4.0 hosts might stop responding when interrupts are shared between VMkernel and service console. You might also observe the following additional symptoms:

  • Network pings to the ESXi hosts might fail.
  • Baseboard management controllers (BMC) such as HP Integrated Lights-Out (iLO) console might appear to be in a non-responsive state.

– VMware KB Article, 1021041

An interesting note is the reference to the service console in ESXi, however the sharing of interrupts between ESX drivers and the service console has long been a problem in ESX (not ESXi since there is no service console)… The second patch does not require a reboot, although it includes an update to VMware Tools which could impact uptime on affected virtual machines (Windows Server 2008 R2 and Windows 7). The KB article says this about the patch:

The VMware Snapshot Provider service is not listed in the Services panel. The quiesced snapshots do not use VMware Tools VSS components in Windows Server 2008 R2 or Windows 7 operating systems. This issue is seen when the user or backup software performs a quiesced snapshot on virtual machines on ESXi 4.0 hosts. This patch fixes the issue.

– VMware KB Article, 1021042

Since VSS quiescence is at issue here, DR snapshots and backups relying on VMware Data Recovery may be unreliable without the new VMware Tools installed. If your systems rely on VMware Data Recovery APIs for backup, this patch should be considered mandatory.

h1

Quick-Take: It’s Time to Upgrade from ESX to ESXi

January 15, 2010

That’s right, I said upgrade from ESX to ESXi – not ESX 3.x to vSphere, but ESX (any version) to vSphere’s ESXi! Ever since VMware PartnerExchange 2009 (April), SOLORI has been advising clients and prospects to focus on ESXi and move away from ESX-based hosts – you know, the one with the “Linux” service console. Personally, I’m glad VMware has strengthened their message about the virtues of ESXi and the direction of their flagship product.

ESXi has a superior architecture and we encourage customers to deploy ESXi as part of any new vSphere deployment. Our future posts will compare ESX 4 and ESXi 4 in detail on topics like hardware compatibility list, performance, and management to demonstrate that ESXi is either on par with or superior than ESX. But for now, here are some key points you should know about ESXi vs. ESX:

  1. The functionality and performance of VMware ESX and ESXi are the same; the difference between the two hypervisors resides in their packaging architecture and operational management. VMware ESXi is the latest hypervisor architecture from VMware. It has an ultra thin footprint with no reliance on a general-purpose OS, setting a new bar for security and reliability (learn more).
  2. In the future, ESXi’s superior architecture will be the exclusive focus of VMware’s development efforts.
  3. New and existing customers are highly encouraged to deploy ESXi. Many Fortune 100 companies have already standardized on the ESXi platform.

VMware Blog, June, 2009

Not unfamiliar with the VI3 version of ESXi, its ease of installation, configuration and management and smaller footprint, I was one of about 10 participants in an “ESXi BoF Breakout Session” with Charu Chaubal of VMware. While discussing vSphere’s ESXi with Charu, I never heard the words “ESXi is a superior architecture,” but I did get a clear message that ESXi was the way of the future. From that point on, it seemed as though any efforts concentrated on (net new) ESX deployments was going to be “time wasted.”

However, it was clear by the whispered tone about ESXi’s virtues that the timing was not right for the real message to be spoken aloud. Remember, this was the launch of vSphere and “ESXi” was strongly associated with “ESXi Free” – not a clear marketing message when license sales and adoption curves are on the line. Perhaps that’s why the “talking points” at PEX2009 always suggested that ESX was the “flagship” hypervisor and ESXi was “targeted for embedded and OEM” deployments.

In practical terms, migration from ESX 3.x to vSphere/ESXi didn’t make a lot of sense for many large or institutional customers at the time due to the lack of third-party driver parity between ESX and ESXi in vSphere. However, for net new installations where thrid-party drivers and service console agents were not a concern, the message about ESXi’s superiority was getting lost. Thomas Paine once said “what we obtain too cheap, we esteem too lightly” and I’d attribute the slow uptake on ESXi to the perception that it was somehow inferior to ESX – a misconception owed to it being offered in a “free” version.

Price is what you pay; value is what you get.

– Warren Buffet

By Warren Buffet’s standards, ESXi is a tremendous value. For the novice entering bare-metal virtualization for the first time, the relative ease with which ESXi installs to a USB flash drive allows for a trial experience that is both quick and reassuring. I’d argue that ESXi’s tiny memory footprint and generous hardware support make it the easiest platform for testing bare-metal virtualization. As I’ve blogged about, ESXi takes about 14 minutes to install to USB flash – from initial CD-ROM boot to configuration console.

Anyone experienced with ESX will have encountered service and performance affecting IRQ sharing issues with the service console. The hours discovering, tracking and mitigating these kinds of issues after a BIOS update or hardware refresh can never be reclaimed. The fact is these problems simply do not exist for ESXi – something many ESXers will appreciate.

I’ve attended a great deal of WebEx sessions on VMware products over the last year and I’m still hearing hushed tones and uncertainty about the role of ESXi versus ESX. To be clear, ESXi is being talked about as “enterprise ready,” but much of the focus is still on ESX. These overtones were still present in an “vSphere: Install, Configure and Manage” course I recently attended to qualify for VCP410. While our instructors were very knowledgeable and experienced, there seemed to be much less confidence when discussing ESXi versus ESX. In fact, the lab guide clearly states:

If you are new to VMware vSphere and you do not have any special needs for more advanced features, use ESXi.

– Page 599, Module 13, Installing VMware ESX and ESXi

The manual – and VMware’s choice of message here – seems to indicate that ESX has “more advanced features” than ESXi. While the “advanced features” VMware is talking about are service console related, it leaves many regarding ESXi as the inferior product in sharp contrast to today’s message. If the statement “ESXi’s superior architecture will be the exclusive focus of VMware’s development efforts” isn’t writing on the wall for the rest of you, here’s a list of VMware’s new talking points on ESXi:

  • Improved Reliability and Security
  • Fewer Patches
  • Less Disk Space
  • Streamlined Deployment and Configuration
  • Reduced Management Overhead
  • Next Generation Hypervisor
  • Superior Architecture
  • Drastically Reduced Hypervisor Footprint
  • Smaller Code Base, Smaller Attach Surface
  • Certified on over 1,000 Server Systems – including USB keys
  • New, Operating System Independent

In contrast, the ESX platform is being re-imaged. Here are some new talking points about ESX:

  • The Older Architecture
  • Relies on Linux OS for Resource Management
  • 20x Larger on-disk Footprint
  • More Complex to Configure
  • Console OS Administration for Configuration and Diagnostics
  • Prone to Arbitrary Code Execution (Console OS)

For many of us familiar with both ESXi and ESX, nothing here is really new. The only real change is the message: build your eco-system around ESXi…

SOLORI’s Take: It’s clear to me that VMware took inventory of its customers and chose to lead with ESX when vSphere was released. I suspect this was a practical decision due to the overwhelming numbers of ESX hosts already installed. However, the change in marketing and positioning we’re witnessing signals that we’re moving toward a time when ESX will be openly considered a dead-end variant.

When will ESX be phased-out? That’s up to market forces and VMware, but the cloud loves efficiency and ESXi is certainly more resource efficient and compartmentalized than its brother ESX. Furthermore, VMware has to maintain two development and support chains with ESX and ESXi and Darwin likes ESXi. If I had to bet, I wouldn’t put my money on seeing an ESX version of the next major release. In any case, when ESX is gone VMware can stop having to make excuses for the “linux console” and the implications that VMware is somehow “based on Linux.”