Archive for the ‘In-the-Lab’ Category

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 5

September 28, 2009

In Part 4 of this series we created two vSphere virtual machines – one running ESX and one running ESXi – from a set of master images we can use for rapid deployment in case we want to expand the number of ESX servers in our lab. We showed you how to use NexentaStor to create snapshots of NFS and iSCSI volumes and create ZFS clone images from them. We then showed you how to stage the startup of the VSA and ESX hosts to “auto-start” the lab on boot-up.

In this segment, Part 5, we will create a VMware Virtual Center (vCenter) virtual machine and place the ESX and ESXi machines under management. Using this vCenter instance, we will complete the configuration of ESX and ESXi using some of the new features available in vCenter.

Part 5, Managing our ESX Cluster-in-a-Box

With our VSA and ESX servers purring along in the virtual lab, the only thing stopping us from moving forward with vMotion is the absence of a working vCenter to control the process. Once we have vCenter installed, we have 60-days to evaluate and test vSphere before the trial license expires.

Prepping for vCenter Server for vSphere

We are going to install Microsoft Windows Server 2003 STD for the vCenter Server operating system. We chose Server 2003 STD since we have limited CPU and memory resources to commit to the management of the lab and because our vCenter has no need of 64-bit resources in this use case.

Since one of our goals is to have a fully functional vMotion lab with reasonable performance, we want to create a vCenter virtual machine with at least the minimum requirements satisfied. In our 24GB lab server, we have committed 20GB to ESX, ESXi and the VSA (8GB, 8GB and 4GB, respectively). Our base ESXi instance consumes 2GB, leaving only 2GB for vCenter – or does it?

Memory Use in ESXi

VMware ESX (and ESXi) does a good job of conserving resources by limiting commitments for memory and CPU. This is not unlike any virtual memory capable system that puts a premium on “real” memory by moving less frequently used pages to disk. With a lot of idle virtual machines, this ability alone can create significant over-subscription possibilities for VMware; this is why it could be possible to run 32GB worth of VM’s to run on a 16-24GB host.

Do we really want this memory paging to take place? The answer – for the consolidation use cases – is usually “yes.” This is because consolidation is born out of the need to aggregate underutilized systems in a more resource efficient way. Put another way, administrators tend to provision systems based on worst case versus average use, leaving 70-80% of those resources idle in off-peak times. Under ESX’s control those underutilized resources can be re-tasked to another VM without impacting the performance of either one.

On the other hand, our ESX and VSA virtual machines are not the typical use case. We intend to fully utilized their resources and let them determine how to share them in turn. Imagine a good number of virtual machines running on our virtualized ESX hosts: will they perform well with the added hardship of memory paging? Also, when begin to use vMotion those CPU and memory resources will appear on BOTH virtualized ESX servers at the same time.

It is pretty clear that if all of our lab storage is committed to the VSA, we do not want to page its memory. Remember that any additional memory not in use by the SAN OS in our VSA is employed as ARC cache for ZFS to increase read performance. Paging memory that is assumed to be “high performance” by NexentaStor would result in poor storage throughput. The key to “recursive computing” is knowing how to anticipate resource bottlenecks and deploy around them.

This brings the question: how much memory is left after reserving 4GB for the VSA? To figure that out, let’s look at what NexentaStor uses at idle with 4GB provisioned:

NexentaStor's RAM footprint with 4GB provisioned, at idle.

NexentaStor's RAM footprint with 4GB provisioned, at idle.

As you can see, we have specified a 4GB reservation which appears as “4233 MB” of Host Memory consumed (4096MB+137MB). Looking at the “Active” memory we see that – at idle – the NexentaStor is using about 2GB of host RAM for OS and to support the couple of file systems mounted on the host ESXi server (recursively).

Additionally, we need to remember that each VM has a memory overhead to consider that increases with the vCPU count. For the four vCPU ESX/ESXi servers, the overhead is about 220MB each; the NexentaStor VSA consumes an additional 140MB with its two vCPU’s. Totaling-up the memory plus overhead identifies a commitment of at least 21,828MB of memory to run the VSA and both ESX guests – that leaves a little under 1.5GB for vCenter if we used a 100% reservation model.

Memory Over Commitment

The same concerns about memory hold true for our ESX and ESXi hosts – albeit in a less obvious way. We obviously want to “reserve” memory for required by the VMM – about 2.8GB and 2GB for ESX and ESXi respectively. Additionally, we want to avoid over subscription of memory on the host ESXi instance – if at all possible – since it will already be working running our virtual ESX and ESXi machines.

Read the rest of this entry ?

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 4

August 26, 2009

In Part 3 of this series we showed how to install and configure a basic NexentaStor VSA using iSCSI and NFS storage. We also created a CIFS bridge for managing ISO images that are available to our ESX servers using NFS. We now have a fully functional VSA with working iSCSI target (unmounted as of yet) and read-only NFS export mounted to the hardware host.

In this segment, Part 4, we will create an ESXi instance on NFS along with an ESX instance on iSCSI, and, using writable snapshots, turn both of these installations into quick-deploy templates. We’ll then mount our large iSCSI target (created in Part 3) and NFS-based ISO images to all ESX/ESXi hosts (physical and virtual), and get ready to install our vCenter virtual machine.

Part 4, Making an ESX Cluster-in-a-Box

With a lot of things behind us in Parts 1 through 3, we are going to pick-up the pace a bit. Although ZFS snapshots are immediately available in a hidden “.zfs” folder for each snapshotted file system, we are going to use cloning and mount the cloned file systems instead.

Cloning allows us to re-use a file system as a template for a copy-on-write variant of the source. By using the clone instead of the original, we can conserve storage because only the differences between the two file systems (the clone and the source) are stored to disk. This process allows us to save time as well, leveraging “clean installations” as starting points (templates) along with their associate storage (much like VMware’s linked-clone technology for VDI.) While VMware’s “template” capability allows us save time by using a VM as a “starting point” it does so by copying storage, not cloning it, and therefore conserves no storage.

Using clones in NexentaStor to aid rapid deployment and testing.

Using clones in NexentaStor to conserve storage and aid rapid deployment and testing. Only the differences between the source and the clone require additional storage on the NexentaStor appliance.

While the ESX and ESXi use cases might not seem the “perfect candidates” for cloning in a “production” environment, in the lab it allows for an abundance of possibilities in regression and isolation testing. In production you might find that NFS and iSCSI boot capabilities could make cloned hosts just as effective for deployment and backup as they are in the lab (but that’s another blog).

Here’s the process we will continue with for this part in the lab series:

  1. Create NFS folder in NexentaStor for the ESXi template and share via NFS;
  2. Modify the NFS folder properties in NexentaStor to:
    1. limit access to the hardware ESXi host only;
    2. grant the hardware ESXi host “root” access;
  3. Create a folder in NexentaStor for the ESX template and create a Zvol;
  4. From VI Client’s “Add Storage…” function, we’ll add the new NFS and iSCSI volumes to the Datastore;
  5. Create ESX and ESXi clean installations in these “template” volumes as a cloning source;
  6. Unmount the “template” volumes using the VI Client and unshare them in NexentaStore;
  7. Clone the “template” Zvol and NFS file systems using NexentaStore;
  8. Mount the clones with VI Client and complete the ESX and ESXi installations;
  9. Mount the main Zvol and ISO storage to ESX and ESXi as primary shared storage;
Basic storage architecture for the ESX-on-ESX lab.

Basic storage architecture for the ESX-on-ESX lab.

Read the rest of this entry ?

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 3

August 21, 2009

In Part 2 of this series we introduced the storage architecture that we would use for the foundation of our “shared storage” necessary to allow vMotion to do its magic. As we have chosen NexentaStor for our VSA storage platform, we have the choice of either NFS or iSCSI as the storage backing.

In Part 3 of this series we will install NexentaStor, make some file systems and discuss the advantages and disadvantages of NFS and iSCSI as the storage backing. By the end of this segment, we will have everything in place for the ESX and ESXi virtual machines we’ll build in the next segment.

Part 3, Building the VSA

For DRAM memory, our lab system has 24GB of RAM which we will apportion as follows: 2GB overhead to host, 4GB to NexentaStor, 8GB to ESXi, and 8GB to ESX. This leaves 2GB that can be used to support a vCenter installation at the host level.

Our lab mule was configured with 2x250GB SATA II drives which have roughly 230GB each of VMFS partitioned storage. Subtracting 10% for overhead, the sum of our virtual disks will be limited to 415GB. Because of our relative size restrictions, we will try to maximize available storage while limiting our liability in case of disk failure. Therefore, we’ll plan to put the ESXi server on drive “A” and the ESX server on drive “B” with the virtual disks of the VSA split across both “A” and “B” disks.

Our VSA Virtual Hardware

For lab use, a VSA with 4GB RAM and 1 vCPU will suffice. Additional vCPU’s will only serve to limit CPU scheduling for our virtual ESX/ESXi servers, so we’ll leave it at the minimum. Since we’re splitting storage roughly equally across the disks, we note that an additional 4GB was taken-up on disk “A” during the installation of ESXi, therefore we’ll place the VSA’s definition and “boot” disk on disk “B” – otherwise, we’ll interleave disk slices equally across both disks.

NexentaStor-VSA-virtual-hardware

  • Datastore – vLocalStor02B, 8GB vdisk size, thin provisioned, SCSI 0:0
  • Guest Operating System – Solaris, Sun Solaris 10 (64-bit)
  • Resource Allocation
    • CPU Shares – Normal, no reservation
    • Memory Shares – Normal, 4096MB reservation
  • No floppy disk
  • CD-ROM disk – mapped to ISO image of NexentaStor 2.1 EVAL, connect at power on enabled
  • Network Adapters – Three total 
    • One to “VLAN1 Mgt NAT” and
    • Two to “VLAN2000 vSAN”
  • Additional Hard Disks – 6 total
    • vLocalStor02A, 80GB vdisk, thick, SCSI 1:0, independent, persistent
    • vLocalStor02B, 80GB vdisk, thick, SCSI 2:0, independent, persistent
    • vLocalStor02A, 65GB vdisk, thick, SCSI 1:1, independent, persistent
    • vLocalStor02B, 65GB vdisk, thick, SCSI 2:1, independent, persistent
    • vLocalStor02A, 65GB vdisk, thick, SCSI 1:2, independent, persistent
    • vLocalStor02B, 65GB vdisk, thick, SCSI 2:2, independent, persistent

NOTE: It is important to realize here that the virtual disks above could have been provided by vmdk’s on the same disk, vmdk’s spread out across multiple disks or provided by RDM’s mapped to raw SCSI drives. If your lab chassis has multiple hot-swap bays or even just generous internal storage, you might want to try providing NexentaStor with RDM’s or 1-vmdk-per-disk vmdk’s for performance testing or “near” production use. CPU, memory and storage are the basic elements of virtualization and there is no reason that storage must be the bottleneck. For instance, this environment is GREAT for testing SSD applications on a resource limited budget.

Read the rest of this entry ?

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 2

August 19, 2009

In Part 1 of this series we introduced the basic Lab-in-a-Box platform and outlined how it would be used to provide the three major components of a vMotion lab: (1) shared storage, (2) high speed network and (3) multiple ESX hosts. If you have followed along in your lab, you should now have an operating VMware ESXi 4 system with at least two drives and a properly configured network stack.

In Part 2 of this series we’re going to deploy a Virtual Storage Appliance (VSA) based on an open storage platform which uses Sun’s Zetabyte File System (ZFS) as its underpinnings. We’ve been working with Nexenta’s NexentaStor SAN operating system for some time now and will use it – with its web-based volume management – instead of deploying OpenSolaris and creating storage manually.

Part 2, Choosing a Virtual Storage Architecture

To get started on the VSA, we want to identify some key features and concepts that caused us to choose NexentaStor over a myriad of other options. These are:

  • NexentaStor is based on open storage concepts and licensing;
  • NexentaStor comes in a “free” developer’s version with 4TB 2TB of managed storage;
  • NexentaStor developer’s version includes snapshots, replication, CIFS, NFS and performance monitoring facilities;
  • NexentaStor is available in a fully supported, commercially licensed variant with very affordable $/TB licensing costs;
  • NexentaStor has proven extremely reliable and forgiving in the lab and in the field;
  • Nexenta is a VMware Technology Alliance Partner with VMware-specific plug-ins (commercial product) that facilitate the production use of NexentaStor with little administrative input;
  • Sun’s ZFS (and hence NexentaStor) was designed for commodity hardware and makes good use of additional RAM for cache as well as SSD’s for read and write caching;
  • Sun’s ZFS is designed to maximize end-to-end data integrity – a key point when ALL system components live in the storage domain (i.e. virtualized);
  • Sun’s ZFS employs several “simple but advanced” architectural concepts that maximize performance capabilities on commodity hardware: increasing IOPs and reducing latency;

While the performance features of NexentaStor/ZFS are well outside the capabilities of an inexpensive “all-in-one-box” lab, the concepts behind them are important enough to touch on briefly. Once understood, the concepts behind ZFS make it a compelling architecture to use with virtualized workloads. Eric Sproul has a short slide deck on ZFS that’s worth reviewing.

ZFS and Cache – DRAM, Disks and SSD’s

Legacy SAN architectures are typically split into two elements: cache and disks. While not always monolithic, the cache in legacy storage typically are single-purpose pools set aside to hold frequently accessed blocks of storage – allowing this information to be read/written from/to RAM instead of disk. Such caches are generally very expensive to expand (when possible) and may only accomodate one specific cache function (i.e. read or write, not both). Storage vendors employ many strategies to “predict” what information should stay in cache and how to manage it to effectively improve overall storage throughput.

New cache model used by ZFS allows main memory and fast SSDs to be used as read cache and write cache, reducing the need for large DRAM cache facilities.

New cache model used by ZFS allows main memory and fast SSDs to be used as read cache and write cache, reducing the need for large DRAM cache facilities.

Read the rest of this entry ?

h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 1

August 17, 2009

There are many features in vSphere worth exploring but to do so requires committing time, effort, testing, training and hardware resources. In this feature, we’ll investigate a way – using your existing VMware facilities – to reduce the time, effort and hardware needed to test and train-up on vSphere’s ESXi, ESX and vCenter components. We’ll start with a single hardware server running VMware ESXi free as the “lab mule” and install everything we need on top of that system.

Part 1, Getting Started

To get started, here are the major hardware and software items you will need to follow along:

ESX-hardwareRecommended Lab Hardware Components

  • One 2P, 6-core AMD “Istanbul” Opteron system
  • Two 500-1,500GB Hard Drives
  • 24GB DDR2/800 Memory
  • Four 1Gbps Ethernet Ports (4×1, 2×2 or 1×4)
  • One 4GB SanDisk “Cruiser” USB Flash Drive
  • Either of the following:
    • One CD-ROM with VMware-VMvisor-Installer-4.0.0-164009.x86_64.iso burned to it
    • An IP/KVM management card to export ISO images to the lab system from the network

Recommended Lab Software Components

  • One ISO image of NexentaStor 2.x (for the Virtual Storage Appliance, VSA, component)
  • One ISO image of ESX 4.0
  • One ISO image of ESXi 4.0
  • One ISO image of VCenter Server 4
  • One ISO image of Windows Server 2003 STD (for vCenter installation and testing)

For the hardware items to work, you’ll need to check your system components against the VMware HCL and community supported hardware lists. For best results, always disable (in BIOS) or physically remove all unsupported or unused hardware- this includes communication ports, USB, software RAID, etc. Doing so will reduce potential hardware conflicts from unsupported devices.

The Lab Setup

We’re first going to install VMware ESXi 4.0 on the “test mule” and configure the local storage for maximum use. Next, we’ll create three (3) machines two create our “virtual testing lab” – deploying ESX, ESXi and NexentaStor running directly on top of our ESXi “test mule.” All subsequent tests VMs will be running in either of the virtualized ESX platforms from shared storage provided by the NexentaStor VSA.

ESX, ESXi and VSA running atop ESXi

ESX, ESXi and VSA running atop ESXi

Next up, quick-and-easy install of ESXi to USB Flash…
Read the rest of this entry ?

h1

In The Lab: vSphere DPM, Quirky but Functional

July 24, 2009
ESX Hosts in Standby Using DRS's DPM Extension

ESX Hosts in Standby Using DRS's DPM Extension

It would be hard to complain that virtualization is contributing to any kind of “global warming” hysteria. In fact and by its very consolidating nature, virtualization offers many advantages over “traditional” computing models that make it “green” even in its most basic format. VMware reinforces this argument with the claims that “every server that is virtualized saves 7,000 kWh of electricity and four tons of carbon dioxide emissions per year.”

However, virtualization’s promise was born out of the recognition that x86 servers were commonly operating with enormous “excess capacity.” In typical deployments, virtual servers are driven to only 30-40% of capacity, and where excess capacity abounds, there exists a potential for slashing available resources in a quest for limiting on-line power consumption.

Enter VMware’s power saving extension to its Distributed Resource Scheduler (DRS) that scrubs VMware clusters for unused capacity, automatically consolidates stray virtual machines away from near-idle members and shuts those members down to conserve power. This magic green genie is called Distributed Power Management (DPM) and, while simple to configure, has a few quirks that not only hinder its effectiveness but actually make no sense (more on that later).

DRS's DPM In Action

DRS's DPM In Action

“VMware DPM monitors the cumulative demand of all virtual machines in the cluster for memory and CPU resources and compares this to the total available resource capacity of all hosts in the cluster. If sufficient excess capacity is found, VMware DPM places one or more hosts in standby mode and powers them off after migrating their virtual machines to other hosts. Conversely, when capacity is deemed to be inadequate, DRS brings hosts out of standby mode (powers them on) and migrates virtual machines, using VMotion, to them. When making these calculations, VMware DPM considers not only current demand, but it also honors any user-specified virtual machine resource reservations.”

vSphere Resource Management Guide, Managing Power Resources

Read the rest of this entry ?

h1

Quick Take: AMD Istanbul Update

May 21, 2009

AMD was gracious enough to invite us to their Reviewer’s Day on May 20th to have a final look at “Istanbul” and discuss their plans for the product’s upcoming release. While much of the information we received is embargoed until the June, 2009 release date, we can tell you that we’ve have received a couple of AMD’s new 6-core “Istanbul” Opterons for testing and review. We’ll look forward to seeing “Istanbul” in action inside our lab over the next couple of weeks. Our verdict will be available at launch.

Instead of typical benchmarks, we’ll be focusing on Istanbul’s implications for vSphere before the new Opteron hits the streets (remember 6-core is the limit for “free” and “reduced capability” vSphere license). If what we saw from AMD’s internal testing at Reviewer’s Day is accurate , then our AMD/VMware Eco-System partners are going to be very happy with the results. What we can confirm today is that AGESA 3.3.0.3+ 3.5.0.0+ is required to run Istanbul, so start looking for BIOS updates from your vendors as the launch date approaches. The systems we reported on from Tyan back in April will be good-to-go at launch (our GT28 test systems are already running it require a beta BIOS).

SOLORI’s take: We made a somewhat bold prediction on April 30, 2009 that “Shanghai-Istanbul Eco-System looks like an economic stimulus all its own” when comparing the AMD upgrade path to Intel’s (rip and replace) where VMware infrastructures are concerned. That article, Shanghai Economics 101, was one of our most popular AMD-related postings yet, and – judging from what we’ve seen already – it looks like we may have been correct!

While we’re impressed with the ability to flawlessly vMotion from socket 940 to socket-F, we were more impressed with the ability to insert an Istanbul into a Barcelona or Shanghai system and immediately realize the benefits. We’re going to look at our review samples, revisit our price-performance data and Watt/VM calculations before making sweeping recommendation. However, we expect to find Istanbul to be a very good match to on-premise cloud/virtualization initiatives.

SOLORI’s 2nd take: VDI and databased consolidation systems running on 4P AMD boxes are about to take a giant leap forward. We can’t wait to see 24-core and 48-core VMmark scores updated over the next two months. Start asking your system vendor for updated BIOS supporting AGESA 3.5.0.0+ (Tyan are you listening? Supermicro’s AS2041M is already there), and get your 4P test mule updated and prepare to be amazed…

h1

Installing OpManage 8.0 on SLES 10 SP2

April 10, 2009

Network and device managers abound, but the OpManage platform from AdventNet keeps building features and capabilities. Unfortunately, their embrace of Windows and Linux installation choices leaves some in the lurch depending on the platform chosen.

Recently I took version 8.0 for a spin on a trimmed-down installation of SLES 10 SP2. As has been typical with this platform on Suse systems, there were some issues requiring resolution before OpManager was up an running. Fortunately, the issues were localized to the apache server that comes with OM8.

First up, in my trimmed-down install (minimal X-windows, no additional servers), no Apache was included as a SLES package (i.e. in an attempt to avoid package collision). This required the following changes to the SLES environment after OM8 installation:

create file: /etc/ld.so.cond.d/opmanage.conf
   edit, add:  /opt/AdventNet/ME/OpManager/apache/lib
run:  ldconfig

Next, trouble-shooting the remainder of the apache-related issues uncovered the following dependencies:

create link:  ln -s /usr/lib/libgdbm.so.3 /usr/lib/libgdbm.so.2
create link:  ln -s /usr/lib/libdb-4.3.so /usr/lib/libdb-3.2.so
create link:  ln -s /usr/lib/libexpat.so /usr/lib/libexpat.so.0

Once completed, test the service by issuing the following:

cd /opt/AdventNet/ME/OpManager/bin
run  ./StartOpManagerServer.sh

OpManager will point-out the service port you chose during installation. Remember SUSE’s firewall services manager if you get stuck accessing OpManager from the “web client” – it is deny-all by default when enabled.

From the first 15-minutes with the new OpManager, it is clear that AdventNet has come a long way since the early days of the product. There are many new features and devices supported. Check it out for yourself if you’re still shopping for a network/server monitoring system that runs nicely in a virtual machine…

h1

In-the-Lab: VSA Shoot-out

March 17, 2009

We’re cooking with gas over the next two weeks, making performance contrasts using similar resource build-outs of:

  1. FreeNAS (0.7 nightly build)
  2. OpenFiler 2.3
  3. NexentaStor (1.1.5 dev)
  4. SunStorage (VSA)

All configured as VSA’s under ESXi with pass-thru access to local attached storage. Why do this? ESX provides some supervisory analysis that makes quick lab analysis easy. Also, VSA storage is a 2009 killer-app and absolutely necessary for targeted micro-deployments.

These are not indepth storage studys, but an out-of-the-box analysis of what to expect and where to spend your time=money. Stay tuned…