h1

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 5

September 28, 2009

If you’re unfamiliar with vSphere’s changes since the VI3 era, you should know that NFS file systems always use “thin provisioning” for virtual machine disks (vmdk’s), however iSCSI volumes required manual, command line creation of VM disks if they were thin provisioned. This required some scripting tricks (using “Remote Command Line”)  since there is no “console” to manage storage from in ESXi (like there is in ESX.) The cumbersome, Perl-based “Remote Command Line” tools could be avoided by using the “unsupported” shell in ESXi, but this is not a recommended “production quality” process. Fortunately, vSphere adds the ability to “thin provision” virtual machine disks stored on iSCSI volumes using VI Client.

Where “thin provisioning” fails is in copying virtual machine disks through the VI Client. Without vCenter in vSphere (or under vCenter in VI3) the only way to preserve the “thinness” of the virtual machine disk is through the CLI process described above. We’ll show an example of how to preserver the storage as “thin” without vCenter’s control in the next section…

Adding the New NFS Volume

Now that the volume is shared, add the the newly available network storage to ESXi’s datastore. This can be done from the vSphere Client by first selecting the ESXi host in the “Inventory” and clicking the “Storage” link in the Hardware panel of the “Configuration” tab. Clicking on the “Add Storage…” link will bring-up the “Add Storage” wizard.

Next, Select “Network File System” as the storage type, and click “Next.” In the “Properties” requester enter the name of your VSA (FQDN) and the file system’s “Folder” (beginning with “/volumes/”) and name the Datastore something like “Virtual-Machines-VSA” where “VSA” is the short name of your VSA. Since the VSA and ESXi servers both have multiple interfaces, only the named interface that matchs

After a few seconds the NFS file system will be added and “Virtual-Machines” will be added to the available Datastores list. Now it is time to create the vCenter VM using this storage backing.

Creating the VM

Right-click on the ESXi host in inventory and select “New Virtual Machine…” (or Ctrl-N keyboard short-cut). This will be a “typical” configuration named “vCenter” created on the “Virtual-Machines” datastore using a guest operating system template of “Microsoft Windows Server 2003, Standard Edition (32-bit)” and an 8GB thin provisioned disk. Once created, we’ll bump the vCPU’s up to two and the memory up to 3072MB, remove the floppy disk, point the CD-ROM to our W2K3 ISO image, correct the network adapter port group to our management network, add a 12GB and 24GB thin provisioned disk and “Force BIOS Setup” on the first boot.

The next step – as should be habit – is to modify the BIOS settings to disable all devices provided by BIOS that vCenter will not use: diskette A, serial ports A & B, parallel port, floppy controller, primary IDE adapter. We will take this opportunity to demote the “Removable Devices” to below the CD-ROM on the Boot menu as well – leaving the Hard Drive above the CD-ROM as it is blank and will default to CD until Windows is installed (saves a return to BIOS).

If you forgot to enable “Connect at power on” for your CD-ROM, you will be greeted with a PXE boot screen. Without rebooting the VM, open the settings menu and select “Connect at power on” and save the changes. Then, select “Reset” from the “VM/Power” menu (the “restart guest” button works only with VMware Tools running.) We’ll pickup after Windows is finished installing, however remember that the first thing to do – prior to patching the Windows system – is to install the VMware Tools in the VM, then apply all relevant Microsoft patches.

Preserving the Windows Installation for Later Use

If Windows Server 2003 STD is something that you will use again and again in your lab, now is the time to save a pristine copy of the OS – before it gets choked with the inevitable bloat. After applying VMware Tools and system patches, create a snapshot of the file system in NexentaStor’s GUI or create a new file system dedicated to cloning, mount it and copy the VM to that new storage. If “new storage” is your target, remember that copying a VM from the VI Client will result in all virtual machine disks to be “expanded” from their “thin” state to their “thick” state. This is an undesirable consequence of the copy process.

Making “Thin” Copies

To avoid storage bloat – especially since 36GB of the bloat is from yet unformatted and unused virtual machine disks – we suggest manually creating the target folder then copying only the “.vmx” file from the source folder to the destination. After adding the “w2k3-master” VM to inventory, non-destructively remove the virtual machine disks from the new VM definition and we’ll re-attach them after this process. Using either the “unsupported” shell, the “Remote Command Line Appliance” or the “Remote Command Line” from Windows, use the following commands to copy the virtual machine disks:

For ESX CLI:

vmkfstools -i /vmfs/volumes/Virtual-Machines-VSA01/vCenter/vCenter.vmdk \
           -d thin /vmfs/volumes/w2k3-master/w2k3-master/w2k3-master.vmdk
vmkfstools -i /vmfs/volumes/Virtual-Machines-VSA01/vCenter/vCenter_1.vmdk \
           -d thin /vmfs/volumes/w2k3-master/w2k3-master/w2k3-master_1.vmdk
vmkfstools -i /vmfs/volumes/Virtual-Machines-VSA01/vCenter/vCenter_2.vmdk \
           -d thin /vmfs/volumes/w2k3-master/w2k3-master/w2k3-master_2.vmdk

For ESXi vSphere CLI (Remote Command Line):

bin\vmkfstools.pl --server <ESXI_HOST> \
                  -i /vmfs/volumes/Virtual-Machines-VSA01/vCenter/vCenter.vmdk \
                  -d thin /vmfs/volumes/w2k3-master/w2k3-master/w2k3-master.vmdk
bin\vmkfstools.pl --server <ESXI_HOST> \
                  -i /vmfs/volumes/Virtual-Machines-VSA01/vCenter/vCenter_1.vmdk \
                  -d thin /vmfs/volumes/w2k3-master/w2k3-master/w2k3-master_1.vmdk
bin\vmkfstools.pl --server <ESXI_HOST> \
                  -i /vmfs/volumes/Virtual-Machines-VSA01/vCenter/vCenter_2.vmdk \
                  -d thin /vmfs/volumes/w2k3-master/w2k3-master/w2k3-master_2.vmdk

You will need to substitute your source and destination volume names form “Virtual-Machines-VSA01” and “w2k3-master,” respectively. In our example, we have renamed the VM “w2k3-master” as well. For the ESX/ESXi servers using the “console” to make the file system copies, the copy progress will show-up in the CLI window. For the “vSphere CLI” users, the feed-back is only visible on the VI Client’s “Recent Tasks” area – not the CLI window (only success or failure will show-up there.)

VMware "thin copy" status in the VI Client "Recent Tasks" pane. This is the only feedback ESXi users will ever see unless they use the "unsupported" shell.

VMware "thin copy" status in the VI Client "Recent Tasks" pane. This is the only feedback ESXi users will ever see unless they use the "unsupported" shell.

Once the copies are completed, check the NexentaStor GUI for the “Used” storage of your “w2k3” VM – you should only see about 4GB of storage in use for that file system. Remember to re-attach storage to both the vCenter and the “w2k3” VM’s before continuing with either virtual machine.

Confirmation that the copied virtual machine disks remain "thin" - allowing 44GB of potential storage to occupy only 4GB.

Confirmation that the copied virtual machine disks remain "thin" - allowing 44GB of potential storage to occupy only 4GB.

Now that the virtual machine disks are copied, we need to add them to the “w2k3” VM one-by-one. To accomplish this, we open the VM’s properties and click “Add…” in the “Hardware” tab, select “Hard Disk” and click “Next.”. For the disk type, we’ll select “Use an existing virtual disk” and click “Next.” Clicking the “Browse…” button, find the “w2k3-master” volume and select the “w2k3-master.vmdk” file as SCSI0:0. Do the same for “w2k3-master_1.vmdk” and “w2k3-master_2.vmdk” as SCSI0:1 and SCSI0:2, respectively.

Attaching the cloned VMDK files as SCSI disks. Note the "actual" size occupied by the 40GB+ of allocations.

Attaching the cloned VMDK files as SCSI disks. Note the "actual" size occupied by the 40GB+ of allocations.

Note that – unless you need the 2nd and 3rd disks – the extra disks are not necessary for your master as you can add additional virtual machine disks to clone VM’s at the time of their use. We cloned them here mainly for those just learning VMware as an example of how to accomplish it. To complete the “mastering” process, remove the “w2k3-master” VM from inventory and “unmount” the volume from the active “Datastores” – then “un-share” the file system from NexentaStor. It is now available from snapshot and cloning for other uses – patched and ready. Don’t forget to “sysprep” Windows Server 2003 before re-use.

Installing VMware Virtual Center Server

The vCenter virtual machine should be running a fully patched version of Windows Server 2003 Standard Edition (32-bit) at this point. With the VM  running, open the VM’s properties and change the CD-ROM’s target to the vSphere vCenter ISO image – then “Connect” the CD-ROM to the VM and close the properties page.

VMware vCenter Installation Menu

VMware vCenter Installation Menu

In the vCenter VM, the ISO should now be available as the active CD-ROM. Access the CD-ROM from the Windows file manager and the vCenter installer menu should “auto-run” upon access (if it did not run on mount).

SOLORI’s Note: If you are having problems with low-performance storage – manifested as pauses in the vCenter VM during high disk activity (like the installation) – adjust the NexentaStor’s VM to have “High” shares to all of its virtual machine disks and likewise adjust the vCenter VM to have “Low” shares to its virtual machine disks. This will reduce the contention for disk access between the VSA and its dependent virtual machine(s). If the virtual machine does seem to stop responding, wait for the vCenter notice and select “Retry” to resume operation, then adjust the shares at the next opportunity.

Our first step is to install the “vCenter Server” application itself, so click the “vCenter Server” link and continue. After selecting your language localization, click “Next” and wait as the installation is prepared. After a short time (depending on raw storage speed) the “Customer Information” requester will pop-up – for now, let’s enter our personal and organization information and leave the “License key” field blank to enable “evaluation mode.” Evaluation mode will enable us to use nearly the full range of vSphere features for 60-days.

Since we’ve factored-in storage for Microsoft SQL Express, we’ll select “Install a Microsoft SQL Server Express instance” for this deployment. If this was a re-installation, we’d select “Use an existing supported database” and find the DNS from the drop-down. For lab use, we’ll select “Use SYSTEM Account” for the database, however this may violate your organization’s security policies. In production environments, it may be necessary to create a local/domain account specifically for the SQL service account.

SOLORI’s Note: If you have not formatted the additional virtual disks, now is the time to do it. In our example, we’re calling them M: (vCenter) and N: (UpdateMgr) for the vCenter and Update Manager – NTFS quick format should be adequate. Likewise, and from here on out, access to vCenter can be from remote desktop (RDP) or the VI Client virtual machine console – whichever is most convenient or offers the best performance.

Likewise, if you had a lot of patching to get Windows Server 2003 STD up-to-date, there are a lot of “uninstall” files consuming disk space in “C:\Windows” that will need removing before vCenter will install. Additionally, you might want to disable the Windows swap file or move it to the M: drive to conserve space on the C: drive.

We will select the M: drive to install vCenter’s components, accepting the default path otherwise. Once the file system location for vCenter has been selected, the installer prompts for “Linked Mode Options.” Linked mode is vCenter’s way of enabling “single pane of glass” visibility to multiple vCenter installations. Once connected (and assuming a user is authorized to the additional vCenter instances) linked mode allows a single login to manage multiple vCenters and their constituent datacenters. Linked mode does not enable vMotion or file sharing between distant vCenter instances, nor does it allow for shared configuration management or scheduled tasks – it is simply a single pane access to multiple managers. For this example, we select “standalone” as we can add linked mode functionality upon turn-up of a subsequent vCenter installation (perhaps from one of our clones…)

Next, the installer will allow choosing application service ports. We recommend accepting the defaults and moving on. These are 443, 80, 902, 8080, 8443, 389 and 636 resolving to HTTPS, HTTP, heartbeat, web services, web services secure, LDAP and SSL, respectively. From here, the installer has all that is necessary to install vCenter to your VM – sit back and relax while the system is installed.

VMware vCenter Installation Complete!

VMware vCenter Installation Complete!

The system will need to reboot to activate vCenter Server. Now is the appropriate time to do so before proceeding to the Update Manager installation.

Next page, Installing the VMware Virtual Center Update Manager application…

Pages: 1 2 3

6 comments

  1. Good Job !!
    and I am still waiting XD

    and BTW , may I translate this series articles to Chinese and post in my Blog ?

    Like


    • Eric:

      Thanks, and yes – feel free to reproduce this series in the Chinese language provided proper attribution to all sources referenced in the original is maintained and links back to the original SOLORI post(s) are included (see our copyright standard). A “reproduction” notice at the bottom of each page with links back to the source would be appropriate.

      Once your Chinese version is complete, let us know and we’ll link back to it from SOLORI’s site to facilitate SEO. We’ll be releasing Part 6 soon – getting into the “recursive” vMotion capability of this lab series.

      Regards,
      Collin C. MacMillan
      Solution Oriented LLC

      Like


  2. very nice blogs, I learned alot from these 5 parts so far. I want to use the storage appliance you are recommending in this or even just SOlaris 10 on a hardware or as as VM, but I did alot of research but I donot see any write ups on ZFS and MS VSS backups, like Active Directory or Exchange. Suppose I have an ESX with the Solaris VM running for ZFS and presending the datastore to ESX and then I have an Active Directory VM and an Exchange 2007 or 2010 server as a VM both using the ZFS datastore, then is there a way to integrate the MS VSS write with the ZFS snapshots? that way I will know the backups are consistant with the application meaning if I do a restore the database of exchange or logs of exchange will recover without any problems. This is the only problem or grey area i am dealing with right now. Once again thanks again for a very nice write up 🙂

    Farid

    Like


    • Farid:

      Thanks for the comment. The ZFS-based NexentaStor solution (licensed, with VM Data Center plug-in) integrates well into VMware, Hyper-V and Xen environments where VSS agents provide Windows file system quiescence (VSS) through VM Data Center API calls to the hypervisor. This allows for clean, NexentaStor Appliance-driven snapshots on a per-VM or per datastore basis (assuming your VSS writers are in place).

      Restoring from a snapshot – even application consistent ones – can have application-specific requirements. For instance, AD virtual machines may require one restoration process and Exchange 2007 yet another – even though VSS is a commonality in both snapshot-driven back-up solutions. If your virtual environment is more complex (i.e. coordinated data sources) syncing the snapshots between sources will be required to result in consistent systemic recovery instead of a bunch of unsynchronized, individual recovery points.

      Fortunately, the open aspect of NexentaStor allows for complex management scripts (from a central management agent, say a Windows server like vCenter or Linux server like vMA) to coordinate and orchestrate many sequential events to insure the needs of your environment are met. These scripts could be simple – occupying 20-30 lines of script code – or very complex depending on your needs, environment and number of VM’s and/or physical machines it is necessary to coordinate (synchronous backup).

      Without testing in your specific environment, it is impossible to promise “recovery without any problems” and the amount of tuning you will need to do to your process will be dependent on the complexity of the application. Likewise, it may be that VMware Data Recovery or a third-party backup solution like Veeam Backup and Replication would offer more/additional value than VM Data Center alone, given their ability to offer file-level recovery. If image-level recovery fits your needs, you should trial NexentaStor and VM Data Center for yourself for 45-days and see how it fits your application.

      We’ll be running some in-depth looks at VM Data Center and NexentaStor towards the end of March, 2010.

      Like


      • What About something like this…

        VMWare vSphere Essentials Plus for 3 ESX hosts

        ESX Host1:
        Hardware-
        HP DL360 with Array controller with 6x SAS Drives setup in a raid5 with hot-spare
        4-6 Gig ethernet ports

        Software:
        ESXi hypervisor
        NexentaStor Enterprise Silver Edition – 4TB (VM) controlling the Raid5 logical volume

        ESX Host2:
        Hardware-
        HP DL360 with Array controller with 6x SAS Drives setup in a raid5 with hot-spare
        4-6 Gig ethernet ports

        Software:
        ESXi hypervisor
        NexentaStor Enterprise Silver Edition – 4TB (VM) controlling the Raid5 logical volume

        Now, both NexentaStor are controlling local DAS or the array logical drive. I want to setup 2 VIPs on the nexenta VM’s on the storage side.

        so VIP1=
        Nexenta1 – Active
        Nexenta2 – Passive

        so VIP2=
        Nexenta2 – Active
        Nexenta1 – Passive

        now, on ESX1 I can setup a VM lets say Active Directory 1 and point its storage to VIP1, on ESX2 I can setup another VM lets say another AD server and point its storage to VIP2. So AD1 on ESX1 is accessing storage on Nexenta1 and AD2 on ESX2 is accessing Storage on Nexenta2. Now, I want the 2 Nexenta in the back end to setup a lets say syncronous mirror or like software raid-1 across the 2 nexenta’s. so when AD1 writes or delets anything from its storage on nexenta1 that is replicated to nexenta2 in realtime and same with AD2. Now lets say the ESX1 server goes offline for anyreason, it will bring down the AD1 nexenta1 along with it. Now the VMWare HA will kick in and move the AD1 VM to the ESX2, at the same time I want the nexenta2 on ESX2 to take the nexenta 1’s VIP as well, so now nexenta2 responds to both vips, vip1 and vip2. Now when the AD1 machine is on ESX2 it will acess its data on VIP1 which is now on nexenta2. Sorry for long text but this is what I have in mind. Wondering if this is doable in nexenta and esx.

        The last piece is I want to setup a 3rd ESXi in DR somewhere, and it has its own nexenta3 VM and its got same kinda hardware, and i want to do async replication to the 3rd ESX in DR using delta changes only to save bandwidth. so in case of Primary Site going down I will failover to DR and bring the 2 AD servers in ESX3’s online using the data on the nexenta3.

        As far as backup goes, can I use the Nexenta Delorean 2.0 and install the windows client in the AD1 and AD2 VM’s and do backup that way? is the Nexenta Delorean 2.0 considered a snapshot backup or is it like disk to disk backup? if its snapshot then that would be nice if its disk to disk backup then in that case I can use any backup software out there that supports ESX, like the new backup exex 2010 with deduplicaiton and offsite replication built right in to their disk to disk backup.

        thanks,
        Farid

        Like


  3. […] we can use for rapid deployment in case we want to expand the […]      by In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 5 « SolutionOriented Blog September 28, 2009 at 6:34 pm […]

    Like



Comments are closed.

%d bloggers like this: