Posts Tagged ‘tyan gt28’


Update: ESXi on GT28

April 12, 2009

Just an updated post to confirm that the GT28 (with KVMoIP SMDC module) has proven to be an excellent platform both in manageability and performance. At a $150/node premium over Supermicro’s AS-1021 dual-node, the quad-gigabit interface appointed Tyan GT28 is great for clustered consolidation and the “roomy” memory allowance gives it scalability. The only thing keeping the GT28 from being “perfect” is its lack of a redundant power option.

Our test systems have been running continuously in the lab for over 3 months without a single failure. We’d like to see an updated BIOS for the SMDC module that gives us a bit more control, but the current platform beats Supermicro’s BIOS stability with their current AGESA 3.3.0+ and KVMoIP offering (AS-1021, AS-2021, etc.).

However, Supermicro has taken it’s 1U/2-node platform to the redundant power stage by offering a 2U/4-node platform with redundant supply built-in. Perfect? Nope – still only twin-gigabit, requiring the single slot be dedicated to additional Ethernet capacity. But then there’s the issue with NO knock-out for its KVMoIP-SMDC module, requiring either ugly hack-it-in wiring or the loss of the expansion slot. A serious mistake? Time will tell…


Installing VMware ESXi on the Tyan Transport GT28

January 15, 2009

Once the GT28 nodes are BIOS-updated to the AGESA v3.3.0.0+ release, a few adjustments are needed to to support my boot-from-flash deployment model. If you are not familiar with boot-from-USB-flash, there are many helpful blogs issued on the subject like this one from Suffice to say, boot-from-USB-flash for ESXi in a relatively simple process to setup:

  1. Make sure your BIOS supports boot-from-USB-flash;
  2. Download the latest release of ESX 3i from VMware;
  3. Mount the ISO image of the 3i installer;
  4. Find the “VMvisor-big” image as a “.dd.bz2” file in the mounted image;
  5. Un-bzip the VMvisor-big image to a temporary directory;
  6. Plug-in your “donor” USB flash device (I’m using the Sandisk Cruzer 4GB);
  7. Find the device handle of the mounted USB device and unmount it (i.e. “umount /dev/sdm”;
  8. Use dd to copy the VMvisor image to the flash device (i.e. “dd if=/tmp/VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd of=/dev/sdm”);
  9. Eject the USB device and label it as ESXi;
  10. Insert the USB flash device into a USB 2.0 port on your equipment and boot;

Preparing the BIOS

To prepare my GT28 for ESX 3i and boot-from-USB-flash, I insert the USB “thumb drive” into one of the rear ports and turn-on the GT28. Hitting the “delete” key on boot gets me to the BIOS setup.  I will start with the BIOS “Optimal Defaults”, and make modifications from there; these adjustments are (follow links for screen shots):

S2935 BIOS screen on boot

S2935 BIOS screen on boot

  1. Reset BIOS to “Optimal Defaults”;
  2. Adjust Northbridge IOMMU window from 128MB to 256MB;
  3. Disable AMD PowerNow in BIOS;
  4. Adjust PCI Latency Timer from 64 to 128 (optional);
  5. Disable nVidia MCP55 SATA controller (ESXi has no driver ESXi has a driver, however may be issues with nVRAID);
  6. Adjust USB Mass Storage, setting the USB flash drive to Hard Disk;
  7. Disable the CD/DVD boot devices to avoid boot conflicts;
  8. Select the USB flash drive as the only boot device;
  9. Finally, save the BIOS changes and reboot;
  10. Now, the system should boot into ESXi for initial configuration;

As you can see, boot-from-USB-flash is “wicked simple” to implement (at least on this platform) and open-up all kinds of testing scenarios. In this case, the ESXi image is now running from USB flash, and only the basic configuration tasks remain. However, it is a good idea to know which Ethernet ports are which on the rear panel of the GT28.

S2935 I/O Ports, Rear

S2935 I/O Ports, Rear

If the PCI bus scan order is configured for “Ascent” the LAN ports will be configured as indicated in the image shown. If you modify the bus scan for “Descent” (i.e. to accommodate a RAID controller) then E2/E3 becomes E0/E1 and E0/E1 becomes E2/E3 due to the new initialization sequence. You may want to, therefore, be cautious when making such a change since ESXi will re-enumerate the interfaces (although any used interface will be pinned to the MAC address.)

Initial Configuration of ESXi

Once your network connections are plugged-in, you should have already mapped-out some IP assignments and VLAN and/or trunking arrangements. While these steps are not strictly necessary in testing, they are a good practice maintain even in testing. To make the initial configurations to ESXi, from the console do the following:

S2935 ESXi demo, Initial Configuration

S2935 ESXi demo, Initial Configuration

  1. Hit “F2” to enter the configuration screen;
  2. Set the “root” password for the ESXi server;
  3. Skip “Lockdown mode” for now;
  4. Configure the management network of the ESXi server;
    1. Select the network adapter(s) to be used for management;
    2. If not using DHCP:
      1. Fix the management IP address
      2. Fix the management IP Subnet mask
      3. Fix the management IP Default gateway
      4. Fix the management DNS configuration;
      5. Update the DNS suffix(es) for your local network;
    3. Hit “Enter” to save (“Escape” exits without change);
  5. Test the management network and restart if necessary;
  6. Exit the configuration menu and hit “F12” to restart;
S2935 ESXi demo, Restarting ESXi after Configuration

S2935 ESXi demo, Restarting ESXi after Configuration

Initial Management with VI Client

Once the ESXi server is up and online, you will need to grab the VMware Infrastructure Client from the web service provided by the ESXi server (http://<ESXi_IP_ADDRESS>/) and install it on your Windows client. If you don’t run windows (like me) you should have a version running in a VM for just such an occasion. I find VirtuaBox to be a better (free) choice for workstation-on-workstation applications and VMware Server a good choice if the client is to be minimal and accessible from multiple hosts.

S2935 ESXi demo, VMware Infrastructure Client Login

S2935 ESXi demo, VMware Infrastructure Client Login

Once the VI Client is installed, run the application and enter the ESXi server’s hostname/IP-address, root username and root password where requested. A VI Client window will eventually open and allow you to complete the setup of the ESXi server as needed.

That’s really all there is to it: we have a reliable, running ESXi platform in mere minutes with minimal effort.


Updated January 15, 2009. Corrected statement that ESX 3i update 3 is not MCP55 aware – support has been added in ESX 3i update 2 and newer. In my test configuration (with the SATA controller enabled) ESX 3i update 3 does properly identify and configure the MCP55 SATA Controller as  an abstracted SCSI controller.

# vmkvsitools lspci
00:00.00 Memory controller: nVidia Corporation
00:01.00 Bridge: nVidia Corporation
00:01.01 Serial bus controller: nVidia Corporation
00:02.00 Serial bus controller: nVidia Corporation
00:02.01 Serial bus controller: nVidia Corporation
00:05.00 Mass storage controller: nVidia Corporation MCP55 SATA Controller [vmhba0]
00:06.00 Bridge: nVidia Corporation
00:10.00 Bridge: nVidia Corporation
00:11.00 Bridge: nVidia Corporation
00:12.00 Bridge: nVidia Corporation
00:13.00 Bridge: nVidia Corporation
00:14.00 Bridge: nVidia Corporation
00:15.00 Bridge: nVidia Corporation
00:24.00 Bridge: Advanced Micro Devices [AMD]
00:24.01 Bridge: Advanced Micro Devices [AMD]
00:24.02 Bridge: Advanced Micro Devices [AMD]
00:24.03 Bridge: Advanced Micro Devices [AMD]
00:24.04 Bridge: Advanced Micro Devices [AMD]
01:05.00 Display controller:
04:00.00 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic0]
04:00.01 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic1]
05:00.00 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic2]
05:00.01 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic3]

This enables the use of the MCP55 SATA controller for flash drives at least. I will do further tests on this platform to determine the stability of the NVRAID component and its suitability for local storage (i.e. embedded VM, like virtual SAN/NAS) needs.


Tyan Transport GT28 Overview, Part 2

January 14, 2009

As I indicated in the earlier posts, the GT28 uses narrow motherboards within a single 1U chassis to create a compact, dual-node HPC platform. Small as it is, the motherboard still packs two socket-F processors, sixteen 240-pin DDR2 DIMM slots, four GigabitEthernet ports, space for an embedded Mellanox Infiniband processor, on board video, a single low-profile PCIe riser slot (x8 slot/signal) and an SMDC slot.

Tyan S2935 Motherboard

Tyan S2935 Motherboard

The GT28 is not a quiet machine once powered-up, and with eight 15,000 RPM fans (3-per node plus 2 power supply fans) the system only gets louder under heavy load. Fortunately, the 45nm Opteron processor is easier on the thermal envelope than its predecessors so the fans stay around 9,000 RPM most of the time. That said, you do not want one of these systems in anything but a full rack enclosure as the tell-tale whine of the system fans is not conducive to office work. This is no appliance chassis and it was not designed to be one, but compared to an 8U AIC storage chassis, this thing is quiet.

The GT28 does provide more than your typical standard compact server motherboard in terms of I/O options – especially network. Although we were not testing the Infiniband variant (about $600 additional) this $1,600 barebones systems comes well equipped for a variety of tasks. As I indicated before, the test system is to become part of a four node storage and hypervisor system based on VMware ESXi and Nexenta NAS (using Solaris’ ZFS storage) . One node will run ESXi with Nexenta running in a virtual machine and three nodes will run ESXi using both NFS and iSCSI as the virtual shared storage medium (provided by Nexenta).

I do want to drift briefly into a dicsussion on cost. Although rock-bottom prices are not a focus, per-node costs are – especially where relative to non-commodity computing models. Read the rest of this entry ?


Tyan Transport GT28 Overview, Part 1

January 13, 2009

First, let me say that I have been using Tyan Barebones Server products for many years. I have found them both robust in features and reliable. Furthermore, I have focused on AMD Opteron Eco-System products since the products became available due to the I/O benefits of hypertransport and the overall better price/performance of Opteron vs. Xeon offerings.

While I have considerable experience with Xeon systems and have participated in many side-by-side comparisons, I am convinced – as a result of such testing – the I/O systems in Opteron-based platforms are far superior than comparable Xeon FSB systems (front-side bus dependent systems). For low TCO systems, the ability to load I/O elegantly is not just an advantage: it’s a must. This loading factors considerably into TCO where per-node utilization and efficiency are large factors.

That said, the GT28 has a Xeon-based counterpart – the Tank GT24 – to satisfy Xeon-based Eco-Systems. The Tank GT24 does not support quad Gigabit Ethernet nor does it support the built-in Mellanox Infiniband interface. The lack of support for these advanced I/O capabilities is further testament to the weak I/O support of the FSB paradigm. Read the rest of this entry ?


Testing the Tyan Transport GT28 (B2935G28V4H)

January 7, 2009

Over the next few posts I’ll be investigating the Tyan GT28, 1U, 2-socket, dual-node (4-socket total) barebones system. This cost effective system comes some impressive features built-in:

  • 1U, Dual-node Chassis (shared 1000W supply)
  • Dual AMD Opteron 2000-series processor (per node)
  • Shanghai processor support, (AGESA v3.3.0.0 required, BIOS v1.03+)
  • 16 DRAM slots (per node, DDR2/REG/ECC)
  • 2 x Dual Gigabit Ethernet (per node, Intel 82571EB)
  • 20Gbps Infiniband 4x port (per node, B2935G28VHI only, Mellanox MT25204)
  • 2 x Hot-Swap SATA/SAS bays (per node)
  • 1 x Low-profile PCI Express x 8 slot (per node)
  • XGI Volari Z9S Graphics Controller
  • nVIDIA nForce Professional 3600 (NFP 3600)
  • IPMI/SMDC slot (per node, Tyan M3295-2/M3296)

I am testing mine with the M3296 SMDC card with IP/KVM support (Raritan KIRA100, firmware version 1.00, build 5772, GT28 r01). WARNING: The units I received did not come with AGESA v3.3.0.0 updated BIOS and would not boot with a Shanghai processor – requiring a BIOS update with an older 2000-series chip.

After BIOS update on both the SMDC and motherboard, system boots without issue in my test configuration (per node):

I’ll have some screen shots of BIOS settings and configurations (courtesy of the M3296’s) soon. Since the Gigabit Ethernet controllers support both “jumbo frame” and TCP off-load, we’ll try to squeeze-in a few performance comparisons of these features as well.