Posts Tagged ‘upgrade’


Quick-Take: NexentaStor 4.0.1GA

April 14, 2014

Our open storage partner, Nexenta Systems Inc., hit a milestone this month by releasing NexentaStor 4.0.1 for general availability. This release is significant mainly because it is the first commercial release of NexentaStor based on the Open Source Illumos kernel and not Oracle’s OpenSolaris (now closed source). With this move, NexentaStor’s adhering to the company’s  promise of “open source technology” that enables hardware independence and targeted flexibility.

Some highlights in 4.0.1:

  • Faster Install times
  • Better HA Cluster failover times and “easier” cluster manageability
  • Support for large memory host configurations – up to 512GB of DRAM per head/controller
  • Improved handling of intermittently faulty devices (disks with irregular I/O responses under load)
  • New (read: “not backward compatible”) Auto-Sync replication (user configurable zfs+ssh still available for backward compatibility) with support for replication of HA to/from non-HA clusters
    • Includes LZ4 compression (fast) option
    • Better Control of “Force Flags” from NMV
    • Better Control of Buffering and Connections
  • L2ARC Compression now supported
    • Potentially doubles the effective coverage of L2ARC (for compressible data sets)
    • Supports LZ4 compression (fast)
    • Automatically applied if dataset is likewise compressed
  • Server Message Block v2.1 support for Windows (some caveats for IDMAP users)
  • iSCSI support for Microsoft Server 2012 Cluster and Cluster Shared Volume (CSV)
  • Guided storage pool configuration wizards – Performance, Balanced and Capacity modes
  • Enhanced Support Data and Log Gathering
  • High Availability Cluster plug-in (RSF-1) binaries are now part of the installation image
  • VMware: Much better VMXNET3 support
    • no more log spew
    • MTU settings work from NMV
  • VMware: Install to PVSCSI (boot disk) from ISO no longer requires tricks
  • Upgrade from 3.x is currently “disruptive” – promised “non-disruptive” in next maintenance update
  • Improved DTrace capabilities from NMC shell for
    • general IO
  • Snappier, more stable NMV/GUI
    • Service port changes from 2000 to 8457
    • Multi-NMS default
    • Fast refresh for ZFS containers
    • RSF-1 defaults in “Server” settings
    • Improved iSCSI

See Nexenta’s 4.0.1 Release Notes for additional changes and details.

Note, the 18TB Community Edition EULA is still hampered by the “non-commercial” language, restricting it’s use to home, education and academic (ie. training, testing, lab, etc.) targets. However, the “total amount of Storage Space” license for Community is a deviation from the Enterprise licensing (typical “raw” storage entitlement)

2.2 If You have acquired a Community Edition license, the total amount of Storage Space is limited as specified on the Site and is subject to change without notice. The Community Edition may ONLY be used for educational, academic and other non-commercial purposes expressly excluding any commercial usage. The Trial Edition licenses may ONLY be used for the sole purposes of evaluating the suitability of the Product for licensing of the Enterprise Edition for a fee. If You have obtained the Product under discounted educational pricing, You are only permitted to use the Product for educational and academic purposes only and such license expressly excludes any commercial purposes.

– NexentaStor EULA, Version 4.0; Last updated: March 18, 2014

For those who operate under the Community license, this means your total physical storage is UNLIMITED, provided your space “IN USE” falls short of 18TB (18,432 GB) at all times. Where this is important is in constructing useful arrays with “currently available” disks (SATA, SAS, etc.) Let’s say you needed 16TB of AVAILABLE space using “modern” 3TB disks. The fact that your spinning disks are individually larger than 600GB indicates that array rebuild times might run afoul of failure PRIOR to the completion of the rebuild (encountering data loss) and mirror or raidz2/raidz3 would be your best bet for array configuration.

SOLORI Note: Richard Elling made this concept exceedingly clear back in 2010, and his “ZFS data protection comparison” of 2, 3 and 4-way mirrors to raidz, raidz2 and raidz3 is still a great reference on the topic.

Elling’s MTTDL Comparison by RAID Type


Given 16TB in 3-way mirror or raidz2 (roughly equivalent MTTDL predictors), your 3TiB disk count would follow as:

3-way Mirror Disks := RoundUp( 16 * (1024 / 1000)^3 / 70% / ( 3 * (1000 / 1024)^3 )  ) * 3 = 27 disks, or

6-disk Raidz2 Disks := RoundUp( 16 * (1024 / 1000)^3 / 70% / ( 4 * 3 * (1000 / 1024)^3 )  ) * 6 = 18 disks

By “raw” licensing standards, the 3-way mirror would require a 76TB license while the raidz2 volume would require a 51TB license – a difference of 25TB in licensing (around $5,300 retail). However, under the Community License, the “cost” is exactly the same, allowing for a considerable amount of flexibility in array loadout and configuration.

Why do I need 54TiB in disk to make 16TB of “AVAILABLE” storage in Raidz2?

The RAID grouping we’ve chosen is 6-disk raidz2 – that’s akin to 4 data and 2 parity disks in RAID6 (without the fixed stripe requirement or the “write hole penalty.”) This means, on average, one third of the space consumed on-disk will be in the form of parity information. Therefore, right of the top, we’re losing 33% of the disk capacity. Likewise, disk manufacturers make TiB not TB disks, so we lose 7% of “capacity” in the conversion from TiB to TB. Additionally, we like to have a healthy amount of space reserved for new block allocation and recommend 30% unused space as a target. All combined, a 6-disk raidz array is, at best, 43% efficient in terms of capacity (by contrast, 3-way mirror is only 22% space efficient). For an array based on 3TiB disks, we therefore get only 1.3TB of usable storage – per disk – with 6-disk raidz (by contrast, 10-disk raidz nets only 160GB additional “usable” space per disk.)

 SOLORI’s Take: If you’re running 3.x in production, 4.0.1 is not suitable for in-place upgrades (yet) so testing and waiting for the “non-disruptive” maintenance release is your best option. For new installations – especially inside a VM or hypervisor environment as a Virtual Storage Appliance (VSA) – version 4.0.1 presents a better option over it’s 3.x siblings. If you’re familiar with 3.x, there’s not much new on the NMV side outside better tunables and snappier response.


Quick-Take: NexentaStor 3.1.3 New AD Group Feature, Can Break AD Shares

June 12, 2012

The latest update of NexentaStor may not go too smoothly if you are using Windows Server 2008 AD servers and delegating shares via NexentaStor. While the latest update includes a long sought after fix in AD capabilities (see pull quote below) it may require a tweak to the CIFS Server settings to get things back on track.

Domain Group Support

It is now possible to allow Domain groups as members of local groups. When a Windows client authenticates with NexentaStor using a domain account, NexentaStor consults the domain controller for information about that user’s membership in domain groups. NexentaStor also computes group memberships based on its _local_ groups database, adding both local and domain groups based on local group memberships, which are allowed to be indirect. NexentaStor’s computation of group memberships previously did not correctly handle domain groups as members of local groups.

NexentaStor 3.1.3 Release Notes

In the past, some of NexentaStor’s in-place upgrades have reset the “lmauth_level” of the associated SMB share server from its user configured value back to a “default” of four (4). This did not work very well in an AD environment where the servers were Windows Server 2008 and running their native authentication mode. The fix was to change the “lmauth_level” to two (2) via the NMV or NMC (“sharectl set -p lmauth_level=2 smb”) and restart the service. If you have this issue, the giveaway kernel log entries are as follows:

smbd[7501]: [ID 702911 daemon.notice] smbd_dc_update: myad.local: locate failed
smbd[7501]: [ID 702911 daemon.notice] smbd_dc_monitor: domain service not responding

However, the rules have changed in some applications; Nexenta’s new guidance is:

Summary Description CIFS Issue

A recent patch release by Microsoft has necessitated a changed to the CIFS authorization setting. Without changing this setting, customers will see CIFS disconnects or the appliance being unable to join the Active Directory domain. If you experience CIFS disconnects or problems joining your Active Directory domain, please modify the ‘lmauth_level’ setting.

# sharectl set -p lmauth_level=4 smb

– NexentaStor 3.1.3 Release Notes

While this may work for others out there it does not universally work for any of my tested Windows Server 2008 R2, native AD mode servers. Worse, it appears to work with some shares, but not all; this can lead to some confusion about the actual cause (or resolution) of the problem based on the Nexenta release notes. Fortunately (or not, depending on your perspective), the genesis of NexentaStor is clearlyheading toward an intersection with Illumos although the current kernel is still based on Open Solaris (134f), and a post from OpenIndiana points users to the right solution.

(Jonathan Leafty) I always thought it was weird that lmauth_level had to be set to 2 so I
bumped it back to the default of 3 and restarted smb and it worked...
(Gordon Ross) Glad you found that.  I probably should have sent a "heads-up" when the
"extended security outbound" enhancement went in.  People who have
adjusted down lmauth_level should put it back the the default.

– CIFS in Domain Mode (AD 2008), OpenIndiana Discussion Group (

Following the advice for OpenIndiana re-enabled all previously configured shares. This mode is also the default for Solaris, although NexentaStor continues to use a different one. According to the man pages for smb on Nexenta (‘man smb(4)’) the difference between ‘lmauth_level=3’ and ‘lmauth_level=4’ is as follows:


Specifies the LAN Manager (LM) authentication level. The LM compatibility level controls the type of user authentication to use in workgroup mode or
domain mode. The default value is 3.

The following describes the behavior at each level.

2 – In Windows workgroup mode, the Solaris CIFS server accepts LM, NTLM, LMv2, and NTLMv2 requests. In domain mode, the SMB redirector on
the Solaris CIFS server sends NTLM requests.

3 – In Windows workgroup mode, the Solaris CIFS server accepts LM, NTLM, LMv2, and NTLMv2 requests. In domain mode, the SMB redirector on
the Solaris CIFS server sends LMv2 and NTLMv2 requests.

4 – In Windows workgroup mode, the Solaris CIFS server accepts NTLM, LMv2, and NTLMv2 requests. In domain mode, the SMB redirector on the
Solaris CIFS server sends LMv2 and NTLMv2 requests.

5 – In Windows workgroup mode, the Solaris CIFS server accepts LMv2 and NTLMv2 requests. In domain mode, the SMB redirector on the Solaris
CIFS server sends LMv2 and NTLMv2 requests.

Manpage for SMB(4)

This illustrates either a continued dependency on LAN Manager (absent in ‘lmauth_level=4’) or a bug as indicated in the OpenIndiana thread. Either way, more testing to determine if this issue is unique to my particular 2008 AD environment or this is a general issue with the current smb/server facility in NexentaStor…

SOLORI’s Take: So while NexentaStor defaults back to ‘lmauth_level=4’ and ‘lmauth_level=2’ is now broken (for my environment), the “default” for OpenIndiana and Solaris (‘lmauth_level=3’) is a winner; as to why – that’s a follow-up question… Meanwhile, proceed with caution when upgrading to NexentaStor 3.1.3 if your appliance is integrated into AD – testing with the latest virtual appliance for the win.


Quick Take: Syslog Stops Working after Upgrade to ESXi 5.0 Update 1

March 24, 2012

If you’ve recently upgraded your ESXi from 5.0 build 456551 and were logging to syslog, it’s possible that your events are no longer being received by your syslog server. It seems that there was a “feature” in ESXi 5.0 build 456551 that allowed syslog to escape the ESXi firewall regardless of the firewall setting. This could be especially problematic if your upgraded from ESXi 4.x where there was no firewall configuration needed for syslog traffic.

VMware notes that syslog traffic was not affected by the ESXi firewall in v5 build 456551. See KB2003322 for details.

However, in ESXi 5.0 Update 1, the firewall rules definitely applies and if you were “grandfathered-in” during the upgrade to build 456551: check your syslog for your ESXi 5 servers. If your no longer getting syslog entries, either set the policy in the host’s Configuration->Security Profile->Properties… control panel:

Enabling syslog traffic in the ESXi firewall within the vSphere Client interface.


Or use ESXCLI to do the work (especially with multiple hosts):

esxcli network firewall ruleset set –ruleset-id=syslog –enable=true

esxcli network firewall refresh

That will take care of the “absent” syslog entries.

SOLORI’s Take: Gotcha! As ESXi becomes more like ESX in terms of provisioning, old-school ESXiers (like me) need to make sure they’re up-to-speed on the latest changes in ESXi. Ashamed to admit it, but this exact scenario got me in my home lab… Until I stumbled onto KB2003322 I didn’t think to go back and check the ESXi firewall settings – after all, it was previously working 😉


NexentaStor 3.0 Announced

March 2, 2010

Nexenta Systems announced it’s 3.0 iteration at CeBIT and in a press release this week and has provided a few more details about how the next version is shaping-up. Along with the previously announced deduplication features, the NexentaStor 3.0 edition will include several enhancements to accelerate performance and virtualization applications:

  • In-line deduplication for increased storage savings (virtual machine templates, clones, etc.);
  • Broader support for 10GE adapters and SAS-2 (6Gbps, zoning, etc.) adapters;
  • Replication enhancements to simplify disaster recovery implementations;
  • An updated Virtual Machine Data Center (VMDC v3.0) optional plug-in with VMware, Xen and Hyper-V support (storage-centric control of virtual machine resource provisioning and management);

Additionally, Nexenta is promising easier high-availability (Simple and HA cluster) provisioning for mission critical implementations. Existing NexentaStor license holders will be able to upgrade to NexentaStor 3.0 at no additional cost. Nexenta Systems plans to make NexentaStor 3.0 available by the end of March 2010.

As a Nexenta partner, Solution Oriented will provide clients with valid NexentaStor support contracts upgrade guidance once NexentaStor 3.0 has been released and tested against SOLORI stable image storage platforms. As always, Nexenta’s VMware Ready virtual storage appliance should be your first step in evaluating upgrade potential.


ExtremeTech: Shanghai to Istanbul

June 22, 2009

ExtremeTech runs some tests on the AMD Istanbul 6-core processor and compares the 2435 (2.6GHz) part to the 2384 (2.7GHz) part in a drop-in replacement using PassMark and Spec_JBB2005. Testing was performed in the same Supermicro system running an updated (AGESA BIOS supporting Istanbul processors.

“Perhaps, the most tell tale result comes from the BOPS rating scored using SpecJBB2005, which simulates a server’s ability to process JAVA code. Here, there was a 20% increase in performance, with BOPS increasing from 380721 to 471440. That 20% performance boost would [definitely] be noticed on a busy server in a data center.”

Loyd Case,

While not as thorough as Scott Wassman’s drop-in testing at TechReport (reported earlier this month), ExtremeTech’s results and conclusions were about the same: Istanbul makes a great upgrade processor.

“It all comes down to simple math, where one has to consider the cost of the CPUs and the time needed to perform an upgrade to see if the return on investment is worthwhile. Most will find that in this case, it is…”

– Lloyd Case, ExtremeTech

“And if you have existing, compatible Socket F servers, the Istanbul Opterons should be an excellent drop-in upgrade. They’re a no-brainer, really, when one considers energy costs and per-socket/per-server software licensing fees.”

Scott Wassman, TechReport

Both ExtremeTech and TechReport make compelling upgrade arguments in their testing. Compared to a new system architecture like Nehalem, it is logistically less disruptive – technologically and economically – to certify a CPU upgrade versus as platform replacement. After internal certification, a BIOS and CPU upgrade takes about 20-minutes per system to implement. In a virtualized datacenter where low-level differences are abstracted-away by the hypervisor certification testing should be much less invasive. Likewise, rolling upgrades in a virtualized datacenter with vMotion technology can provide a non-disruptive path from 4-core to 6-core. As Case puts it:

“Simply put, by just upgrading five servers in a data center, data center managers can eliminate the need to purchase an additional server to meet performance needs.”

Lloyd Case, ExtremeTech

However, this “upgrade proposition” is a difficult position for AMD as it does little to sell new systems. Historically, CPU upgrades only happen in 10-15% of the installed base, making CPU sales based on BIOS/drop-in upgrades an interesting footnote. Integrators want to move new hardware with Instanbul pre-installed, not sell “upgrade packages.” Perhaps the dynamics of the new economy will drive a statistical anomoly based on the strength of the Istanbul proposition. Datacenter managers face a familiar dilemma with some new twists.


VMware’s vSphere – Available Today

May 21, 2009

VMware’s new flagship product, vSphere , has hit general release and is now available.

As we reported earlier, vSphere’s license model has changed so existing SnS customers will need to either:

  1. Check your e-mail for updated licenses;
  2. Redeem their VI3 licenses (under SnS) for new vSphere versions;
  3. Download the vSphere trial and request evaluation license (recommended);
  4. Contact their VMware Partner for assistance;

We are strongly recommending a 30-day trial period for Existing VMware customers to vet vSphere in your environment and Eco-System. We’ll be releasing some guidance for upgrades and vSphere ESXi deployment to flash over the next 30-days. We also recommend:

  1. Remember that vSphere’s ESX Server 4.0 is 64-bit ONLY – your 32-bit machines are not upgradeable;
  2. Download the vSphere Evaluation Guide from VMware and review it completely;
  3. Take advantage of VMware’s On-Line Guided Evaluation Training;
  4. Check the vSphere Hardware Compatibility Guide for issues specific to your hardware;
  5. Arrange for a Demonstration and Training through your VMware Partner;
  6. Backup your vCenter database (or virtual machine and database) prior to upgrading;
  7. Sign-up for free vSphere 4 QuickStart training;

Not all of the announced features may be available in licensed version your SnS upgrade entitles you to receive. Please make sure that you are working with your VMware Partner or Consultant to insure that your VI3-to-vSphere trial, training and upgrade goes smoothly.


Quick Take: New Car Syndrome

April 9, 2009

John De Gelas over at Anandtech poses the “million dollar question” of upgrading memory and CPU versus replacing the entire system. So far, there is a 2:1 margin in favor of “throwing out the baby with the bath water” and replacing the entire server even when new CPUs are available. This curious “new car syndrome” that seems to have affected IT decision makers in the past will have to change in the “new” economy. Is Intel’s marketing strategy sound?

Some in the AMD fan-boy camp have accused Anand of “bending” to Intel by “always” presenting new Intel products in the glowing light of an post-coital cigarette. However, Intel has consistently “outed” more technology than AMD on many more fronts: hands-down, Intel gives Anand’s group much more to talk about! However, this relentless cycle of “product renovation” from Intel is an interesting formula of update and obsolescence that – like the “latest model” vehicle update – drags the user to a buying decision much earlier than the life-cycle warrants.

SOLORI is big on stable image platforms – Intel has SIPP for desktops and AMD’s cover desktop and server – and expanding an equipment line’s useful lifetime from 18 to 36 months with simple memory and processor upgrades seems an obvious choice. Perhaps the Intel-focus at AnandTech creates a reader bias towards disposable platforms than it would with more AMD followers. AMD Opteron users are likely more familiar with processor swaps than Intel users, given that a single-core AMD platform purchased in 2006 can be upgraded to 6-core Istanbul later this year with only another BIOS update – adding another 12-months to its life: that’s a 48-month product lifetime in tech! Read the rest of this entry ?