Data Center Strategies

Red Hat and Microsoft: The Odd Couple?

Red Hat and Microsoft announced last week that they have completed the cross-certification of each other’s server and hypervisor products.  I have been blogging about the need this vendor interoperability for more than a year now and am glad to see that this has finally been completed – another quest that I can now mark completed!

There are some interesting aspects behind this announcement with the biggest being that Red Hat submitted the KVM hypervisor technology included in RHEL 5.4 as the third party hypervisor to be certified in the Microsoft server virtualization validation program (SVVP).  RHEL Xen is not included in any of the certifications.  As a result, this certification doesn’t help existing RHEL Xen deployments at all.  I interpret this move two ways:  First that Red Hat is telling its virtualization customers to move to KVM based virtualization from RHEL Xen, and second that Red Hat is done with Xen – they do not want to invest any more than the bare minimum to meet their RHEL 5 product lifecycle support obligations. 

I had blogged last year indicating that Red Hat needed to come out and tell the world clearly where it is going with virtualization.  Red Hat has not publically talked about Xen since it acquired Qumranet last year.  They have not clearly articulated their plans for Xen, but when asked, they indicate that it will be supported throughout the RHEL 5 product lifecycle and then change the subject to the benefits and greatness of KVM.  RHEL 5 first shipped March 14, 2007 meaning that it will end of life near the same day in 2014. 

But Red Hat’s actions tell the story loud and clear:  Red Hat virtualization customers: 2010 should be your year to move from Xen to KVM if you desire to stick with Red Hat virtualization for the long term.  This story will become even more clear once Red Hat releases the Red Hat Enterprise Virtualization (RHEV) platform sometime later this year or early next year.  RHEV consists of a set of virtual infrastructure management tools and cloud enabling components all built around the KVM hypervisor core.  It will also include a stand-alone hypervisor that is effectively a stripped down RHEL 5.4 distribution.  RHEV hypervisor is planned to have RHEL 5.4 kernel parameters tuned, such as thread scheduling and dispatching, for KVM based virtual machines as opposed to natively hosted applications.

The interesting part of this certification for Microsoft is that they now have two Linux distributions certified on Hyper-V: Novell SLES and Red Hat RHEL.  The Red Hat distribution versions certified on Hyper-V include RHEL 5.2 through 5.4.  Preparing for this work, Microsoft submitted paravirtualized Linux guest device drivers for Hyper-V to the mainline Linux kernel, and those were accepted this past summer – another huge step forward in collaboration between Linux and Microsoft.

But Microsoft still has some way to go with Hyper-V and Linux.  Even with the latest Hyper-V R2 release coming soon, Hyper-V does not support virtual SMP for Linux guest operating systems.  This limits Hyper-V’s ability to penetrate into heterogeneous environments.

[Posted by: Richard Jones]

Oracle’s ExaData 2 Results and $10M Challenge

Last month I covered two ads that Oracle had published about their commitment to Sun’s hardware line and its performance vs. IBM as a database server (see here.) The second ad showed an IBM server at 6 million transaction/sec side-by-side with an unannounced SUN configuration at “XX” million transactions/second. I’m a literal kind of guy, so I assumed that “XX” implied a double-digit result greater than 10 million transactions/second. So I was disappointed with the actual Sun result released on Monday (see here). The SPARC-based ExaData 2 cluster achieved 7.7 million transactions/second using the latest hardware, scale-out clustering, and a boat load of FLASH memory instead of high-performance disk. This clustered result represents ~25% improvement over IBM’s single system result from 18 months ago which didn’t use FLASH at all.

The 25% improvement brings me to Oracle’s performance challenge, “offering a 10 million dollar reward if your application isn’t twice as fast on Sun hardware”. If the best that Sun/Oracle’s engineers could manage was a 25% improvement, it sounds like it should be pretty easy to win. But before you start spending the money, read the conditions of the benchmark. First, if you aren’t using IBM storage for your database, you can’t even play (see rule 2-B). Second, you have to prove that the application performance is limited by the database, rather than other way around, which may be difficult. If that doesn’t turn you off, how about section 5-D which states “Sponsor, in its reasonable discretion, reserves the right to limit the participation of any entity or person in the Challenge, amend or interpret these Official Rules”.

Posted by: Nik Simpson

A degraded world awaits you

Anyone have a petabyte scale storage environment? Maybe hundreds of terabytes? Raise your hands. For those with hands raised (you can put them down now), no doubt your life is occupied replacing failed disks. Keeping such a large storage system running can be like me plugging leaks in my roof during a Texas down pour (everything is bigger in Texas after all!). Hardware failures, like my roof leaks during a Texas down pour, are inevitable, continual and just part of life. The bigger the system is, the more likely that at any moment a disk rebuild is underway and someone is madly hunting for the failed disk to replace it, like me hunting for the hole in my roof. And woe to the datacenter that lets its temperature rise – disk failures accelerate as ambient temperature goes up.

Let’s look at an example of a Texas gully washer size system: In 2005, IBM built a supercomputer system for the government (Lawrence Livermore National Laboratory) called Purple/C. Here’s some specs:

• 1536 node 100 teraflop
• 2 PB GPFS (general parallel file system)
• 500 RAID controllers, 11000 disk drives, 4 disks per RAID 5 set
• 126GB/s parallel IO to a single file

Now that’s a big system. Lots of disks. IBM tells me that at any given time, up to 20% of the disks are in a rebuild mode. Since these are SATA disks, that’s not surprising. A rebuild can happen not just due to an outright disk failure but also from a single block corruption. So calculating just on the basis of a disk’s MBTF is misleading. One must also account for data reliability for each disk. The more active a disk is the more likely a bad sector will be encountered and thus require a RAID rebuild. Even with a background disk scrubbing routine.  

That’s all well and good (well, not so good), and certainly there are other large supercomputer-like systems dealing with the same laws of physics, but the lesson, for those not in the supercomputer business, is this:

Large enterprises moving to a cloud storage environment based on distributed commodity storage nodes should expect the same. That is, regular hard failures throughout the system. For some admins this may seem alarming. Disk failures are never a good thing. Grouping 1000s of disks (SATA or otherwise) into a system, however, ensures the presence of failed disks – there is no avoiding the laws of probability.

To make large implementations feasible requires a resilient distributed file and management system capable of living with constant disk failures. Some startups such as Parascale and Gluster are in that space as are the likes of Symantec with their just announced FileMover product, IBM’s SoFS system based on GPFS, IBRIX, and Isilon, etc…

So if you plan a cloud storage deployment within your IT, get use to and plan for disk failures. This means staging boxes of disk drives by the system, a rational way to identify failed disks, and then someone to replace the disk.

To simplify maintenance, maybe a robotic tape library can be repurposed to plug and unplug failed disks (tongue-in-cheek). Unfortunately, no such option exists for my roof.

Posted by Gene Ruth

Storage is too complex

Advanced storage features that  seem whimsically inviting in individual storage systems may prove to be sources of great complexity in a large and a diverse storage infrastructure. Little involvement with business process stakeholders, IT architects and administrators, enables the purchase of “coolest” storage arrays that collectively grow into an unruly management mess.  The lack of proper oversight and adult supervision can result in an inefficient and dysfunctional storage environment so twisted that no team of rocket scientists can unravel it without pyrotechnics.(regards to Robyn Hitchcock – Uncorrected Personality Traits).

hey, cheer up!

In a past blog I touched on the importance of managing information across an enterprise, identifying its value and mapping it accordingly onto storage hardware. Ok, I’ll admit that is not easy, but anything is better than nothing. Even if all storage hardware performed the same with no differentiation, one would still need to ensure that information is properly secured, privatized, discoverable and discarded. None of those is easy to figure out, so let’s turn to another part of the problem:

Storage tier definition

I get asked a lot about the definition of a storage tier. The theory being that once armed with a clear storage tier definition, an organization can place their orders with <favorite vendor here> for tier 1 through tier x. Oh, if only it was so easy, it is not. Storage tiers are in the eye of the beholder. Each organization must set its tiers of service and group storage hardware appropriately. Tier “one” storage to a health insurance records processor is tier “unusable” for a financial institution processing stock transactions.

complexity makes life interesting and ensures full employment.

One truism in the storage industry continues to be, well, ah, true: there is no shortage of storage product to pick from when defining a storage tier. Vendors and technology abound, innovation lives. But in that innovation lies the seeds of confusion and poor choices. Like the Cambrian explosion, many possible forms of storage products simultaneously coexist, but someday their diversity will diminish as market forces willow down the weak contenders.

I need to be careful here lest I rile anyone’s feathers but to test the “complexity” theory open the web page of your favorite top tier storage vendor. I mean companies like IBM, EMC, NetApp, HP, HDS and the like. Count the storage product offerings. On one site I counted 22 major product families and countless model numbers. Talk about covering your bases! Now of course all these models exist because someone is buying them – a testament to good selling techniques. But really, can’t things be simplified? Does the perversity of products help end customers of just allow vendors to inextricably infuse themselves into IT infrastructure? 

Not to let the less-than-giant vendors off the hook, they too attempt to fill every nook and cranny hoping to either make it big (a few do) or get bought by bigger companies (win the lottery) or just fade away to oblivion. [quick side note: I did a briefing with KOM Networks recently which makes archive equipment. They’ve been in business for 40 years! amazing.] Again these “ less-than-giant vendors” slice and dice the storage market to the slimmest of  differentiation.

Narrow your choices

Look at your data and service-level requirements to begin narrowing down the useful storage technologies. Is a unified Ethernet in the cards or do you need to carry along a legacy infrastructure? For most, considerations for an existing infrastructure must be made. Think of the future state of where a storage infrastructure needs to be. Plan out the obsolescence of existing equipment and its replacement candidates. If high transactions rates are called for, then a tricked out SSD product is in your future. But don’t put any service on high performance storage without a justification. Unless proven otherwise, services belong on “good enough” storage. Maybe that’s NAS, maybe that's iSCSI or maybe your server virtualization needs will drive the decision.

In any case, keep the “storage tier” count small. Drive down complexity by pushing a “one size fits all” approach. Avoid mixing vendors in a common tier – this can only lead to heart ache. Now that’s not likely to fly but at least the lunatic fringe case will surface and can be dealt with.  As many vendors offer multiple and significant performance levels in a common frame consider these products to span as many service levels as possible

to simplify your life.

Backup and archive will loom large in any well thought out environment. Apply deduplication and recognize that replication across sites or systems (for disaster recovery) will often lock down a common vendor choice between systems to make the replication work efficiently.

Now you may be thinking, I know this, tell me something new. Well, that’s the trouble we know but do not act. Set a plan in motion to minimize the storage product count in your storage environment. And, don’t ever allow a application to be released into the IT wilds without an information and storage mapping plan. Taking this approach should make a CIO smile, knowing that tangible efforts are being taken to drive down costs and align IT infrastructure with business processes.

Posted by Gene Ruth