Data Center Strategies

New Readers Begin There!

Just to let everybody know, this will be my last post on the DCS blog.

<pauses briefly to allow the moaning, wailing, and gnashing of teeth to die down>

Ok, don’t panic, I’m not disappearing from the blogosphere, just moving my blog onto the Gartner Blogging Network effectively immediately. So if you want to continue to see my thoughts, musings, and general ramblings on topics such as server design, blades, benchmarks, I/O virtualization, operating systems, cloud computing etc, just point your feed reader to:

http://blogs.gartner.com/nik-simpson.

Posted by: Nik Simpson

Microsoft in Data Center ARM Race?

Yesterday, an eagle-eyed reporter at EE Times noticed an interesting job on offer at Microsoft in its BING Autopilot group. According to the job ad, Bing Autopilot is responsible for designing the infrastructure behind Microsoft’s Bing search platform, and as part of its duties the group evaluates new technologies with solid state disk (SSD) and ARM processors specifically referenced. Microsoft looking at SSD technologies doesn’t come as big surprise (if you aren’t looking at SSD, you need to get with the program), but the mention of the ARM processor has certainly turned some heads.

In years past, Microsoft has been associated almost exclusively with the x86 server platform, yes there have been ports to other platforms (I’ve personally seen Windows on MIPS, Alpha, PowerPC, and CLIPPER) but they’ve all died a pretty rapid death with Itanium being the latest casualty.  There has been speculation about Microsoft doing something with ARM going back at least a year but this is the first concrete indication of their interest, so the question has to be asked, what does Microsoft want ARM for?

If you look at how search engine infrastructure is built you can get a good idea. Search engines are invariably built on scale-out principles with hundreds or more likely thousands of individual servers working in concert to deliver search results. More interestingly, these servers tend to be relatively weak in terms of CPU power, they’re not using 4-socket 128-thread monsters, for the most part it’s cheap and cheerful two-socket servers with an emphasis on power efficiency and small form factor, which is where ARM comes in. ARM may not be fast compared to the latest Intel and AMD processors, but it’s wickedly efficient in terms of power consumption which brings two benefits:

• Reduced power consumption/node: if your processor operates at 10 watts instead of 50 watts, that’s a large potential saving.
• Smaller form factors: low power requirements translate to small power supplies and passive cooling both of which help designers to pack more servers in given amount of data center real estate.

What Microsoft no doubt wants to know is whether using ARM in the servers that power Bing would allow them to maintain or increase performance within a given data center power foot print. For example, if ARM enables Bing to process 10% more search requests for the same amount of power, then that’s a potentially big win.

As a footnote, if you’re getting excited about the possibility of Windows running on an ARM-based netbook, don’t hold your breath. First, we know next to nothing about the underlying software for Bing, the OS platform may not be Windows at all, it could be a kernel specifically designed to support search operations, or horror of horrors it might even be LINUX ;-) Second, even if it is Windows underneath Bing, there’s no guarantee that work done to support Bing would ever make it outside Microsoft data centers.

Posted by: Nik Simpson

When All You Have is Hammer…

Everything looks like a nail, a truism that ASHRAE (The American Society of Heating, Refrigerating and Air-Conditioning Engineers) has just demonstrated in its advice to data center operators on how to achieve greater cooling efficiency in the data center (see ASHRAE Standard 90.1). In ASHRAE’s opinion, the way to achieve data center cooling efficiency is through the use of various economizer techniques (air-side, water-side …) to reduce the amount of energy used by the cooling plant. Unfortunately, this is a rather narrow view of the problem, focused on the stuff that ASHRAE members do best, i.e. air-conditioning and chilled water plants. The ASHRAE standard has now drawn a universal thumbs down from some of the largest data center operators (Google, Microsoft, Amazon, Digital Realty Trust, DuPont Fabros Technology and Nokia) who issued a joint statement urging ASHRAE to re-think it’s position.

The truth is that economizer are just one approach to achieving more efficient energy usage and the best approach will vary based on a host of factors. For example, one company might favor widespread adoption of server virtualization as a way to reduce energy consumption, while another might be able to scavenge energy from the waste heat produced by the data center and use it heat buildings. Both approaches lead to more efficient use of energy without requiring the use of economizers. The problem with ASHRAE’s approach is that there is a danger it will get built into building construction codes and potentially restrict innovation in the data center. James Hamilton’s blog entry on the subject (see here) does an excellent job of describing the problems of such a restrictive definition.

Posted by: Nik Simpson

A Busy Week (Part 2)

Funnily enough, the week turned out to be so busy that I never got around to talking about AMD’s Opteron 6000 series. The main changes vs. the previous high-end Opterons are:

• Twice as many cores: The 6000 family tops out at 12 cores/socket, a feat achieved by taking two 6-core modules and putting them on the same chip.
• Memory: The two channel DDR2 memory controller is replaced by a four channel DDR3 memory controller to improve both memory capacity (12 DIMMS/CPU) and memory throughput.
• 2- and 4-socket support: Vendors can build both 2- and 4-socket servers using the Opteron 6000 family.
• No 8-socket support: Anybody waiting for 8-socket Opteron 6000 server should prepare for a disappointment. Existing users of 8-socket Opteron servers will have to turn to Intel if they need 8-sockets in the future.
• Direct Connect 2.0: Previous generations of 4-socket AMD-based servers had the sockets connected to each other in a ring configuration, so that accessing memory attached to another socket might involve two hops. The new architecture connects the sockets in a mesh, with each socket connected directly to the other 3 sockets. The net result is reduced memory latency for access to memory attached to another socket.

All this adds up to a rather different view of the how the commodity server market will develop. AMD is putting it’s faith in sweet spot of the market, large 2-socket servers and cheaper 4-socket configurations, with an emphasis on power efficiency. Given AMD’s desire to get a larger percentage of the commodity server market this makes a lot of sense, A few extra percentage points of market share in the 2-socket market will do much more good for their bottom-line than complete ownership of the 8-socket and above market.

The wisdom of AMD’s decision to cede the 8-socket and above market to Intel’s Xeon 7500 series remains to be seen. On the one hand, historically this market has been tiny, for several reasons:

• Reliability: The market for applications that require large servers has always required high levels of reliability as well. Put simply, any application that demands tens of processors and terabytes of memory is almost by definition “mission critical” since it usually serves an important function in the IT infrastructure.
• Cost: Systems that have offered 8 or more sockets have required specialist chipsets to deliver the required scalability and availability. As soon as you start needing proprietary chipsets, the cost goes up dramatically.
• Scale out: Many applications that used to require large socket-count systems have been redesigned to scale out across many smaller servers.

On the other hand, Intel’s Xeon 7500 removes at least two of the reasons, reliability and cost, by borrowing the reliability features of Itanium and enabling OEMs to build “cookie-cutter” 8-socket configurations that can support 1TB of memory with 8GB DIMMs.

Footnote: At launch, AMD didn’t offer any VMmark numbers, leaving that job to their OEMs who will no doubt have something to say on the matter when they officially launch their AMD-based offerings later this quarter.

Posted by: Nik Simpson

Microsoft Pulls The Plug on Itanium

Microsoft will not support Itanium processors with future releases Windows Server according to the Windows Server Division blog. So Windows Server 2008 R2 is the last release of Windows on Itanium, with support ending in July 2013 (extended support continues to 2018). It’s no coincidence that this announcement comes in the same week that Intel announced the Xeon 75xx family which brings the Itanium’s reliability features to the x64 commodity server platform (see here for more information). For Microsoft, reliability was the only thing that Itanium had going for it, the number of Windows licenses sold on Itanium is negligible compared to the x64 business. So the decision to drop Itanium was probably a relatively easy one.

For the Itanium platform, the news is bad, but not disastrous, at least for now. The vast majority of Itanium processors go into systems running HP-UX, so losing Windows support isn’t the end of the world. The problem is that Microsoft isn’t the only vendor abandoning the Itanium platform, Redhat made a similar decision at the end of 2009, announcing that it would not support Itanium on the 6.0 release of Redhat Enterprise Linux (see here). That leaves only Novell (Suse Enterprise Linux), HP (HP-UX, OpenVMS), and Groupe Bull (GCOS) in the operating system business for Itanium. There are other LINUX distributions that support Itanium, but none of them have much in the way of enterprise ready credentials. Of those operating system platforms, only HP-UX and OpenVMS actually require Itanium. GCOS and SEL are both supported on x64 processors, which will probably hasten the end of support on Itanium now that x64 can match Itanium for reliability.

So HP is likely to be the last man standing in the Itanium systems business, but the new features and scalability of the 75xx Xeon beg the question, “will HP adapt its Itanium hardware to support the Xeon 75xx and port HP-UX or OpenVMS to the x64 platform?”

Technically, adapting the Integrity Superdome server platform probably isn’t that hard. The 93xx Itanium and 75xx Xeon processor families share the same memory controller and quick path interconnects, simplifying the task considerably, and porting HP-UX and OpenVMS to new architectures is nothing new. Right now, I expect that HP is at least taking a look at this and talking to some of their larger customers and independent software vendors to gauge their reaction. The big question for HP is whether those customers and ISVs would find jumping ship to Power or SPARC to be a more palatable alternative than following HP to Itanium-less future?

Posted by: Nik Simpson

A Busy Week (Part 1)

Apparently Intel and AMD don’t care about poor analysts trying to get their content completed by the end of the quarter, as evidenced by their decision to launch major server processor refreshes (AMD, Intel) on consecutive days this week. Needless to say there has been a vast amount of coverage from web pundits already, so I’m not going rehash it in detail.

Instead I’m going to focus on things that should matter to anybody buying servers this year. First, lets look at Intel’s Xeon 75xx and 65xx processors with respect to:

• Virtualization performance
• Memory subsystem
• System reliability
• Scale up

I’ll look at the AMD Opteron 6000 in a later blog.

Virtualization Performance

Virtualization is one of the few workloads that really make sense on these new processors, and judging by the numbers, they really do deliver. Using VMware’s VMmark benchmark, Intel claims a top score of 71.85 @ 49 tiles using 32-cores (4 x 8-core processors) on an IBM System X3850 X5. To put that in perspective, the previous best 32-core result topped out at 31.56 @ 21tiles on HP ProLiant DL785 using 8 x 4-core AMD processors. The result also trounced a 64-core result (48.23 @ 32 tiles) using 16 of the previous generation Intel XEON processors. That’s a hugely impressive result for a 4-socket system!

Memory Subsystem

This release drives the final nail into Intel’s aging “Front Side Bus” (FSB) memory architecture that dates back the mid-1990s. The FSB architecture had all the processors connected to a common memory controller (AKA “Northbridge”) so that each processor competed with its peers for access to a common pool of memory, thus becoming a bottleneck for performance. The new processors have integrated memory controllers supporting up to 16 memory DIMMs for a total of 64 DIMMs on a 4-socket server (512 GB with 8GB DIMMs), or 32 DIMMs on a 2-socket server ( 256 GB with 8 GB DIMMs).

System Reliability

In the past, x86-based servers have have used a unsophisticated approach to error handling; tell the operating system that something horrible happened and keel over in a heap on the floor. The new Xeons take a very different approach that allows the hardware and operating system to react in a much more flexible way to errors. For example if an unrecoverable memory error occurs, the operating system can look at the error and simply map that memory location out of use, or kill the process (or virtual machine) that is using the affected memory location.  In effect, Intel has blurred the line between it’s own high-end Itanium architecture and Xeon, which is good for Xeon, but not so good for Itanium.

Scale Up

The new Xeon architecture can be used to build 8-socket servers with off-the-shelf components (sometimes referred to as a “glue less design”). But they don’t stop at sockets, the Xeon 7500 family in combination with third-party developed memory hubs is designed to support configurations as large as 256 sockets (4096 threads) with 16 TB of memory! The combination of a highly scalable architecture and high system reliability has the potential to put commodity server architectures into direct competition high-end RISC architectures. For a glimpse of where the future of scale up architectures may lie, see SGI’s Altix UV announcement.

Posted by: Nik Simpson

Is That a Graphics Card in My Server?

For many years, one of the common factors in x86 servers has been a graphics subsystem characterized by the cheapest graphic chip the vendor could find to put on the motherboard. The logic (if you’ll forgive the pun) behind this was simple; you don’t buy servers to run graphics intensive applications, that’s what workstations are for! However, 2010 is shaping up to be different with two applications for graphics cards helping to make the case for graphics in the server.

The first of these; high performance compute clusters with graphics cards used to accelerate math calculations applications is not entirely new. NVIDIA has been pushing it’s CUDA program language graphics cards for at least 18 months with ATI/AMD expected to join the fray this year. However, the second application; accelerating graphics for virtual desktops is very new, with Microsoft’s announcements related to it’s RemoteFX protocol setting the stage for the use of high-performance graphics in servers (see here for more on RemoteFX). One of RemoteFX’s capabilities is the ability to use host-based graphics hardware acceleration to offload the graphics processing needed to support hundreds of Windows 7 virtual desktops (RemoteFX may support Vista as well, but I’d be surprised if they back-port the capability to Windows XP).

If Microsoft is successful with RemoteFX, then the real question will be how will graphics cards be integrated into today’s server platforms, and it’s not as simple as you might think. Today’s hi-end graphics cards have a number of requirements that largely rule them out for use in servers:

• Power requirements: Typical graphics cards require 150 watts or more of power, and that’s not going to be easy to satisfy in typical server designs where the power supplies tend to be small, and limited by the server form factor. It’s also quite common for the graphics card to require it’s own dedicated connection to the power supply, a feature not found on server power supplies.
• PCIe slot requirements: Graphics cards typically require a PCIe x16 slot on the motherboard, and these are rare on server motherboards where a pair of PCIe x8 slots is a more common configuration.
• Physical size: Graphics cards are big beasts, NVIDIA’s Tesla C2050 card is a full length, double width PCIe card, and that’s a lot of space to find in most servers.

So off-the-shelf graphics cards aren’t good fit for servers, and have components that aren’t needed at all such as the ability to send graphics out over VGA or DVI connections. If things are difficult for conventional rack mount servers, they are much worse for blades where the restrictions on power and physical space are even greater.

My bet is that we’ll need something a bit different if this type of hardware acceleration is going to take off. Here’s what I think we’ll see:

• A PCIe card designed for servers, with lower power requirements,a more reasonable form-factor, and a PCIe x8 (or even x4) interface.
• External, dedicated graphics engines connected via a PCIe ribbon cable to the server.  NVIDIA has already gone down this path for HPC applications with products like the Tesla S2050. This approach may work for blades as well as rack mount servers.
• Graphics cards in a blade form factor, i.e. a graphics card that takes up a blade slot and connects via PCIe over the blade chassis backplane.

The external graphics engine may also be a good place to make use of multi-root I/O virtualization that can share the graphics engine between multiple servers. Anyway this is certainly going to be an interesting space to watch as desktop virtualization becomes a mainstay of enterprise desktop strategies.

Posted by: Nik Simpson

Intel Takes the Wraps off Its Spring Collection

This morning Intel announced the first of its 2010 line of server processors, the Xeon 5600 family. The Xeon 5600 processors are a development of last year’s Xeon 5500, with several key differences:

• Manufactured on a 32nm process versus last years 45nm process
• Up to six cores and 12 MB of cache on each processor (50% greater than the Xeon 5500 family)
• Extended instruction set with new instructions to support AES-based cryptographic algorithms in hardware
• Reduced power consumption due to the ability to power down more unused areas of the processor

So far, the only performance numbers available are from the Intel press release and website, so we’ll have to see what the vendors announce. However, just the numbers released by Intel are pretty impressive, particularly on the VMmark benchmark where they cite a Cisco UCS B250M2 blade that reached 35.83 @ 26 tiles. To put that in perspective, compare the Xeon 5600 with it’s immediate predecessors the 5500 and 5400 families (results separated by 15 months):

CPU	VMmark	Power Envelope
Xeon X5470 3.33 GHz (4 cores/CPU)	9.15 @ 7 tiles	120 Watts
Xeon X5570 2.93 GHz (4 cores/CPU)	25.06 @ 17 tiles	95 Watts
Xeon X5680 3.33 GHz (6 cores/CPU)	35.83 @ 26 tiles	130 Watts

So in little more than a year, the performance of 2-socket Intel servers have increased by almost 4x! But it’s about more than just performance, many are now focusing on performance/watt and the improvements there are equally compelling. The following table compares estimated power costs for a sixteen server virtualization cluster built in 2008 versus a four server cluster using the latest Xeon X5680, assuming that the average power consumption is 400 Watts/server.

	Annual kW-hours	Power @ $0.1/kW-hour
16 servers @ 400 W	56,064	$5,604
4 servers @ 400 W	14,016	$1,401

Add on savings for cooling and the new servers would save in the order of $25,000 over three years just on the power bill. Add in reduced spending on hypervisor licensing etc, and the business case for replacing even relatively modern servers is pretty good. The case for replacing older servers is even more compelling. Assume each of the new servers can host virtual machine workloads equivalent to fifteen aging physical servers, so four servers can replace 60 physical servers:

	Annual kw-hours	Power @ $0.1/kW-hour
60 servers @ 400 W	210,240	$21,024
4 servers @ 400 W	14,016	$1,401

Add on savings for cooling and the new servers would save ~$120K over 3-years, more than enough to pay for themselves.

Posted by: Nik Simpson

Federal Trade Commission Sues Intel

Today, the FTC decided to go after Intel for what characterized as anti-competitive practices, this follows in the wake of the EU’s $1BN fine and Intel’s equally expensive settlement with AMD. Most of charges are similar to those already litigated in Europe, with the additional charge that Intel is trying to prevent NVIDIA and others from competing in the market for onboard graphics controllers. The surprise is a complaint relating to Intel’s compilers and how they intentionally hobbled AMD processors (see [1]). This charge surprises me on a number of fronts:

1. I don’t see how it’s Intel’s job to write compilers that produce optimal code for AMD processors, surely that’s AMD’s job.
2. The vast majority of commercial code on x86 architectures is compiled using either Microsoft’s Visual C++ or the GNU C compiler (AKA GCC) so even if the allegation is true, I don’t see it would impact commercial or open source software.
3. The only place where you would typically use Intel’s C compiler would be to produce highly optimized code for benchmarks, and surely nobody from AMD would use Intel’s compiler for this purpose.

I’m not surprised that the FTC has taken this step, having been at Intergraph when they fell out with Intel in the 1990s I’m well aware that Intel plays hardball. The question is whether what Intel has done is anti-competitive, or simply competitive. Anyway, this should make for an interesting story next year.

Posted by: Nik Simpson

What Now?

The EU’s decision to formally object to the Oracle/Sun acquisition is yet another twist in this long running saga, but it may prove the most damaging. Back when the EU first announced their intent to take a look at the deal, Larry Ellison made it pretty clear that that he wouldn’t spin-off MySQL to placate the EU (see Ellison: Oracle won't spin off MySQL), but if that was a bluff, then it’s been called. The decision puts the ball firmly back in Oracle’s court as they decide how to respond. I can see three choices, none of which are particularly attractive:

1. Oracle can walk away from the entire deal, which would leave Sun in an untenable position. If the deal fails, Sun has few good choices; sell itself at a knockdown price to a competitor assuming there are any buyers, or make massive cuts in its expenditure to better align with shrinking revenues. There may also be some legal implications to walking away as well, depending on the terms of the deal that Oracle struck with Sun back in April.
2. Oracle can acquiesce to the EU and sell off MySQL, but as the Motley Fool points out, they would be unlikely to get a fair price given the weakness of their bargaining position. Even if they can get a fair price, the big problem is “who could buy MySQL?” at a price north of one billion dollars. I think you can rule out the likes of Red Hat or Novell, and the more obvious potential suitors such as IBM and Microsoft would have the same problems that Oracle has with the EU.
3. Oracle can fight the EU, but this is a high risk strategy, Intel and Microsoft had little luck in this regard, and it’s hard to see Oracle fairing any better. If they lose then they’re back to options (1) and (2). Even if they win it will take time and the victory may be Pyrrhic, I’m not sure how much of Sun’s business will be viable at the end of a protracted legal fight!

Meanwhile, HP and IBM have declared open season on Sun’s customer base and there are no bag limits. Not pretty really, no good choices!

Posted by: Nik Simpson