Data Center Strategies

When All You Have is Hammer…

Everything looks like a nail, a truism that ASHRAE (The American Society of Heating, Refrigerating and Air-Conditioning Engineers) has just demonstrated in its advice to data center operators on how to achieve greater cooling efficiency in the data center (see ASHRAE Standard 90.1). In ASHRAE’s opinion, the way to achieve data center cooling efficiency is through the use of various economizer techniques (air-side, water-side …) to reduce the amount of energy used by the cooling plant. Unfortunately, this is a rather narrow view of the problem, focused on the stuff that ASHRAE members do best, i.e. air-conditioning and chilled water plants. The ASHRAE standard has now drawn a universal thumbs down from some of the largest data center operators (Google, Microsoft, Amazon, Digital Realty Trust, DuPont Fabros Technology and Nokia) who issued a joint statement urging ASHRAE to re-think it’s position.

The truth is that economizer are just one approach to achieving more efficient energy usage and the best approach will vary based on a host of factors. For example, one company might favor widespread adoption of server virtualization as a way to reduce energy consumption, while another might be able to scavenge energy from the waste heat produced by the data center and use it heat buildings. Both approaches lead to more efficient use of energy without requiring the use of economizers. The problem with ASHRAE’s approach is that there is a danger it will get built into building construction codes and potentially restrict innovation in the data center. James Hamilton’s blog entry on the subject (see here) does an excellent job of describing the problems of such a restrictive definition.

Posted by: Nik Simpson

A Picture is Worth a Thousand Words on HA/DR

If there's one constant in an analyst's life, it's travel. When you travel as much as we do, you begin to become very familiar with airports. I make it a habit to observe the IT equipment in every airport (terminal consoles, wireless access points, etc) because I like to get a sense of what the airport IT department is dealing with and how up to date they are. You can tell a lot about the airport based on their IT infrastructure.

Well, the other day I was walking through a well-known airport (I won't say which one), and I snapped a picture of what is probably one of the worst IT blunders I've ever seen. But the really funny part is the picture is -- in a microcosm -- the way we all think of IT high availability. Here's the pic:

 

So yes, that is a server rack, full of running IT equipment (red circle), in the public exit pathway, with no security guard in sight (green circle)....and an "unplug" sign in the background (yellow circle). Too funny!

Got me to thinking...

Isn't this the way we treat HA/DR in many of the data centers today? We put the IT equipment in a risky situation, accessible to lots of people (usually via a network), with no security measures in place, and signs that tell you exactly how to bring it down.

HA/DR should not be an afterthought. As Richard Jones (our service director) would say "Executive management must take fiduciary responsibility for HA and DR. It must be driven from the top, practiced, and reviewed on a regular basis".

He's right...

Remember, you can chat with Richard, myself, Chris Wolf, or Nik Simpson regarding HA/DR, virtualization, cloud computing, data center cost initiatives at Catalyst Europe in Prague April 19th -22nd. The promotion code for the early bird registration price of €995 is "INSIDER".

[posted by: Drue Reeves]

Burton Group Institute - Data Center Workshops

Last month Burton Group launched, Burton Group Institute, a new on-demand training resource for enterprise technologists. The collection includes data center operations workshops that feature real world experiences from Burton Group analysts and consultants for enterprise technologists to utilize technologies challenging IT. Topics span several technologies, but the short list of data center training courses includes:

• Data Center Economics
• Data Center Architecture
• Advanced Server Virtualization

The training library is a collection of Burton Group workshop styled presentations focusing on a range of technologies and implementation strategies that include: identity management, security and risk management, networks and telecom, enterprise social networking, and content and collaboration. As an extra benefit iaap (International Association of Privacy Professionals) credits are available for completion of select Burton Group Institute workshops.

The workshops are delivered in a video format with integrated slides and audio and include presenter/audience interaction for further details on the topic. An example of the workshop can be viewed featuring Chris Wolf and Richard Jones presenting “Advanced Server Virtualization Workshop.”

As a reader of the Data Center Blog, Burton Group would like to offer a 20% discount with the coupon code DCB38.

Which Reminds Me…

I was just reading an article about IT’s desire to better understand costs (see here). The article reminded me that back in July, I posted (here) an invitation to a survey that Burton Group was conducting as part of my ongoing research on data center cost models. At the time I promised to share some of the results on the blog. Before I go into the details of the survey I’d like to link to some additional information that may be helpful.

1. If you are Burton Group customer, you can download “Counting the Cost of the Elephant in the Data Center”  which contains a more detailed look at cost modeling and also includes a sample spreadsheet that can be used a starting point for looking at infrastructure costs (i.e. what does it cost each year to run a server).
2. There is a recording of my 1/2 day workshop on data center economics from Catalyst 2009 which is available to anybody with a browser. In the interests of full disclosure, this is $199 to purchase.

And while I’ve got your attention, if anybody is interested in talking to me about data center cost models and how they are used in your business, please get in touch via email to nsimpson@burtongroup.com.

Anyway, on the results of the survey…

The 80+ respondents to the study were well distributed across industry segments and company size:

• Participants in all major IT sectors including; education, government, pharmaceuticals, aerospace, energy and consulting.

• Companies ranging in size from less than $50 million to greater than $10 billion in annual revenues.

Out of all the companies surveyed, less than 25% had a comprehensive model for their data center costs. A similar number had no cost model of any kind. The rest of the companies surveyed had some form of cost model that was limited in scope and application.

Of the companies that did not maintain a cost model or only maintained a limited model, the key problems were either lack of motivation (i.e. costs are not a major part of the decision making process) or difficulty in collecting the necessary information because of it’s spread across multiple fiefdoms within the organization (i.e. facilities, purchasing, human resources, etc.)

As to the contents of the data center cost model, at least 25% of those using a cost model did not include any information about power, cooling, or facilities costs. This represents a serious shortcoming as these costs are a significant part of the overall IT costs.

Another point that emerged is that the old idea that a data center should be expected to last twenty years is no longer valid. Of the survey respondents, more than 85% had built or upgraded a data center since the year 2000, with 45% having done the work in the last year to 18 months. Yet 65% of the respondents expect to outgrow their current facility within five years. The major factors dictating data center life were limits on floor space and power distribution, with cooling coming in a distant third.

The biggest gap in terms of data required for a data center cost model was in the area of power. Only about 25% of respondents could track power consumption down to the individual piece of IT equipment, rest had varying levels of insight into their power usage. Interestingly this 25% figure was mirrored almost perfectly by the number of respondents who had calculated PUE for their data center.

Another weakness common to the majority of respondents was the availability of detailed purchase price information for equipment in the data center. Roughly 20% of respondents had the purchase price information in the corresponding Configuration and Management Database (CMDB) entry for the equipment. Another 23% had a spreadsheet that has to be maintained by somebody in the IT organization. But over 50% of respondents would have to track down the information from other sources in the company such as a purchasing department.

Overall, the survey indicated that many IT organizations are not in a position to fully understand the costs of running their data center. The lack of a complete understanding of data center costs creates a potentially dangerous hole in the IT decision making process, especially because IT decisions are increasingly cost driven.

Posted by: Nik Simpson

ASHRAE says “turn it up”

I’ve just finished writing a report on data center cooling (which should be published later in the quarter), and one of the recommendations was that data center operators should set the temperature to at least 77^F (25^C), per the recommendations of the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE), as a way to reduce energy consumption in the data center. So I was interested to note that ASHRAE has just changed that recommendation to 80.6^F (27^C) measured at the inlet of the equipment in the rack (coverage here, here, and here).

But before you go and adjust the thermostat in your data center, there are a number of potential impacts that need to be understood:

• Fans in equipment in the racks will have to work harder to keep the equipment temperature in range, so they will be noisier. More importantly, more active fans in equipment will move some of the power load from the HVAC equipment to the data center power distribution network. That’s fine if you have ample headroom in power delivery to the racks, but if you are operating at the edge with respect to power delivery then you might start tripping power breakers.
  
• The hot-aisle will get hotter. If you assume that the temperature difference between the hot- and cold-aisle is 55^F (13^C), then hot-aisle is going to be a toasty 135^F (57^C), not what you would a call a pleasant working environment if you have to work at the back of an equipment rack.
  
• The recommendation is for inlet temperatures of IT equipment, and these will vary depending on where the equipment is located in the rack. So wandering around the cold-aisle with a thermometer isn’t good enough, you need to place several temperature sensors into the racks to get an understanding of the temperature gradient from bottom to the top of the rack.
  
• Increasing data center temperatures will allow for a more humid environment which may lead to water condensing in the computer room air conditioners, reducing their efficiency.

Perhaps the most important concern for IT  will be the perception that hotter equipment fails more frequently than cooler equipment. On the surface, this is correct, if you let equipment overheat, then it’s more likely to fail. But the ASHRAE recommendation isn’t intended to make the equipment run hotter. The increase in cold-aisle temperature will be counteracted by the fans in the equipment so there should be no net increase in the internal temperature.

But the question you have to ask at the end of the day is how much difference will this make to overall data center energy consumption? The answer to that question depends on what you’re current operating temperatures is. For example if you are at the old ASHRAE recommendation of ~77^F (25^C) then the answer is “not much”. But if you are operating at temperatures substantially below the new recommendation (which is surprisingly common) then you really need to take a hard look at how you operate your data center, because modern IT equipment is much more tolerant of high temperatures than we give it credit for. Consider the following table which lists the operating temperature range for a sample of common IT equipment:

Device	Temperature range
Dell PowerEdge r805	50F-95F (10C-35C)
Cisco Nexus 5000	32F-104F (0C-40C)
Sun SPARC Enterprise M9000	41F-89.6F (5C-32C)
NetApp FAS 6000	50F-104F (10C-40C)

Add the fact that mean time between failure (MTBF) numbers are calculated at the high end of the range (i.e. 95^F or 35^C for the PowerEdge r805) and it’s clear that 80^F isn't extreme as far as the equipment is concerned

Posted by: Nik Simpson

Putting Your Data Center on a Diet

So Christmas is over and we are all looking somewhat ruefully at our waist lines and hoping to get back in shape before the summer rolls around. Data center operators have another problem, how to cut some of the fat out of their data centers and get their energy budget in shape for the new year. Just this week I was asked the question “How can I trim a million KW-hours energy consumption this year?” That sounds like a daunting, but worthy exercise, a million KW-hours sounds like a huge number, but at 8-cents a kilowatt-hour trimming 1 million kilowatt-hours would save $80,000 dollars in the first year (of course, price/KW-hour varies considerably by location, so your mileage may vary.)

The first point to note is that while the number seems huge, that simply reflects the size of the customer’s data center, the real question is “what changes will have the most impact on energy consumption?” So here’s some tips on how you can reduce energy consumption in the data center:

1. Reduce the amount of equipment in use: The quickest way to significant savings is to cut back the amount of equipment in use through technologies such as server virtualization. Let's assume that 1/3rd of the servers in a data center are suitable candidates for virtualization and that on average a 5:1 consolidation ratio is possible (and that’s a conservative estimate for the consolidation ratio.) A datacenter with a thousand servers would be able to virtualize 333 of them, replacing those 333 physical servers with just 67, a net reduction of  266 physical servers. Now lets assume that these original servers were 1U pizza-box servers each consuming 300 watts of power, and that the new servers are bigger (to accommodate the extra memory, processors etc) and consume 600 watts of power, the net power saving is almost 60 kilowatts. Such an approach would also reduce power consumption for cooling by a similar margin (assuming 1:1: ratio between power consumed by equipment and power consumed by cooling.) This step alone would save a million KW-hours, throw in some improved storage management practices and deduplication for data and the energy savings quickly pile up.
2. Get rid of old equipment: Some data centers continue to use equipment until it finally dies, that can mean servers that are 5 or even 10 years old (I've seen even worse cases). These servers are energy and space hogs and deliver very little work for the power consumed compared to a more modern system, especially if the more modern system is combined with server virtualization. 
3. Improve power distribution: This can take the form of a number of improvements and upgrades. For example old UPS equipment is substantially less efficient (i.e. wastes more energy) than modern UPS systems, replacing the old UPS could yield a 5-10% reduction in energy consumption. Also minimize the number of voltage conversions between the power as delivered by the utility and the power consumed by the equipment, i.e. bring high-voltage power as close to the racks as possible.  Data centers in the US still using 120V AC to power equipment in the racks, need to take a long hard look at switching to 240V AC, the equipment doesn’t care and it will save energy and simplify the wiring.
4. Cooling improvements: Air-side economizers can deliver big savings, especially in more temperate climates. Assume roughly half the energy consumed by the data center is used for cooling, and that the airside economizer can function for half the year, then potential savings in energy consumption are pretty obvious. Also improve hot-aisle/cold aisle separation or implement fully ducted hot-air removal from racks can increase cooling efficiency inside the data center.

The big lesson here is that the quickest payback for reducing energy consumption is to reduce the amount of equipment in the data center and opportunity presented by maturing server virtualization technology is too good to pass up! The downside is that to save money in the long term, you’ll have to spend some money in the short term.

Posted by: Nik Simpson

Dell and EMC renew their vows

Today Dell and EMC announced a new 5 year agreement to remain pals, hold hands and sell storage together.

The previous 5 year engagement of Dell and EMC has helped both Dell access EMC enterprise customers, and EMC expand into the SMB/SME space, Dell's traditional strong hold. With the renewal of the Dell - EMC agreement for the next 5 years, expect more of the same.

As it was 5 years ago, Dell is clearly avoiding the heavy investment necessary to develop enterprise class storage and is instead leveraging EMC capabilities in that space. As during the previous agreement, some storage product overlaps exist and are not likely to go away. EMC is the development muscle, Dell is the marketing muscle.

Dell's EqualLogic products and recently announce D2D products using Quantum's deduplication engine will overlap the low end of the EMC offerings - but these issues have existed in the past and were workable from a customer contact sharing point of view.

Will this relationship be more of the same or will both EMC and Dell collaborate on the implementation and offering of cloud based systems? I think the answer is yes as indicated in the announcement but who will take the top position when selling into large accounts is yet to be seen.

Is this relationship good for customers? Potentially for those interested in having a single vendor to deal with on purchasing decisions, but once the equipment lands in a data center, when the going gets tough each vendor will have to be dealt with. Certainly Dell's pressure to commoditize storage, reducing pricing to the bone is a good thing, but that's not new. In the past Dell had a very low touch model when it came to EMC products - little more than slapping on a logo and shipping it. I see no change there.

Watch for Clariion and EqualLogic sales folks arm wrestling in the parking lot over who's products get pitched to new accounts. Either way, no matter the outcome, both companies win. 

Its just too bad that Dell is not up to doing their own thing in the enterprise storage space. I suppose in these economic times there's just not room for another player.

Posted by Gene Ruth

Not Your Father's Data Center

When you decide to build a data center that holds 500,000+ servers and consume 120 MW of power it makes sense to think really hard about what that data center should look like. Do you take a conventional design and simply scale it up, i.e. a big empty box of a building filled with rows of racks that seem “to stretch out to the crack of doom”? Or do you start from first principles and question almost every piece of conventional wisdom about data center design including power distribution, cooling, and availability?

Interestingly, Microsoft chose the latter approach, and in the process, they have created their Generation 4 Modular Data Centers (I’ll call them G4M for short) which look nothing like a conventional data center. The details of their G4M Data Centers were publicized this week in a very interesting blog posting from Mike Manos at Microsoft. The key aspect of G4M data center is that it is completely containerized, not just the servers but everything, power, cooling etc all come in pre-built standard container configurations. The containers are deployed in the open air (think “trailer park” data center) and new modules can be added rapidly to meet demand. Different configurations of containers can be used to deal with different availability requirements ranging from very low (relying on air cooling and no redundant power) all the way to full “tier IV” availability with redundant everything. Just to give you an idea of how different this data center looks, here’s one of the graphics from the blog entry…

So in parting, if you are at all interested in the future of data center design, go read the blog!

 

--

Posted by: Nik Simpson

Intel's Enterprise X25-E SSD Performance

The last time I blogged about SSD performance I had a Intel MLC based SSD,  intended mostly for laptop or read intensive applications. Looking back at that blog, I reported pretty decent performance numbers with the X25-M Intel drive.

Christmas came early this year for me - I recently received several Intel X25-E enterprise SLC SSDs for evaluation. As an analyst I normally don't get the chance or have the time to get down and dirty, but this opportunity was too good to pass up. Besides, my career has been spent developing products and diving into details. Its hard to leave that legacy behind while looking at a box of SSDs just begging to be run through their paces. As, apparently, the only analyst to receive these drives, I felt obligated to take them for a ride and see if my previous enthusiasm was justified.

Lets get right to it:

I ran tests, using IOMeter on a 2.66GHz quad Core 2 Intel CPU, 45nm, 12MB L2 cache, with a 1333 front side bus, 4G memory, SATA II 300MB/s ports. The tests were run on a single SSD, both the M and E version, as well as the E version in a 4 drive RAID 5 configuration. Unfortunately, I don't have a decent RAID adapter (hint) so I used the onboard NVidia MediaShield RAID function. 

While I have more data, for simplicity I've plotted IO transaction rates for 512, 4096 and 32768 block sizes for random reads and writes. Using all random reads and writes provides significant stress on the SSD and is a good reference point for comparison to HDD performance.

Take a look at the graph:

In the graph, I plot the transactional performance of the X25-M, X25-E, X25-E in RAID 5 and a SATA HDD as a function of block size.

Its worth pointing out that the tests I ran are far from real world, but they do highlight performance under extreme conditions. Measuring performance can be a tricky business, but I believe the tests I’ve run are a good reference point and easily repeatable – except for a weirdness that I’ll point out in a few…

Take a close look at the results. The performance for the X25-E is very compelling. For random reads, the X25-E's (as a single drive and RAIDed) performance tops out around 12,000 IOPs as does the X25-M. You'll need to look closely to see the plotted lines as they overlap at the top of the graph. I suspect that the drives are capable of much more and are bottlenecked by the upstream motherboard and driver stack limitations. I didn't spend much time tuning my system so I suspect that the read number could be far higher. In any case, the values leave the poor SATA HDD in the dust.

The random write performance is equally compelling for the X25-E, operating far faster than the X25-M and making the HDD look like a stone.

The "X25-E RAID5 - Write" test, using 4 disks, stands out like a turkey in a chicken ranch. The RAID performance is actually worse than a single disk. Hmm, why is that?

When doing writes in a RAID 5 configuration, an XOR operation is required (not so when reading). Since the RAID function on my motherboard is driver based, no doubt my system is the bottleneck. This limitation does point out the stress placed on RAID adapters when dealing with high transaction rate devices. Most RAID adapters are best suited to dealing with single threaded devices (e.g. hard disks) operating at hundreds of IOPs not thousands of IOPs as SSDs can do. I'll have to wait to get my hands on a decent RAID adapter (hint number 2) before this can be explored further.

but there is some weirdness, look at the following graph:

As I prepared to collect performance data, I ran the random 4k block write test a few times. I noticed that the result varied over time and depended on the state of the SSD before the test was run. That's weird. With a hard disk, performance is very predictable and constant over time. Apparently not so for an SSD. I think we knew this but the graph proves the point. Before the test, I had conditioned the X25-E with 64K random block writes. Not scientific, but the results shown in the above graph are curious none the less. The random write performance varied four to one over the period of 30 minutes where I collected performance data at 5 minute intervals.

While much more performance testing and analysis is needed, such as the examination of latency values, I'll leave that to others with more time on their hands.... 

The performance of the Intel X25-E is remarkable compared to a hard disk. Unfortunately, the unexpected performance variability was a surprise and adds a new dimension to interpreting performance data.

Oh, and btw, the X25-E hardly got warm to the touch throughout the testing. So while I don't have a way to measure power, the X25-E clearly uses far less power than my SATA HDD that I can use as a donut warmer.

So this brings up a good point, and I'll end the blog on this note:

The industry needs a standard way to test SSDs. Period.

Please feel free to comment. 

Posted by Gene Ruth

SSDs based subsystems are getting interesting...proceed carefully

Two interesting things happened in the computer storage biz on Monday: Sun announced their new "Amber road" storage products and Violin Memory, a small startup, introduced a high performance pizza box storage appliance. Both these products are leveraging NAND flash SSD technology. Both in interesting ways.

BAM, new era...sorry couldn't resist.

It would be easy to go all gaga over these products but caution is advised. Both are innovative in their own right, both challenge the status quo of the data center class storage system business. These are innovative products. No matter what my comments to follow are, keep this in mind: these products are integrating new technologies in new ways and therefore, regardless of the technical prowess of the suppliers, caution should be exercised before inserting these products into a data center on a large scale.

Try'em to verify'em.

In my past blogs about SSDs, I've mentioned that flash SSDs are about transactional performance not capacity.  I stated that capacity will have to wait until flash memory density increases and pricing decreases dramatically - we will be waiting for awhile. So, I went on to suggest that a storage system based on SSDs, to be well rounded, must offer high performance but solve any capacity shortfall by including high capacity SATA disks. To make that workable, some magic is needed to move data around appropriately to match the data's dynamics.

That's what the new Sun storage product claims to do.

Sun, by leveraging their open source ZFS file system, with some crucial tweaks, marries SSD technology for performance and SATA hard disk technology for capacity. Compared to an all HDD system, Sun claims up to 3x read transactional performance, 5x less power, at about the same cost.

Not bad.

Turns out Sun uses SSDs tailored both for write performance to handle an internal logging function and then read optimized SSD to act essentially as a really large cache. All wrapped around their open-source ZFS file system hidden within the storage subsystem. Sun is demonstrating technology leadership by tightly integrating SSD technology into a complete storage subsystem - yes others have done a pluggable replacement for a HDD, but that's fairly obvious and less then optimal. And there have also been some demonstrations such as IBM's Quicksilver science experiment based on Fusion-io, but no significant shipping product.

There are certainly other nice features in the Sun "Amber Road" products such as an improved management interface, analytics and such but blah, blah, blah. The industry expects those things, they are the price of entry and are the nuts and bolts of any system. Of course you need good pricing, support, quality and the like - that's all good, but its expected. If a vendor does not live up to the expectation - they get voted off the island - fast.

Sun has a window of opportunity here to make some market share gains, before other major vendors show up to the party. Its on you Sun. Price the product to move, forget premium pricing. Get support right, and accounts will be won. Don't forget: others will arrive at the SSD party soon as well. Grab market share while you can. Show customers how Sun is setting a new price-performance bar only reachable with SSD technology. Game on. Watch your back, I expect strong competition in the SSD space over time, the vendors who don't react are no worry since they will not survive in the long run.

Back to Violin. Violin introduced a 2u (that's about 2 pizza boxes) flash based storage appliance. While this is not an holistic performance and capacity solution, it does offer an excellent example of what happens when you take the "D" out of SSD. No longer constrained by the HDD form factor, the Violin 1010 uses memory card like flash modules that plug into a motherboard. As part of the design, there is a RAID like redundancy for the flash modules with hot plug support in case one fails - but I gotta admit, I wouldn't touch the thing while its running. Find an intern to do it and if things go wrong make the intern the scape goat.

For those running transactional applications, like a database, this product could be a godsend. But I fear the pricing may be too high and you do need to accept working with a newbie in the space. The Violin 1010 attaches directly to PCIe (it also does FC and Ethernet) . PCIe is needed to achieve high iops - fibre channel and scsi in general add too much latency but that's the subject of a future blog.

The Violin product is interesting on its own but consider: Combine the Violin 1010 with a bunch of 1TB SATA hard disks and Sun's open storage software with ZFS. Add in a few tweaks for auto-tiering, do some soul searching about pricing and you could have an awesome product. 

So where is all this SSD technology going? Into data centers, sooner than anyone has imagined.