IDEAS Insights

Building Large SMP Systems with Virtualization

As a former designer of big iron, scale-up computing remains dear to my heart. Some specialized applications still perform best on large, shared memory, tightly-coupled SMP servers. Even for workloads that need only fractions of a large server, the opportunity to share "headroom" allows multiple applications to reside on the same large SMP, typically via virtualization partitioning. These systems can run at a much higher utilization percentage by pooling the resources that are "reserved for peaks" than if each workload were deployed on separate, individual servers. That flexible pool of hardware resources is often just as valuable as the ability to host a large, monolithic application. 

Unfortunately, the sophisticated scale-up designs created by innovative engineers must amortize their development expense across a relatively small volume of servers, compared to the ubiquitous industry-standard x86 platforms shipping as near-commodities. While I remain convinced that large, scale-up, shared memory servers are crucial for many computing environments, I acknowledge that x86 implementations offer compelling price/performance.

Is there a way to combine the best of both attributes? That is, can large shared memory servers be created from commodity-priced, industry-standard building blocks? Can some "glue," be it hardware or software, allow a collection of small, inexpensive servers to appear to perform as if a large, shared memory system? This week, two firms, 3Leaf Systems and ScaleMP, have introduced alternatives they each believe will address significant portions of the market for economical scale-up systems by coupling standard x86 servers.

Both firms unveiled their solutions just before SC09, November's annual USA venue showcasing innovative high performance computing solutions. Why SC09 and not a trade show targeting the market for broader commercial computing? Attendees of the supercomputing conference usually have very demanding compute requirements, often needing very large servers, but they are also typically budget-constrained. The promise of a large, shared memory, server at commodity price points understandably excites these users.   

Of course, a virtual "appears like a very large shared memory SMP" server will not perform identically to an actual large hardware-optimized SMP server. Therein lies the difference between 3Leaf and ScaleMP solutions. The 3Leaf platform employs a 3Leaf-designed ASIC hardware chip that melds together the memory space of multiple x86 motherboards (currently AMD Opteron, but planning Xeon support with the next iteration of Intel QuickPath) to deliver low-latency shared memory reminiscent of SMPs. ScaleMP, on the other hand, forgoes the expense of unique hardware, and accomplishes coherent shared memory via software. Admittedly, 3Leaf's hardware memory coherence is faster than ScaleMP's software mechanism, but ScaleMP claims its pre-fetch and local caching algorithms deliver the desired low latency appearance without the expense of specialized hardware.

High Performance Computing (HPC) users are acutely aware that the road promising affordable high performance is littered with firms who dreamed of delivering champagne performance at beer budgets. NUMA, non uniform memory access, is at the heart of most "affordable" scale-up designs. Flat, uniform memory access, designs prove to be costly as systems grow larger and larger. The key is to have NUMA's apparent non-uniformity seem as small as possible. KSR in the early 90s, Convex, SGI's Distributed Shared Memory, and other Scalable Coherent Interface (SCI) users (such as Sequent and Data General) all promised the performance of sophisticated, optimized SMPs at commodity server prices. Fairly recently, Virtual Iron attempted a hypervisor that could join multiple smaller servers into a virtual larger SMP, but it eventually abandoned that effort to focus only on applying virtualization for partitioning (the company has since been absorbed into Oracle.)

Clearly, a virtualized NUMA SMP will not adequately substitute for a real SMP for all environments. The key question is which workloads will be satisfied by ScaleMP's software virtualization, which will benefit from 3Leaf's hardware memory coherence, and which applications demand the more uniform memory access of true large SMP implementations.  All current operating systems have evolved to understand how to efficiently keep processing and associated memory physically close to each other on NUMA platforms. However, some workloads have less sensitivity to NUMA than others. Targeting an announcement at the supercomputing conference recognizes that many HPC applications depend less on the shared memory updating that plagued early NUMA attempts to address general purpose commercial computing. The inherent parallelization of HPC applications designed to exploit large infrastructures makes them better prepared to run well on 3Leaf and ScaleMP platforms. Optimized large SMP hardware designs will continue to offer the best performance for applications that exploit multiple compute engines and large shared memory. However, other application environments may benefit from hardware and/or software mechanisms that lash together lower cost commodity servers to create the virtual appearance of a large system, yet with very attractive pricing.

We believe there will be growing customer demand for virtualized SMP solutions like 3Leaf and ScaleMP, which offer lower cost, even with the tradeoff of potentially lower performance.  3Leaf Systems' hardware ASIC is impressive in its use of InfiniBand switching as a substitute for custom SMP backplanes. The low latencies of today's InfiniBand implementations are likely to provide an adequate SMP-appearance for many applications. ScaleMP's software-only solution may not offer the low latencies of 3Leaf, but it is likely to attract a number of customers who desire large memory accessibility at a very economical cost, albeit with somewhat slower latencies. We predict that 3Leaf and ScaleMP will see first adoption among HPC users, but each will eventually carve out a niche among commercial application users. The optimized design of monolithic large SMPs will still be needed by many customers, but virtualized SMPs, aggregated from low cost commodity servers, will offer an attractive price/performance alternative for a variety of workloads.

Will SAS Replace FC in Disk Array Making?

Serial-Attached SCSI (SAS) technology has been available for a few years. So far, SAS has proven itself to be a robust technology through various product implementations; there have been no major complaints from storage end-users. It seems like SAS-based products have fulfilled the design proposition of the SAS protocol, which is to overcome the I/O bus limitation of the Parallel-SCSI protocol in order to achieve higher performance and scalability.Hence, many customers are now wondering when they will be able to replace their costly Fibre Channel (FC) disks with more affordable SAS disks. For some of them, the answer is today, and for some of them, the answer is indefinite.

Customers with direct-attached external RAID disk systems can migrate from FC-based disk enclosures to SAS-based disk enclosures by swapping the FC HBAs inside the server with SAS host adapters (for JBOD) or SAS RAID cards. FC HBAs typically do not offer RAID capability and often rely on host-based software RAID. SAS JBOD adapters are available for those users who prefer software RAID. However, more users today prefer to deploy hardware RAID (host adapter with RAID-processing ASIC) to avoid the performance impact of RAID processing on the host.

Customers with storage servers, such as Windows- and Solaris-based network storage servers, can enjoy SAS today as well. Storage servers are made from industry-standard servers running storage-serving operating systems. They contain direct-attached storage subsystems, which can be SAS-based enclosures. For example, the Sun Storage 7000 series is one commercial-grade storage server implementation that employs SAS-based disk subsystems. Note that the Sun 7000 platform does not use hardware RAID, but implements the Solaris ZFS zRAID.

Customers with network storage systems, known as SAN platforms, may face a longer, indefinite FC to SAS transition cycle. While all the entry-level network storage platforms from major storage vendors – such as the Dell|EMC AX4, HP MSA2000, HDS SMS100, IBM DS3000, and NetApp FAS2000 systems – have already transitioned to SAS-based disk subsystems, the midrange and high-end storage platforms from these vendors are still using FC-based disk subsystems. The only exception is the midrange HDS AMS2000 line, which provides a large number of I/O slots to populate SAS adapters and connect to SAS-based disk enclosures. To date, the AMS2000 series is the only midrange network storage platform from a major storage vendor that supports FC host connectivity with a SAS back end. None of the high-end platforms have transitioned to a SAS back end.

Vendors have their reasons for not switching to a SAS back end. Midrange and high-end storage platforms from major storage vendors such as EMC, HDS (resold by HP and Sun), and IBM commonly employ a switched-disk subsystem design. A switched design provides non-blocking, point-to-point connection between disks and controllers. Engineers argue that a switched connection is more advanced than a serial connection. Serial connections can potentially become congested and cause performance degradation when they are shared by heavy I/O operations. Hence, it is very likely that the midrange and high-end storage platforms from these vendors will retain their switched back-end architectures for the foreseeable future. However, the HDS AMS2000 midrange arrays producedindustry-leading SPC-1 benchmark results, and did not show any signs of I/O congestion for the test workload.

Nevertheless, SAS is without a doubt more affordable than FC. The traditional FC SAN platforms are increasingly being challenged on the cost front by storage servers based on industry-standard technologies. As more and more customers are attracted to the affordability of SAS-based disk systems, the leading storage vendors may have no choice but to switch to a SAS-based back end. However, it is also possible to implement a SAS switch, instead of an FC switch, to have the best of both worlds. Such a SAS switch is available today – just not in a storage product implementation. Take for example, the HP ProLiant BL SAS switch. Therefore, while the day that SAS disks replace FC disks in high-end network storage platforms is still unknown, it may not be too far away.

End User Interest High at Virtualization Forum

Yesterday I attended VMware’s Virtualization Forum 2009 held at the Sydney Convention & Exhibition Centre, Darling Harbour. Virtualization in its various forms has been a hot topic for years and looking at the growth of this event in terms of the number of attendees, is probably an independent verification of the interest in virtualization in the market with real users. Even in these tough economic times.

In his keynote address, Paul Harapin, Vice President VMware Australia and New Zealand, commented that this was the fourth annual event of the Virtualization Forum, which began with just 350 attendees a few years ago. This year’s event attracted 2,500 attendees. This may not be a big number by international standards, but Mr. Harapin noted that by VMware’s measurements, virtualization penetration in Australia is higher than any other country. Other high-adopters are Holland and Israel. This doesn’t come as a great surprise because Australia has had a history of adopting new technologies. In the 80s for example, Australia had a high market penetration of what was then a hot new technology, the fax machine. In fact, probably because of this predisposition for all things new, Australia has a reputation as a test market for many companies where they will trial new products and/or packaging, before rolling them out around the world.

Carl Eschenbach, EVP of Worldwide Field Operations was the second keynote speaker and he took the audience through VMware’s product portfolios and directions for the company.  VMware claims around 150,000 VMware customers globally and although initially, the motivation for adopting virtualization is often to save costs, according to Mr. Eschenbach he is being told by CFOs that greater business agility is the key motivator, especially at the big end of town.

It wouldn’t have been an event on Virtualization without “Cloud” being mentioned and so the audience wasn’t disappointed. In fact, Cloud Bingo was alive and well on the Twitter feeds. (#vForum #vForumAust & vForum). Having said that I don’t think VMware overplayed its hand with Cloud for the event, given the hype out there, it has to be addressed in such a forum.

VMware’s definition of Cloud is very broad, being simply “Deliver IT as a Service”. In a subsequent discussion with Dr. Stephen Herrod, CTO at VMware, he pointed to a VMware survey that found 15% or respondents had adopted Cloud in some form and 26% were evaluating. Although, it wasn’t clear what ‘adopting Cloud in some form’, means exactly.

Beyond Cloud a clear message from the event was that VMware is getting involved up and down the stack in middleware and now applications, with its recent acquisition of SpringSource.  In fact, VMware appears to have interests everywhere except for the hardware and operating system layers. Mr. Herrod was quite clear however, that it is VMware’s intention to stay away from the operating system and for VMware to add value virtualizing elsewhere.

The event saw a strong push to message Desktop Virtualization and Mr. Eschenbach’s believes that 2010 is the tipping point for desktop virtualization. This is because in his opinion many factors are coming together, such as the increased interest in virtualization generally and the expected Windows 7 inspired desktop refresh. I would add to that the 3G telecommunications rollouts around the world means that mobile devices and applications give viable alternatives for users to still be productive when not at their virtualized PC and fixed line broadband connection.

The business justification is certainly there. If you think servers have low utilization, desktops generally have an even lower utilization. Because of this low relatively lower utilization, potential consolidation ratios for PCs are even higher than desktops. So, where you might be able to consolidate say 4 to 8 physical servers onto a virtual platform, the desktop consolidation ratio can be as high as 10 or 20 to 1. Desktop virtualization has always had what seemed good business cases, but for one reason or another it has never taken off in a big way.
 
I would be interested in your views on the future of Desktop Virtualization. Is this something you are now considering and why now? Or, do you still see impediments to the take up of this technology?

Beyond the keynotes, the Forum had an exhibition hall with around 60 vendors marketing their offerings including many well known brands and not so well known organizations. Feedback was generally positive and response in terms of the number of delegates shows that even in these difficult financial times, whether they are committed VMware users or novices interested in understanding more about moving into the world of virtualization, there is still interest from users where they perceive there is value to be had.

Did you attend the event? If so, what did you think of it? Let us know by posting a response to this blog.

Gary Burgess
Senior Vice President, Research & Operations
 Follow Gary on Twitter

IBM’s Overlapping Design Teams Ensure Flow of New Servers

IBM's recent earnings release - lower revenues but higher earnings - brings to mind an old saying "You can't save your way to prosperity."  Perhaps some cutbacks may turn out to be short-sighted, but those that challenge/change business fundamentals may yield long lasting benefit.

Locales suffering IBM's "stealth layoffs," under the guise of "resource rebalancing," may bemoan the shift of employment offshore.  But, the merging of "z," "p," and "i" design engineers allows more frequent product refreshes across all server platforms.  

Designing a complex system does take many years.  Mainframes used to see significant updates every 5+ years, but now that mainframes leverage POWER building blocks, IBM has indicated that its next mainframe update is targeted for 2010, just a couple years after its 2008 introduction of z10.

Back in the pre-90s mainframe heyday, IBM had multiple overlapped design teams – while one was wrapping up a design, another was in the midst of design and yet another was exploring alternatives for the next, future design.  As one design finished, engineers were shifted onto a future design.  The overlapped teams meant that a new system could be released twice as often than if there were only a single team.  Over the years, cost cutting reduced mainframe engineers from 2.5 teams to something like 1.5 teams, which lengthened the introduction of new architecture.

That overlapping team approach is still used for complex chip designs like IBM's POWER.  The POWER5/5+ team had a different crew of engineers than the POWER6/6+ team.  The POWER7 team picked up some of the POWER5 guys while the POWER6/6+ guys were still hard at work.  POWER8 is now the skeleton exploration team.  Sure, there is a cadre of gurus who oversee all, but the grunt work involves overlapping teams of engineers.  Like clockwork, the overlapped POWER teams have introduced new designs every 3 years (POWER4=2001, POWER5=2004, POWER6=2007, POWER7 promised for 2010) with speed-bump midlife-kickers in between.

The mainframe continues to support an instruction set architecture (ISA) that has evolved over the past 40+ years while steadily maintaining backwards compatibility.  It is clearly not the same ISA as POWER, with its RISC roots.  But, ISA aside, there are a lot of common functions – cache hierarchies, memory interfaces, even execution units (IEEE floating point, Decimal floating point, etc.) – that can be leveraged between POWER RISC and mainframe CISC.  That's just what happened with z10 – a bit of unique mainframe ISA design coupled with some fundamental cache/memory building blocks. 

In 2010, the same design leverage is expected.  POWER7 is promised for staggered 2010 rollout across Power Systems UNIX servers.  And, IBM's recent earnings call indicated that the next mainframe launch will also be in 2010.  Downsizing the number of design teams is truly heartbreaking for now unemployed system designers.  But, combining design teams across products and leveraging fundamental building blocks has allowed IBM to reduce its costs and still deliver timely product refreshes.

What are the Performance & Energy Tradeoffs of 15K vs.10K RPM Disk Drives?

Have you ever wondered what the combined performance and energy tradeoffs are when considering whether to go down the high-capacity lower-performance 10K rpm drive route, compared with using faster 15K RPM, but lower capacity drives? And thus, which ones really fit your needs? The Storage Performance Council (SPC) is starting to provide these sorts of answers, with its next release of a performance benchmark for storage with energy extensions.

Green IT concerns, such as power consumption and carbon emissions, continue to grow in importance with end users. While cost-cutting is a tangible incentive for Green IT programs, such initiatives are no longer just about saving money. Higher level issues, such as understanding and minimizing an organization’s environmental impact, are driving organizations to quantify their carbon footprint and take action to reduce it. As such, demand is growing for suitable tools that help IT decision makers weigh their energy reduction goals against the performance requirements of their organization.

This leads to the inevitable question of to how to measure and quantify the situation. So, pressure is growing for suitable tools to help IT decision makers navigate this new world.

SPC-1/E Benchmark

The Storage Performance Council (SPC) is one organization working to meet that demand. Today, the SPC announced the SPC-1/E Benchmark, its second storage performance benchmark to include energy extensions. Historically, industry-standard benchmarks have focused primarily on the performance and price/performance of the tested configurations. The SPC-1/E Benchmark adds real-world energy consumption to the mix, offering more detailed insight into the various tradeoffs (performance, energy usage, and cost) of deploying one product – or one technology – over another.

Two New Results

Two SPC-1/E results have been released to coincide with the introduction of the new benchmark. Xiotech, a storage solution provider, has posted two separate results for the same Emprise 5000 product. What is very interesting is that two tests for the same product have been released at the same time, with only the drive type changing between the tested configurations. The first test used 146 GB 15K RPM FC disk drives while the second used 600 GB 10K RPM FC Seagate drives. Everything else about the configurations remained the same.

So, even if you are not in the market for a Xiotech product, the outcomes of these tests are interesting because they begin to give some visibility on more general issues, such as what is the real impact on performance and energy consumption between using 15K RPM and 10K RPM drives.

So which is best?

As expected, the faster 146 GB 15K RPM drive configuration provides higher levels of performance. It also consumes more power. But if absolute performance is not a primary factor then the lower energy profile of the slower disks may be of interest. Interestingly, the relationship between performance and power consumption is not linear. Compared to the 146 GB 15K RPM configuration, the 600 GB 10K RPM configuration consumes 19.4% less energy yet delivers only 12.9% less performance. (Chart 1) These results suggest that a modest tradeoff in performance can result in potentially significant energy savings. These potential savings may be further enhanced once the extra storage capacity of the 600 GB drive is factored in. More food for thought!

Free Research Report

IDEAS has produced a free Research Report titled ‘Energy and Performance Come Together with Storage Benchmarking’, which can be downloaded from the Sample Research section of www.ideasinternational.com  This report analyzes the outcomes of the two test results in more detail to help users understand which disk technology may be better suited to their needs.

In Review

With the addition of energy extensions, IDEAS believes that the SPC Benchmarks are taking on a new level of relevance. In light of the increasing focus on Green IT, the IT community is crying out for independent, quantifiable, and verifiable insights into the tradeoffs involved in choosing one technology over another. The addition of the energy extensions to the performance perspectives of the SPC Benchmarks enables more in-depth analysis of the technologies being tested. The benchmarks enable users to probe multiple dimensions – not only in isolation, but in conjunction with each other – in order to choose the technology that best suits their needs.

Gary Burgess
Senior Vice President, Research & Operations
 Follow Gary on Twitter

 

Storage Performance Council

The SPC is a non-profit corporation founded to define, standardize and promote storage benchmarks and to disseminate objective, verifiable storage performance data to the computer industry and its customers. The organization’s strategic objectives are to empower storage vendors to build better products as well as to stimulate the IT community to more rapidly trust and deploy multi-vendor storage technology.

www.storageperformance.org