Thread: SC09: Three Trends Worth Watching

The annual Supercomputing Conference, called SC09, was held recently in Portland Oregon. This was the 22nd consecutive year for the event and it was rather well attended given the recent economic down turn. The final attendance numbers have not been released, but, I was told (unofficially) that conference registration was actually higher than last years levels. Given that the show seemed downright lively at times, I think the HPC industry has weathered the economic storm rather well.

As always, there is more news than one person can cover. An article summarizing all of the show news and events would be far too long (and boring) for my taste. Instead of trying to "see it all," I try and look for trends and themes that are developing within the market. This year, there were many people asking "What is new?" or "Have you seen anything new?" Valid questions, but, I think many of the big companies did not have market-moving stories this year, and people were forced to look elsewhere for the real news.

Of course, there was the Top500, where ORNL's Jaguar Claws its Way to Number One, Leaving Reconfigured Roadrunner Behind. Along with Intel and AMD processor news -- more cores, who would have guessed. And, IBM announced it had produced a Cat-Brain Simulation Breakthrough that has since given rise to some controversy. All in all not a whole lot to get excited about.

As the show progressed, I got a sense for three trends that have been developing. Some of these trends may not be on many folks radar screens just yet, but, in my opinion these ideas and technologies may change the way we do HPC. The trends include, GP-GPU computing, Cluster 3.0 (application driven job scheduling), and big data clusters (HPC data mining).

Hybrid (GP-GPU) Computing

The GP-GPU (General Purpose Graphics Processor) has been a source of excitement in recent years. There has always been an issue around whether it would become a mainstream technology rather than a curiosity that was good at accelerating certain applications. For those that are not that familiar with GP-GPU computing, standard PC video processing is a highly parallel process. All modern video processors have adopted a general purpose parallel hardware design whereby they place hundreds of video processing cores on a single chip. In addition to video processing, these "parallel computers" can be used for general purpose computing and in particular data parallel HPC applications.

In the HPC arena, NVidia has been leading the GP-GPU computing charge. They have provided specialized HPC hardware and software tools with which to create HPC applications. As Figure One indicates, the floating point performance for GP-GPUs has far outstripped the conventional general purpose processor. Thus, for problems that require data parallel floating point accelerations, GP-GPUs are a big win. (Not all problems or computer applications fit into this category.)

Figure One: GFLOPS vs Processor Architecture (Courtesy of NVidia)

There were two main issues that have prevented GP-GPU from broad acceptance in the HPC sector, support for ECC memory and poor double-precision floating point performance. To address these issues, NVidia recently announced the Fermi architecture, which it demonstrated at SC09. Fermi includes support for ECC memory and an 8X increase in peak double precision performance over previous designs. Many believe that the Fermi architecture will move GP-GPU computing into the mainstream of HPC.

Dr. Jack Dongarra of Top500 and ATLAS fame attended SC09 and can be seen here talking about the Magma Project which brings optimized dense linear algebra routines to GP-GPUs and other heterogeneous/hybrid architectures. Such mainstream HPC efforts tend to affirm the GP-GPU approach to HPC, since linear algebra is the basis for many important applications.

The other player in GP-GPU computing is AMD/ATI. You can learn more about the hardware and software options in a recent HPC Community article entitled Programming GP-GPUs: Options and Answers.

Cluster 3.0

Cluster provisioning has always been an important topic. The first HPC clusters were built by hand, installing Linux software on each of the nodes. This process was time consuming, but worked well for small clusters. In my classification scheme, I call this the Cluster 1.0 approach. As the size (number of nodes) of clusters grew, the need to manage nodes became more important. Two methods were developed to address this problem. The first was image management where complete system (server) images were automatically distributed to the nodes. All updates or changes to the nodes required a re-imaging of the nodes. The second method employed a disk-less booting strategy where the nodes received a minimal "in memory" install over the network. The minimal install was designed to support MPI applications rather than to provide a general purpose compute server. I consider this approach Cluster 2.0, where fixed software installs were managed by a central server.

In both the Cluster 1.0 and 2.0 scenarios, the software environment was fixed on the nodes. That is, the worker nodes had to reflect the host node software environment and applications had to fit into a fixed software environment. This approach often reduces the utilization of the cluster, as it is only suited for specific applications. It is a static approach where the software is "nailed" to the hardware.

Cluster 3.0 provides a new dynamic approach to HPC provisioning. Instead of offering a fixed software environment, a more fluid approach is used. Traditionally, the user was required to "fit the application" to the Cluster 2.0 (or 1.0) environment. With Cluster 3.0, the application now tells the scheduler (execution environment) what type of resources it needs (Linux version, Windows, etc.) and the nodes are automatically provisioned for the user at run-time. This approach is known as dynamic provisioning.

The HPC hardware has become virtualized, but not in a traditional "cloud computing" sense. In the case of dynamic provisioning, there is no meta layer that sits between the hardware and client operating system. The nature of HPC applications (long run times) allows the entire application hardware stack (sub-set of the cluster) to be configured for each individual application. In essence, the Operating System (OS) environment has become a scheduled resource option for the end user.

As you would guess the leading providers of job scheduling and management tools have integrated this capability into their products. A good example is the Platform ISF Adaptive Cluster which turns static clusters and grids into dynamic, shared computing environments using heterogeneous physical and virtual HPC resources. The ISF Adaptive Cluster allocates resources dynamically based on Platform LSF and Platform Symphony workload demands. Adaptive Computing (Previously Cluster Resources) also offers the Moab Adaptive Operating Environment. While these products are new, I expect this to be the way HPC will be accomplished in the future. Ideally, the user will be able to submit their application and the scheduler will find and provision the needed resources, including interconnect and I/O requirements as well as processor and OS choice. Of paarticular note is the fact that this technology extends far beyond the HPC market and is generally applicable to the needs! of today's modern data center.

Big Data Clusters

Another significant piece of news at SC09 was the $20 million grant to San Diego Supercomputing Center (SDSC) from the National Science Foundation (NSF) to build and operate a powerful supercomputer dedicated to solving those problems with huge data sets. These types of problems are not the same as the typical number crunching applications. Indeed, a Top500 computer will actually perform poorly on data heavy problems because the number crunching cluster is not optimized for large file input and output (I/O). For example, the new SDSC cluster will allow the manipulation of massive graphs that arise in many data-intensive fields, including bioinformatics, social networks, and neuroscience.

SDSC will use the Appro Xtreme X cluster design as a basis for the system and employ a vast amount of flash memory storage (SSDs) to help speed solutions that are limited by slower spinning disk technology. Also, the cluster includes new "supernodes" that will exploit virtual shared-memory software (ScaleMP) to create large shared-memory systems that can accommodate large problems that would not fit on a standard HPC compute node.

The cluster goes by the nick name of "Gordon" (as in Flash) and is the follow-up to SDSC's previously announced Dash system, the first supercomputer to use flash devices. According to SDSC Interim Director Michael Norman, who is also the project's principal investigator,

"This HPC system will allow researchers to tackle a growing list of critical 'data-intensive' problems. These include the analysis of individual genomes to tailor drugs to specific patients, the development of more accurate models to predict the impact of earthquakes on buildings and other structures, and simulations that offer greater insights into what's happening to the planet's climate."

When fully configured and deployed, Gordon will feature 245 TFLOPS of total compute power (one TFLOPS is one trillion calculations per second), 64 TeraBytes of RAM, 256 TeraBytes of flash memory storage, and four PetaBytes of disk storage. For sheer power, when complete, Gordon should rate among the top 30 or so supercomputers in the world. Though impressive, these statistics only explain part of the machine's special capabilities. Gordon is ideally suited to tackle a variety of problems involving large data sets that are less concerned with raw performance than I/O productivity. Indeed, the performance for Gordon will be measured in I/O operations per second (IOPS) rather than the more traditional FLOPS.

As mentioned, a key feature of Gordon will be 32 "supernodes" based on virtual shared-memory software provided by Scale MP, Inc. Each supernode consists of 32 compute nodes, capable of 240 GFLOPS/node (one GFLOP billion calculations per second) and 64 GBytes of RAM. A supernode also incorporates two I/O nodes, each with four TBytes of flash memory. When tied together by virtual shared memory, each of the system's 32 supernodes has the potential of 7.7 TFLOPS of compute power and ten TBytes of memory (two TBytes of RAM and eight TBytes of flash memory).

Gordon's 32 supernodes will be interconnected via an InfiniBand network, capable of 16 gigabits per second of bi-directional bandwidth -- that's eight times faster than some of the most powerful national supercomputers to come on-line in recent months. The combination of raw power, flash technology, and large-shared memory on a single supernode, coupled with high-bandwidth across the system, is expected to reduce the time and complexity often experienced when researchers tackle data-intensive problems that don't scale well on today's massively parallel supercomputers.

The Coming Year

In the coming year, these three technologies will continue to play an increasing role in HPC. As the market seems to be looking "beyond the cores" to other issues such as hybrid computing, application driven scheduling, and massive problem sizes, one thing will remain constant. The need for "more everything" seems to drive the HPC market each year. The constant need for more floating point cycles, storage space, bandwidth, flexibility, and people seems to push the HPC market into new and interesting areas. Next year, at SC10 in New Orleans, I expect there will once again be more HPC than I could possibly see in three days!