Thread: Announcing the first release of SnowFlock

Hi everyone,
we're releasing Snowflock to the general public. We're making a binary and source relase, under the GNU General Public License (GPL). The release is available at Snowflock - Impromptu Virtual Clusters.

Briefly, Snowflock lets you clone Xen VMs into dozens of identical replicas running in different hosts. Snowflock can do this in less than a second and with very low runtime overhead. With Snowflock you can, for example, perform parallel computations on the fly by scaling "instantaneously" your computing footprint in a shared cluster. Snowflock is a research prototype, hence the 0.1 major-minor. A minimum degree of experience with Xen and Linux is necessary to use the system. The contact address for snowflock is snowflock-users@cs.toronto.edu.

Of particular interest to HPCcommunity.org may be our bindings to Platform EGO. In Snowflock we use our modified Xen hypervisor and control stack to swiftly spawn clone VMs. However, before cloning we need a resource allocator to tell us how many clones and on which machines we can create. The SnowFlock control plane can use Platform EGO as a resource allocator, posting requests for slots on which VMs will be cloned. We have created a simplification library to post slot requests to EGO called libegosf, with C and python bindings, which you may find useful for your own purposes.

More technically:
Snowflock is our prototype implementation of the / Impromptu Cluster (IC)/ abstraction. In an IC, an application encapsulated inside a virtual machine (VM) is swiftly forked into multiple copies that execute on different physical hosts, and then disappear when the computation ends. ICs simplify the development of parallel applications and reduces management burden by enabling the instantiation of new stateful computing elements: workers that need no setup time because they have a memory of the application state achieved up to the point of forking. This approach combines the benefits of cluster-based parallelism with those of running inside a VM.

Snowflock provides swift parallel VM cloning that makes it possible for Internet applications to deliver near-interactive performance for resource-intensive highly-parallelizable tasks. Snowflock makes use of four key techniques: /VM descriptors/ (condensed VM images that allow for sub-second suspension of a running VM and resumption of a of replicas); a /memory-on-demand/ subsystem that lazily populates the VM's memory image during runtime; a set of / avoidance heuristics/ that minimize the amount of VM memory state to be fetched on demand; and a /multicast distribution/ system for commodity Ethernet networking hardware that makes the overhead of instantiating multiple VMs similar to that of instantiating a single one.

Use, enjoy, give us some feedback, and contribute to mankind
The Snowflock team at University of Toronto

This sounds very interesting. Are there any plans to utilize migration for better load balancing and resource management as well?

(I am involved with a project which aims to provide a run-time infrastructure to control the migration of Xen guests running parallel jobs for this purpose).

We are thinking about migration for scheduling, and such things, but no concrete plans on this currently.