Symphony WorkManager: A Novel HPC Multi-thread Programming Model V1.2

[simonlee]

Hello every one,
A new version of Symphony WorkManager has released.
Thanks for your supports.


================================================
Contents
================================================

1 About CommonJ WorkManager
2 Brief review about the previous version
3 What's new in this package
4 Benefits and potential useage
5 Release Notes
6 Future Plans
7 Links and Downloads


================================================
1 First we give you some brief explain of Commonj WorkManager.
================================================

Why JSR237 WorkManager is a Market Need:
* Unmanaged multi-thread execution may either overload or underutilize system resources
* Loosely coupled JMS is complicated yet a unmanaged programming model
* Traditional J2EE application developers are limited to using single-thread execution

To solve these problems IBM and BEA has came up with a joint specification:
JSR237 CommonJ WorkManager



* Clients submit work items & data to WorkManager
* WorkManager schedules work items & data to a “thread” pool on an app server cluster
* Work items are executed under the "logically correct" execution context as if they are done by a thread pool in the same JVM as the client’s. The execution context usually includes user security info, session info, shared global data, and etc. It is propagated by a JSR237 implementation
* Clients can wait for the results asynchronously


================================================
2 Brief review about the previous version
================================================

In our previous version : Symphony-WorkManager: An implementation of (CommonJ) WorkManager under symphony API we have implemented all the kernal functions of CommonJ WorkManager specification using Sympony Java API. It provides J2EE developers a simpler way to build multi-thread applications in HPC environment.



And our tests prove that the implementation is solid.

However, there are some insufficiencies of the previous version:
1 Developers have to pre-deploy all the application binaries onto the symphony compute hosts in advance. It is quit inconvenient in distributed applications.
2 The Symphony WorkManager is running on the same host where the CommonJ client submit workloads. It is hard for administrators to manage the environment.


================================================
3 New Features in this version
================================================

As planed in the previous release, we have acomplished to add these new features into v1.2:

1 Dynamic transfer application class files onto the symphony cluster, the user program will work properlly without predeployment. The entire Symphony WorkManager environment is transparent to the front-end CommonJ consumers.

2 Add sharing input files support. All instances of the same user defined Work will share the same input files. Symphony WorkManager will dynamically transfer all the runtime environment of a User-defined Work on the Symphony Cluster.

3 Use flags to control when to remove class files on compute hosts and whether to use dynamic transfer.

4 Integerate Symphony WorkManager with j2ee enviroment. Administrator could deploy Symphony WorkManager instances onto a Jboss server and provide a JNDI path to the CommonJ consumers.

5 Refined Session management strategy. In order to economize Symphony system resources we designed both interval cleaning and post-execution cleaning to clean ideal Sessions.

6 Multi-Works execution support. The new version SymService will support different User-defined Works execute on the same workaround on a compute host concurrently.

The new architecture of Symphony WorkManager system is shown in the following picture:





================================================
4 Benefits and potential useage
================================================

* Promote market awareness of Symphony functionalities and capabilities via supporting a simple, powerful, and industry standard programming model.
* Provide managed execution and scheduling for concurrent and asynchronous Java/J2EE applications.
* Grid-enable a Java/J2EE application less-intrusively or even without code change, if it follows JSR237 standard.
* Transparent to the front-ent CommonJ consumers and compatible to existing CommonJ applications.
* Provide a much simpler way to build multi-thread applications in a HPC environment.

The following picture shows the overall structure of how to use Symphony WorkManager:




================================================
5 Release Notes
================================================

The author would like to give the following colleagues sincerely thanks. Thanks for the time you've contributed to this project.

Pro. Xiaohui WEI, College of Computer Science and Technology, Jilin University, ChangChun, P.R.C.
Zane Zhenhua HU, Platform Computing, Canada.
Shutao YUAN, Platform Computing, Canada.
Shaocheng XING, College of Computer Science and Technology, Jilin.
Ji LI, College of Computer Science and Technology, Jilin.
Gang Wang, Platform Computing, Xi'an, P.R.C.


================================================
6 Future Plans
================================================

There are some future plans about our project:




A: There is no data sharing mechanism defined in Work Manager specification.
We are planning to extend Work Manager with a shared memory mechanism that enables running Works communicate with each other more conveniently.
B: Symphony WorkManager only enable read-only common data among in-flight Works currently.
We plan to support shared file system among Works with Gfarm data grid.



================================================
7 Links and Downloads
================================================


You can download and try out our packages, which contains in the following URLs completely free:

SymphonyWorkManager Admin Guide.pdf

Readme.txt

SymphonyWorkManager-1.2.tar.gz

SymphonyWorkManager-1.2.rar

The project package contains:
-The administrator examples.
-CommonJ application examples.
-SymWorkManager front-end.
-SymService back-end (Symphony Service program package).
-All the source code files of our project.

Just please let us know your opions after trying it. Your advices are quit important to our team. Thank you.

The Chinese new year "Spring Festival" is coming in a week. Happy new year to all my colleages. And wish you all a more productive year 2009.




Hongliang LI, College of Computer Science and Technology, Jilin University, ChangChun, P.R.C.
simonlee@email.jlu.edu.cn

17th, JAN, 2009

=========================
Copyright 1994-2009 Platform Computing Corporation,
2004-2009 Lab of Distributed Computing and System Architecture, Jilin University
All Rights Reserved.
=========================

Although the information in this document has been carefully reviewed, Lab of Distributed Computing and System Architecture, Jilin University ("DCSA Lab.") does not warrant it to be free of errors or omissions. DCSA Lab. reserves the right to make corrections, updates, revisions or changes to the information in this document.

UNLESS OTHERWISE EXPRESSLY STATED BY PLATFORM, THE PROGRAM DESCRIBED IN THIS DOCUMENT IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANYKIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT WILL PLATFORM COMPUTING BE LIABLE TO ANYONE FOR SPECIAL, COLLATERAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING WITHOUT LIMITATION ANY LOST PROFITS, DATA OR SAVINGS, ARISING OUT OF THE USE OF OR INABILITY TO USE THIS PROGRAM.

[zane_hu]

Hi Simon, you are right that such distributed thread pooling model needs a distributed shared memory mechanism to ease up data sharing. However we also need to consider minimizing the scalability impact of data sharing when writing such parallel apps.

In general, there are 3 types of shared data here.

1. Real-only if a dataset can only be read by every thread.
2. Single-writer if a dataset can only be written by one thread who owns the dataset, and read by every thread.
3. Multiple-writers if a dataset can be written by any threads, and read by every thread.

Platform Symphony promotes the first 2 types of data sharing to write large scale SOA apps, because there is no data locking needed for these 2 types. Type 1 can be supported by Symphony common data. And type 2 can be realized by using a Symphony feature called "common data update" -- any service instance can broadcast a common data update to other instances.

For type 3, you can use a distributed cache system like GemFire, Coherence, and GigaSpaces to name a few. However it is like a critical section in an ordinary multi-thread app on an SMP box. You may have to use data locking to protect the data sharing consistency, which may potentially slow down the parallel execution of your app because some threads have to wait for a lock. If a thread may lock two or more datasets for writing, then a deadlock or other concurrency complications may come in play to your app...