Thread: Known Issues with Symphony DE 4.0

This thread can be used to report and discuss known issues with Symphony DE 4.0.

You cannot do asynchronous retrieval of data with .NET 2.0 if your C# Async client is executing in a non-default Application Domain. This is due to a security restriction now enforced in .NET 2.0 and later.

In our package we have all the main samples for .NET including Async client that works fine, because it uses the default Application Domain.

Monitoring service instances (SIs) can be asked for.
For example,
1. How many SIs are up on which machines for a particular application?
2. What is the SI status: idle, running or crashed?

Currently we display task status per application not service. We don't display SI status as it will never be idle with default scheduling policy unless the SI is prestarted. Also, in case of SI crashing, we log a message in the SSM log file* and throw an event that can be captured. The “crash” state is transient since the SIM will restart it** or blocked the host based on you application profile configuration.

* It is easier to look for error message specifically about SI crash, as there is just one SSM logfile per application whereas there can be multiple SIM logs.
** SIM manages the life time of SIs.

If multiple clients try to open the same session at the same time using openSession(), some of the clients may hang, rather than being rejected. Only one client is allowed to re-connect to a closed session.

Impact
Client programs may get stuck in the openSession() method.

Recovery
The client needs to be stopped and restarted.

Avoidance
Ensure only one client at a time attempts to re-connect to a closed session.

.NET and Java services may not release memory after session is closed. Service Instance could throw an Out-Of-Memory exception to the SSM, if garbage collection is not triggered.

Impact
Occurs only when large data is passed through Symphony.If the exception is thrown in onSessionEnter() or onSessionLeave(), the compute host will be blocked.

Recovery
In Symphony DE, restart the cluster and try again with one of the avoidance techniques.

Avoidance
- Set any unused objects to NULL
- Try to trigger garbage collection in service code (e.g. in .NET with GC.Collect(); or in Java with System.gc()) in each service call (e.g. onSessionEnter(), onInvoke(), onSessionLeave())
- Break up data into smaller pieces
- Read data from a shared file system rather than passing data through Symphony
- Do not create consecutive sessions to be run on the same service instance with common data messages that have more than 250 MB of a single byte array/string.

Introduction

The MPI-2 process model allows for the creation and cooperative termination of processes after an MPI application has started. It provides a mechanism to establish communication between the newly created processes and the existing MPI application. It also provides a mechanism to establish communication between two existing MPI applications, even when
one did not “start” the other.
In the latter case and MPI application can establish communication as a server listening to a specific port for any incoming connection request from other MPI applications. Traditional batch system cannot handle such case as resources are statically bound to a MPI application and assigining additional resource to run new MPI applications attempting to connect to the server is not possible.
Unlike the batch system, Symphony DE has the capability of dynamically binding resources to new workload during the life-time of the application. Therefore, it's reasonable to think of Symphony as dynamic MPI process distributor.
Attached sample package is a Symphony application using Symexec feature to integrate with MPI-2 written in MPICH2 and implements Server/Client communication model. Issues with the integration will be addressed.

1. Overview of the integration
As shown in figure below, Symexec client first submits a pre-command to start up the environment in order to facilitate the group communication among parallel MPI2 processes that will be spawned on the resources in Symphony DE grid. Specifically MPICH2 provides command to start up such environment (MPD-Ring) through a command called, mpdboot.
Once the mpdboot is called, the client will submit server and client MPI-2 application in sequence. In the runtime environment Server and client will exchange messages and exit.



2. Issues with the integration
You can submit as many MPI-2 servers and clients as you want through Symexec client, however, Symphony does not facilitate group communication within server group or client group. Because MPI-2 processes were not submitted through mpiexec that uses MPD-ring to set up communication group. mpiexec can be submitted through Symexec client. But that's nothing more than redirecting the command through Symphony and Symphony will have no control over the spawned group of MPI-2 processes and dynamic scheduling of resources will not be applied for those processes.

3. Variation of integration.



Above diagram illustrate that Symphony client can function as a MPI server accepting connection requestion from distributed MPI clients as SIs in the grid. This is one many reasonable variations. However, group commuication among clients is still not possible, as MPI clients are still not submitted through mpiexec.

4. Misc
The sample package is provided mainly to address issues with Symphony that is not able to facilitate group communication among MPI processes. Please play with the sample. You're further feedbacks will be appreciated

Click here to download the package.