1. Introduction

One of our members posted this question in the forum: "What happens if a SOAM process becomes unavailable?"

We will briefly describe the behavior and recovery steps when Symphony DE components (both system daemons and client applications) or the host machine become unavailable.

2. Background

First of all, the automatic failure recovery features available in the full version of Symphony are not applicable to Symphony DE. Symphony DE is oriented towards a development environment, not a production environment that the full version of Symphony is intended to serve. Maintaining cluster reliability and availability is not the primary objective of Symphony DE, nor is it the concern of developers when building and tuning their applications. In the event of hardware or daemon failure, developers can always restart Symphony DE or reboot the host without serious impact on the application development process.

Furthermore, Symphony DE middleware components (RS, SD, SSM, SIM, start_agent) are built and tested to be fault-resilient and reliable daemons. It is uncommon for theses processes to crash or hang.

3. System Daemons

In the event that a system daemon does become unavailable, for instance manually killed by a user, please refer to the table below for the specifics on each Symphony DE process.

Process	Result	Recovery
start_agent	Symphony DE shutdown on detection	Manually restart Symphony DE
rs	Symphony DE shutdown, unrecoverable workload lost	Manually restart Symphony DE
sd	Symphony DE shutdown, unrecoverable workload lost	Manually restart Symphony DE
ssm	SIM shutdown, SD restarts new SSM, unrecoverable workload lost	Automatic
sim	SI shutdown, SSM restarts new SIM, task running on previous SI reran	Automatic

4. Client Application

Symphony DE also provides features to handle Application failure recovery in case of abnormal termination of a client or service process due to coding or environment error. Refer to the Feature Reference document for details.

Application	Result	Recovery	Reference
Service Instance (SI)	SI restarts, rerun workload (Configurable)	Automatic	Service error handling feature
Client	Client disconnects from session	Relaunch client and reconnect	Client recovery

5. Host

The host which Symphony DE runs on can also become unavailable due to hardware or power failure outside of Symphony DE. Symphony DE runs as a System Service on Windows, thus it will launch automatically as soon as Windows is restarted.

Host	Result	Recovery
Linux/Solaris Host	All Symphony DE processes exit, unrecoverable workload lost	Restart host, restart Symphony DE
Windows Host	All Symphony DE processes exit, unrecoverable workload lost	Restart host, Symphony DE automatically starts

6. Summary

The diagram below offers a general overview of what we have discussed above: