Ever wonder how application binaries get copied to where they are required on a compute host? Some middleware vendors recommend that the system administrator deploy binaries to a shared file system location. Other middleware use a file system synchronization mechanism to move binaries to a compute host`s file system. These systems don't tightly integrate an application with the deployment of its binaries, so it is difficult to determine which version of a binary is actually being used at a given time. This makes trouble-shooting more difficult when the binaries are updated very frequently.

Symphony DE has a tightly-coupled deployment sub-system. It is responsible for moving and installing application service binaries on to compute host file systems. The cluster administrator simply associates a service (binary) package with the application requiring it. If an in-use service package is updated, Symphony DE will first warn the user and then stop the running workload. It will then update the service package and deploy it to the cluster automatically.

This article describes in detail how the service deployment system works and some of the pros and cons of the design.

Deployment System Summary

Symphony DE uses pull or on-demand deployment. Your application's service packages are only deployed to a compute host just prior to the execution of the service. This design is suited to large clusters with multiple applications. In a large cluster, a service package is not required on every host and it is difficult to pre-determine which compute host will be assigned to an application. Compute hosts that were unavailable during deployment or are newly added will automatically download the service package only if it is required.

Symphony DE can be configured to use a shared file system, but this is not recommended due to the increased binary load times and network bandwidth usage and security issues with sharing drives.

There is a short time during which the application will be idle while the service package is being deployed and installed. In a small Symphony DE cluster, the one-time initial latency of on-demand deployment is very low and should not be an issue.



Symphony DE uses a lazy service deletion scheme. A service package is deleted from its deployed location on a compute host only when it is updated. Service packages may also be removed from a compute host if the package is deleted from the RS. The removal checking is only done during the service package installation process, so if a package doesn't need to be installed, the deleted service package will remain on the compute host for ever. If this is a concern, the packages must be manually deleted.

You can read the Service Package Deployment documentation for more information about adding and deleting packages.

Deployment Architecture

The service deployment feature uses a client server model with the soamdeploy command line utility as the client and the Repository Server (RS), the server. Soamdeploy can be used to add a package to the RS and retrieve a package from the RS.



The server processes in the preceding diagram do the following;

• Session Director (SD) - The SD is a long running server that inserts information about the package version into the application profile and then restarts the application
• Repository Server (RS) - The RS is a long running server that persists the latest version of every application's service package on it's local file system. It responds to add, remove and list requests from the soamdeploy command.
• Service Session Manager (SSM) - The SSM is a server that starts/stops SIMs and directs task data to the SIMs.
• Service Instance Manager (SIM) - The SIM is a server that starts/stops service instances and manages the service package cache on the local file system.

The Repository Server (RS) manages the service package repository on it's local or shared file system. It maintains a relationship between the application and service package so that it knows when it is safe to update and delete a package.

The RS caches service packages in memory to optimize the distribution of the package to compute hosts. The RS can typically handle thousands of concurrent service package requests without delays or errors.

The soamdeploy command has two roles as a client to the RS server. It is a user command that allows a cluster administrator to add, remove and view packages stored in the RS. It is also used by the SIM to download service packages on demand to the compute host's local file system.

Service deployment is triggered when a SIM goes to start the service instance. The SIM first checks it's local cache for the service package. If the package isn't installed, it initiates soamdeploy to do a package copy through a direct request of the RS. Soamdeploy uses point-to-point tcp/ip communications and doesn't rely on having a shared file-system enabled.

Impact on Running Workload

If you update a service package on an active application, all running session data will be lost. All sessions will have to be manually restarted. The down time will be only be a few seconds. This is the time required for the cluster administrator to update the service package to the RS.

Symphony DE ensures that there will never be a case where a session can use multiple versions of a service package deployed to different compute hosts.

As Symphony DE is a developer's version, stopping workload is generally the desired behavior. The developer can make a code change to the service package and then restart testing to evaluate the change.

With Symphony DE in a production environment, service package updates must be managed carefully by the cluster administrator. The admin must either;

1. choose an update time in which the cluster is idle or
2. make a copy of the application, make the service package update and switch clients over to the new application

If the admin can not find an idle moment and there are many updates to the cluster, option 2 is preferable.

To implement option 2, the application developer must make the application name in the client configurable and allow the cluster admin to update the name. All client sessions will use the new application and service package the next time they are started.

GUI Support for Deployment

The Symphony DE GUI console has full support for managing a service package. You can add, remove and view service packages. You can also link service packages to applications using the application profile editor.



Deployment Configuration

The RS stores service packages in the $SOAM_HOME/.cache/ directory. This location can be modified by changing RS_STORAGE_DIR environment variable on the RS host.

The RS will log information to the $SOAM_HOME/logs directory. The debug level can be configured by modifying the $SOAM_HOME/conf/rs.log4j.properties file.
Change INFO to DEBUG in the following line
“log4j.rootLogger=INFO, RS” to “log4j.rootLogger=DEBUG, RS”

The SIM caches packages in the $SOAM_HOME/deploy/ directory on each compute host. Deployment errors are logged in the SIM log.

Controlling Deployment

There are three options for deployment;

1. Deployment using soamdeploy and the RS
2. Customized deployment using your own deployment tool
3. Static or pre-staged deployment

Option 1 is the default and is enabled in the application profile by defining the packageName field. The package is deployed to the compute host in a location specified by ${SOAM_DEPLOY_DIR}. Use this variable in the service's start-command, to define the file-system location of the service command.

Option 2 allows the administrator to define a command to download a service package to a pre-determined location. This command is called the preExecCmd in the application profile and is executed unconditionally by the SIM just prior to starting the start-command for the service.

Option 3 is defined by not specifying the packageName nor preExecCmd in the application profile. The SIM assumes that the service start-command pre-exists in that location and does not do any deployment actions.

Summary

Symphony DE has a scalable and easy to manage deployment sub-system. It uses a pull-mode deployment model to efficiently distribute service packages to compute hosts. The service packages are cached on the compute hosts until the service packages are updated or deleted at the Repository Server.

Cluster administrators should be careful not to update service packages during times of heavy workload, as this work will be lost once the package is updated.

Symphony DE provides three options to customize deployment, although most customers will choose the default.

References

1. Service Deployment Documentation

Questions

Do some customers actually use 3rd-party deployment tools?

So far, no customer has used their own deployment tool with Symphony, but this feature is nice to have if there is a problem with our deployment system.

It is possible that a customer may already have tools to deploy files efficiently. In this case, they can easily integrate with our deployment system.