Thread: Compel Symphony to load-balance

We have noticed that when the task time is 0.001 sec or less Symphony stops to load-balance. That is, even if there are 4 compute nodes available, Symphony uses only one of them. This is a good policy indeed, because the communication cost will be higher than the compute cost in this scenario. But to find out the communication overhead, we would like to compel Symphony to load balance, even in such lower task loads. Is there any way that we could make that possible?

Thanks,
Jaison

Hi Jaison,

If there are multiple slots available and more tasks than slots, Symphony will use all the slots. The scheduling policy will take all the available tasks and schedule them simultaneously. Possibly, the client is not sending enough tasks at one time. Try sending say 1 million tasks at one time and then see how they are scheduled. Also make sure you pre-start the SIMs.

- Ajith

Originally Posted by jmulerik

We have noticed that when the task time is 0.001 sec or less Symphony stops to load-balance. That is, even if there are 4 compute nodes available, Symphony uses only one of them. This is a good policy indeed, because the communication cost will be higher than the compute cost in this scenario. But to find out the communication overhead, we would like to compel Symphony to load balance, even in such lower task loads. Is there any way that we could make that possible?

Thanks,
Jaison

Hi Ajith,

Regarding scheduling issue, you suggested that it may be because of "client is not sending enough tasks at one time" and advised to use "say, 1 million tasks at one time"! That trick worked. I used 1000 tasks for Matirx-Vector Multiplication application of Matrix Order 2000. But as you know it was highly inefficient. It is because communication cost exceeds computational advantage.

However, I found that Symphony (now I use 4.1) does load-balance with number of tasks = 8 or 16 or 32 or 40 or 80 (for Matrix of Order 2000 with 8 SIs in a Matirx-Vector Multiplication application) when the task time (time to complete a task at onInvoke) is above 0.005 sec (I create this load by putting extra iterations (10 and 100) to get the same result at onInvoke!).

But on a real implementation of Matrix-Vector Multiplication application (with out creating extra load) when number of tasks = 8 or 16 or 32 or 40 or 80, task time (at onInvoke) is of the order of 0.001 sec or less. But in this scenario symphony stops load-balancing ( ie, when tasks = 8 or 16 or 32 or 40 or 80 but not 1000!).

Another interesting thing, when number of tasks = 1000, task time (time to complete a task at onInvoke) is 0.0001 sec or less, but still Symphony load-balance.

If we want to make use of Symphony for scientific apps, we need to get this thing right.

What would be causing this? What code in the source may have to be modified?

One more tip.. even though Symphony load balance using 1000 tasks (as explained above), it is very much skewed. Here I attach a pie-chart to show how it was load balancing in our clusters.

Hi Jaison,

I still believe the problem may be as Ajith said. Your tasks may not be reaching Symphony quickly enough to saturate the grid.

You may want to try the Message Aggregation feature to see if you can saturate the grid a bit better. If you do not use Message Aggregation, your client app will block on the sendTaskInput() call until the workload manager (SSM) acknowledges the receipt of your task. With Message Aggregation, your client app can send tasks more quickly and the acknowledgements will come back asynchronously. There is a Message Aggregation sample included with Symphony DE that demonstrates how to use the feature.

Hope this helps!