Thread: Tasks throttling

Hi,

I am looking for best design that would allow to throttle tasks submitted to the grid.

One of the usage scenarios I need to support involve submitting single job with huge amount of tasks with data size of 1M or more. What I would like to avoid is overwhelming Symphony DE infrastructure with dozens/hundreds gigabytes of data waiting in queues.

Is there any native support (cluster parameter, for example), that would allow me implement this easily?

Thx,

Gennadiy

Hi Gennadiy,

Symphony has internal logic to deal with low resources. The module is called the boundary manager and can be configured based on cpu and memory availability. In the CRITICAL state, all tasks will be paused. For the most part, application developers don't have to configure the parameters, the defaults should work OK. If you run into problems, you can change the levels.

Alternatively, you can throttle in the application client by just not submitting tasks as quickly using a sleep() in you send loop.

- Ajith

1. how can i configure it?
2. What do you mean by tasks being paused. I do not want task being executed paused. What I am trying to prevent is SSM queue to grow to unmanageable size.

Ideally I would like call to sendTaskInput to block if queue is full, so that client software is unaware about any limitations on server side and they are applied automatically for any clients.

The problem with doing this on client side is that I need to encode information about cluster setup into client software, which does not seems like right design looking on a big picture. This may be especially bad if master host load and resources are dynamic and can change without knowledge of the client.

Gennadiy

Tasks in the input queues to the Session Manager (SSM) are paused, to prevent the service client from overloading the SSM. Running tasks are not paused.

The boundary manager configuration is in the application profile's SSM section.

<boundaryManagerConfig>
<boundaries>
<boundary elementName="AvailableMemory">
<event name="BEV_PROACTIVE" value="50"/>
<event name="BEV_SEVERE" value="40"/>
<event name="BEV_CRITICAL" value="0"/>
<event name="BEV_HALT" value="0"/>
</boundary>
<boundary elementName="AvailableVirtualAddressSpace">
<event name="BEV_PROACTIVE" value="50"/>
<event name="BEV_SEVERE" value="40"/>
<event name="BEV_CRITICAL" value="25"/>
<event name="BEV_HALT" value="15"/>
</boundary>
</boundaries>
</boundaryManagerConfig>

Each BEV level will cause the SSM to manage memory a bit more efficiently. At no time will executing tasks be effected. These level are only for the SSM.

Sorry, but can you be more specific?

What will happened once the boundary condition occur?

Will SSM stop accepting new tasks? Where are they going to be queued then? In client output queue? Or call to sendTaskInput will block?

Thanks,

Gennadiy

Have a look here at the documentation:

SOAM section

The boundary manager info is about 1/2 way down.

When the task queues are paused, the tasks are buffered in the client output queue. The client thread won't be affected. The SSM will stop accepting tasks while the critical state is set.

- Ajith

That's what I was afraid. I actually want the client thread to be affected. I do not want the client output buffer to overflow as well. Instead I'd prefer task submission call to block.

Is there any way to make client work in this setup?

Or maybe I can query somehow the size of the output queue?

Gennadiy

Hi Gennadiy ,

Sorry I was mistaken. The client sending thread will block if the output queue is paused by default. You can override this behaviour by setting the SendOverlapped flag when creating the session.

See Session class Reference

- Ajith

Thanks a lot, Ajith!

Apparently everything should work as desired by default ;o)

Rare piece of good news in our world