Thread: job chaining

Hello,

I want to run a series of chained jobs, namely the second job will start when the first job is done, the third job will start when the second is done, and so on. I understand I can take advantage job dependencies by using the "bsub -w" option. But since the number of jobs is no small, I prefer using a shell script to do job submission. And my question is how to pass the job id of the previous job to the submission command of the next job in a shell script.

Thank you!
David

You can use jobnames instead of jobids, so in that case u can just have a script that names each jobs as job1, job2 etc ...

You could also parse the output from the bsub command to get the jobID, and use that in the subsequent bsub calls.

Something like:
jobid=`bsub .... | sed "s/.*<\([0-9]*\)>.*/\1/"`

If you're going to use job ids, you have to make sure and deal with a possible race condition where the first job starts changing state before your dependent job has been submitted. You can avoid this by submitting the first job with a hold, and then resume it after the dependent job(s) has been submitted.

Here's a /bin/sh script fragment that I've used in the past:

#!/bin/sh

jobid=`bsub -H -o /dev/null sleep 30 | awk '{print $2}'|sed -e 's/[<>]//g'`
bsub -o /dev/null -w "done($jobid)" sleep 30
bresume $jobid