Thread: BES++ (HPC Basic Profile), DRMAA, SAGA, and plain REST

I would like to start discussion on which API should be integrate into Lava and which should be the first. I think we should start by listing the APIs first

• BES++ (HPC Basic Profile) - (AFAIK) BES++ is an implementation for OGSA-BES job scheduling compliant with HPC Basic Profile, a protocol for accessing job scheduling mechanism from Grid computing (standardize by OGF). Implementing Lava support for BES++ should be as easy as creating a C++ adapter to communicate with Lava command line
• DRMAA - API base job scheduling (similar to BES++). There is a reference implementation at SourceForge.net: FedStage DRMAA for LSF - Develop. DRMAA is a set of API (in C/Java/Python) to communicate with job scheduler. It lacks some capabilities for fine grain control over job.
• SAGA (Simple API for Grid Applications- Another API) - Another API-base access to job scheduling. Reference implementation at SAGA :: A Simple API for Grid Applications - SAGA C++ Implementation
• REST - There seems to be no REST (or RESTful) standard interface for job scheduling yet. With the growing and popularity gain from Cloud services (like Twitter, Facebook, Google Data API), accessing services on the internet through REST API has becoming popular.

(Please correct me if anything above is wrong)

For me, I think REST interface will be very interesting (imagine job submission through curl or wget) and making Lava different from other job scheduler. For the others, I think the priority is BES++ -> SAGA -> DRMAA, in this order. What do you think?

As one of the authors of the BES specification, and as the implementor of BES++, I of course think you should do BES++ integration first. It has a nice side effect of being useable in a number of programming languages, as long as there is decent SOAP support (I figure most languages have this). It would also be quite straightforward, as BES++ already supports LSF, so the Lava support could use the existing LSF module as a starting point.

REST is very interesting. Do you have any ideas as to the resources you'd export and how to manipulate them?

Yes I agree that BES++ could be implemented first easily. That might be the good starting point.

AFAIK, it seems there is no standard REST API defined for Job scheduling yet. I think, if we are going to implement this, we should use the outlined from any other API (SAGA or HPC Basic Profile) to define our own protocol and format. I think the key is to keep it as simple as possible.

It's definitely a good idea to be informed by specs like the HPC Basic Profile and SAGA when doing a REST API, but I think it's worth designing the API from the ground up. For one, you really have to consider what types of "things" you'll be manipulating, and how to manage those things with the relatively constrained set of operations that HTTP provides. Another thing to consider is how to export the state model to users. REST allows one to "discover" the state machine from the resources one examines, as opposed to the already known state model you get from HPC Profile and SAGA. Perhaps this allows for state model extension to be easier?

Hmm... It seems that the first thing that I should do is to study more deeply into these API . I will look deeper on HPC Profile and SAGA, then draft the idea here.

For the start, I think we should make REST API as easy and simple as possible, just like Twitter, so I agree that ground up design might be better choice.

One concept you probably want to explore in the context of REST is "hypermedia as the engine of application state" (HATEOAS ... you can google it).

Check this link: InfoQ: HATEOAS as an engine for domain specific protocol-description