Thread: data staging

When I submit a request without the data staging portion it runs fine.
I can successfully ftp to and from BARRETT01.
When I submit with data staging I get

% ./besclient ... create stage.jsdl stage.epr
SOAP 1.1 fault: "http://schemas.microsoft.com/net/2005/12/windowscommunicationfoundation/dispatcher":InternalServiceFault [no subcode]
"The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs."
Detail: [no detail]

I've asked SysAdminGuy to do as the above message suggested but it seems to be not so simple, he's working on it. Is there any other way to get an error message that is useful?

It is not real clear from the documentation whether the Source/Target URI should be the client or the server, but I've tried assorted combinations of either way. Below, barrett01 is the server and just to be sure, files exist in both places.

cat stage.jsdl
<?xml version="1.0" encoding="UTF-8"?>
<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl">
<JobDescription>
<JobIdentification>
<JobName>stage</JobName>
<JobProject>FileStaging</JobProject>
</JobIdentification>
<Application>
<HPCProfileApplication xmlns="http://schemas.ggf.org/jsdl/2006/07/jsdl-hpcpa">
<Executable>echo</Executable>
<Argument>Hello</Argument>
<Output>stage.out</Output>
<Error>stage.err</Error>
<WorkingDirectory>\\BARRETT\scratch\blast</WorkingDirectory>
</HPCProfileApplication>
</Application>
<Resources>
<TotalCPUCount>
<Exact>1</Exact>
</TotalCPUCount>
</Resources>
<DataStaging>
<FileName>stage.in</FileName>
<CreationFlag>overwrite</CreationFlag>
<Source>
<URI>http://barrett01.msi.umn.edu:22/stage.in</URI>
</Source>
<Credential xmlns="http://schemas.ogf.org/hpcp/2007/11/ac">
<UsernameToken xmlns="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<Username>gibson</Username>
<Password>secret</Password>
</UsernameToken>
</Credential>
</DataStaging>
<DataStaging>
<FileName>stage.out</FileName>
<CreationFlag>overwrite</CreationFlag>
<Target>
<URI>http://barrett01.msi.umn.edu:22/stage.out</URI>
</Target>
<Credential xmlns="http://schemas.ogf.org/hpcp/2007/11/ac">
<UsernameToken xmlns="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<Username>gibson</Username>
<Password>secret</Password>
</UsernameToken>
</Credential>
</DataStaging>
</JobDescription>
</JobDefinition>

From the soap fault it looks like you're talking to a server running on Windows HPC Server? I'm not sure what other information you can get from that type of service other than what is suggested in the fault.

It doesn't really matter which (client or server) is running the file service. The Source and Target URIs can be any running file server. One thing that I do notice is that you're using 'http://' as the protocol rather than 'ftp://', but you mention that you have ftp set up (and 22 is normally the ssh/scp port?). Perhaps the Windows BES service is trying to use http to talk to an ftp server?

Due to security firewall things, SysAdminGuy is reconfiguring our Windows HPC Server with ftps but that is not done yet. So I can't actually try ftps yet. ftp and sftp were having issues. I'm assuming ftps will be on port 21.

The documentation says:

Definition
A Source element contains the location of the file or directory on the remote system. This file or directory MUST be staged in from the location specified by the (optional) URI before the job has started. If this element is not present then the file does not have to be staged in.
Multiplicity
The multiplicity of this element is zero or one.

I want to stage all files from desktop/inputdir into hpc/inputdir
and then return all files from hpc/outputdir back to desktop/outputdir.
If desktop/inputdir and desktop/outputdir are both desktop/currentworkingdir that works out fine.

I want to stage in x1.in x2.in x3.in and return x1.out x2.out ...
The script on the hpc side will handle file names and directories.
Will my URI be

<Source><URI>ftp://hpc.umn.edu:21/data/inputdir</URI></Source>
and
<Target><URI>ftp://hpc.umn.edu:21/data/outputdir</URI></Target>

Correct. Each of the DataStaging elements corresponding to each input file (i.e. you'll have three DataStaging elements ... one for x1.in, x2.in and x3.in) will have a Source element as you describe above. And each of the DataStaging elements for output will have the Target as you mention above. Like so:

<DataStaging>
<FileName>x1.in</FileName>
<CreationFlag>...</CreationFlag>
<Source><URI>ftp://hpc.umn.edu:21/data/inputdir</URI></Source>
...
</DataStaging>
<DataStaging>
<FileName>x2.in</FileName>
<CreationFlag>...</CreationFlag>
<Source><URI>ftp://hpc.umn.edu:21/data/inputdir</URI></Source>
...
</DataStaging><DataStaging>
<FileName>x3.in</FileName>
<CreationFlag>...</CreationFlag>
<Source><URI>ftp://hpc.umn.edu:21/data/inputdir</URI></Source>
...
</DataStaging>
<DataStaging>
<FileName>x1.out</FileName>
<CreationFlag>...</CreationFlag>
<Target><URI>ftp://hpc.umn.edu:21/data/outputdir</URI></Target>
...
</DataStaging>
<DataStaging>
<FileName>x2.out</FileName>
<CreationFlag>...</CreationFlag>
<Target><URI>ftp://hpc.umn.edu:21/data/outputdir</URI></Target>
...
</DataStaging>