Hello, We are looking at integrating some tools into Galaxy for a domain where many of the existing programs are Windows only. My desire is to write tool files and have the certain jobs run remotely on a Windows host (our Galaxy servers are Linux). It wasn't clear how to do this with the existing job runners since they all seem to require a shared file system and absolute file paths that are the same on the Galaxy server as they are on the remote execution hosts. I developed a new job runner that allows for the remote execution of jobs that also rewrites paths and stages files as part of the job execution. There is a server application that needs to be running on the remote machine (*nix or Windows). This application is likely easier to setup/configure than SGE/PBS/etc... because it is a simple Python paste application, with just a few configuration options. There may be valid use cases for this approach for non-Windows applications as well - if you would like a lighter weight server component on the remote system, if shared file systems are impossible/impractical, etc.... The path rewrites are not going to cover every possible way an absolute path can sneak into the execution, so this is only useful for a subset of tools. But it does handle normal file inputs and outputs, config files, and scripts in the tool definition directory. I welcome input/advice on this approach or the implementation. If there is any way I can improve the code base to the point where it can be included in galaxy-central I would be more than happy to make the desired changes. If not, it is out there for people to use if they have a similar use case. The server application as well as a patch to add the runner/client to a galaxy-central installation can be found at: https://bitbucket.org/jmchilton/lwr/src Thanks for your time, John ------------------------------------------------ John Chilton Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 E-Mail: chilton@msi.umn.edu
Hi John, This is fantastic, thanks for writing this. I've committed the Galaxy-side code in 5050:74048a9ab689, so the patch should no longer be required. I'll try to get something placed on the Cluster page in the wiki, too. Thanks, --nate John Chilton wrote:
Hello,
We are looking at integrating some tools into Galaxy for a domain where many of the existing programs are Windows only. My desire is to write tool files and have the certain jobs run remotely on a Windows host (our Galaxy servers are Linux). It wasn't clear how to do this with the existing job runners since they all seem to require a shared file system and absolute file paths that are the same on the Galaxy server as they are on the remote execution hosts.
I developed a new job runner that allows for the remote execution of jobs that also rewrites paths and stages files as part of the job execution. There is a server application that needs to be running on the remote machine (*nix or Windows). This application is likely easier to setup/configure than SGE/PBS/etc... because it is a simple Python paste application, with just a few configuration options. There may be valid use cases for this approach for non-Windows applications as well - if you would like a lighter weight server component on the remote system, if shared file systems are impossible/impractical, etc....
The path rewrites are not going to cover every possible way an absolute path can sneak into the execution, so this is only useful for a subset of tools. But it does handle normal file inputs and outputs, config files, and scripts in the tool definition directory. I welcome input/advice on this approach or the implementation. If there is any way I can improve the code base to the point where it can be included in galaxy-central I would be more than happy to make the desired changes. If not, it is out there for people to use if they have a similar use case.
The server application as well as a patch to add the runner/client to a galaxy-central installation can be found at:
https://bitbucket.org/jmchilton/lwr/src
Thanks for your time, John
------------------------------------------------ John Chilton Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 E-Mail: chilton@msi.umn.edu _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
participants (2)
-
John Chilton -
Nate Coraor