Hi John,
Thanks for your reply. Question 1: I used the relative path (e.g. ../../../applications/executable) which still does not work. I have managed to put the script in the same folder which works though against the initial design. Question 2: I pass the $output to the perl script within which a file is generated with the filename contained in $output. The Galaxy server can recognize the output. However when I generate another file with the name "primary_$job.id_additional_visible_txt" in the same folder as the $output. In theory, it should also be recognized by Galaxy as being in the job_working_directory. But it is not. Could you reckon the reason? Question 3: Your explanation is very clear. Thank you again for this. The initial purpose of asking this question is related to question 2: I was wondering which "correct" folder I should use to put the additional optional outputs. Now as you suggested, I should not worry about it at all. Question 4: I have managed to convert a RAW file (900M) to mzML (2.3G). But if there is a limit on the file size, the spectral file maybe need to be split into multiple mgf files.
Best regards! Jun
-----Original Message----- From: John Chilton [mailto:jmchilton@gmail.com] Sent: Monday, March 3, 2014 2:27 PM To: Jun Fan Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] LWR questions
Hey Jun,
Thanks for your continued interest in the LWR. Responses inline.
On Sun, Mar 2, 2014 at 7:00 PM, Jun Fan j.fan@qmul.ac.uk wrote:
Hi all,
LWR sounds very attractive and promising. However how to programon galaxy side to utilize LWR server is not thoroughly documented.
Critique noted, I have tried to make it seamless but when you move a job from Linux to Windows there are bound to be severe limitations and I should document these more clearly and provide a guide for writing Windows compatible tools (https://bitbucket.org/jmchilton/lwr/issue/9/provide-documentation-for-writin...).
I have several questions to ask:
I separate the tool wrappers (.xml) and the applications (.pl, .pyetc) in two different folders. In the traditional way it works fine. When I use the legacy way of using LWR (adding galaxy:tool_runners section in the universal_wsgi.ini file), the error message says that the application cannot be found. Does the executable have to be in the same folder with the wrapper?
Do you hard code absolute paths to your scripts in the tool XML files then? The LWR does not support that. If you place a single script in the same folder as the XML (a simple light-weight wrapper as is fairly common) - that is supported - otherwise you will need to place all the required dependencies on the LWR's path on the remote Windows server. This could get awkward if used with interpreter tag for instance - if you send me examples of your tools I can provide some guidance.
I guess in general my advice for Windows tools would be if you can use a simple one-file wrapper script placed next the tool you are fine - just make sure the underlying application is loaded from the path instead of via absolute paths - and then make sure all dependencies for the wrapper script are installed on the remote server and the underlying application is on the LWR's path.
(If your remote server is a *nix server there is new support for using tool_dependency_dir style Galaxy dependencies that resolve on the remote server - but this will not work with Windows (enhancements to provide this would be welcomed though) ).
In the threadhttp://dev.list.galaxyproject.org/Managing-Data-Locality-td4662438.htm l#a4662570 John only mentioned $GALAXY_ROOT/databases/files, does this mean that the dynamic multiple output is not supported as it uses $GALAXY_ROOT/databases/tmp folder?
https://bitbucket.org/galaxy/galaxy-central/src/eafb4208db665dfeed7d8f3e6f0d...
My understanding is that writing those files to the databases/tmp is somewhat deprecated - ideally these should be written to the working directory of the job. The newest version of the LWR *should* attempt to transfer various kinds of working directory files back to the Galaxy server after the job is complete - these files *should* include such variable output files. This hasn't been tested on Windows though - if there are problems please let me know and I can try to address them -
https://bitbucket.org/jmchilton/lwr/src/51f097d3d517064159bd959c61cee71f7894...
What types of files are supposed to be stored under tools directory(tool_files), working directory (working), inputs (inputs), configs directory (configs), all of which are subfolders of ${staging_directory}/${job_identifier}?
In theory, you shouldn't need to be worried about these implementations details as an application programmer, it should feel just like Galaxy (again with limitations). But for what it is worth - the LWR client will transfer tool wrappers it infers are necessary to run the job to `tool_files`, the contents of any `configfile` templates defined for the job to `configs` (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#A.3Cconfigfile.3...), and any inputs it infers are nessecary to `inputs`.
`working` is will be the working directory for the job - so any files your script or application writes to relative directories at runtime will be relative to this directory and finally all output paths (say you have $output defined in your Cheetah) will be rewritten so they target the outputs directory.
Is there file size limit on the LWR server?
It is quite possible there is. Before I had left MSI we had transferred multi-gigabyte files to and from the LWR - but I don't know if we had ever actually transferred a file over 4Gb (an obvious potential limit). I get the sense that the underlying paste server is kind of taxed by these. Like Galaxy itself, it could presumably be hosted in a more robust web server. I have done some looking at chaussette for instance which sees to have support for Windows.
Hope this helps somewhat, -John
Best regards!
Jun
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/