tool.xml "&>" console redirection within command-tag
Dear all, that has probably been asked 10e6 times, however, I don't find any comments on how to properly escape special characters in a toolxy.xml. What I want to achieve is the following: <command interpreter="bash">$myScript &> $logFile</command> where myScript is dynamically created (as part of the configfiles-section) and the logFile is where stderr and stdout should go to. Now the problem is obviously the "&"-character, because it causes the following error-msg on reloading the toolxy.xml: ExpatError: not well-formed (invalid token): line 5, column 50 where the 5/50 refers to the "&"-character. By the way: redirecting either stdout or stderr by specifying just ">" (instead of "&>") works fine. I tried a lot of different notations in the meanwhile, but neither something like "\&>" nor "\&>" works. Any suggestions? Many thanks in advance and Cheers! Uwe
On Mon, Oct 24, 2011 at 9:01 AM, Appelt, Uwe <uwe.appelt@nct-heidelberg.de> wrote:
Dear all,
that has probably been asked 10e6 times, however, I don't find any comments on how to properly escape special characters in a toolxy.xml. What I want to achieve is the following:
<command interpreter="bash">$myScript &> $logFile</command>
where myScript is dynamically created (as part of the configfiles-section) and the logFile is where stderr and stdout should go to. Now the problem is obviously the "&"-character, because it causes the following error-msg on reloading the toolxy.xml:
ExpatError: not well-formed (invalid token): line 5, column 50
where the 5/50 refers to the "&"-character. By the way: redirecting either stdout or stderr by specifying just ">" (instead of "&>") works fine.
I tried a lot of different notations in the meanwhile, but neither something like "\&>" nor "\&>" works. Any suggestions?
Many thanks in advance and Cheers! Uwe
Don't you need to escape both the & and the > characters here? i.e. &> should work. Peter
Hi Peter, thanks a lot for your quick reply. However "&>" doesn't seem to work as well. Also, I do have to admit that two different "doesn't work" exist. Notations that are not accepted during parsing of toolxy.xml: &> \&> \\&> Notations that are accepted, but don't produce any content written to "$logFile": &> &> &> Any other suggestions? Cheers, Uwe -----Ursprüngliche Nachricht----- Von: Peter Cock [mailto:p.j.a.cock@googlemail.com] Gesendet: Montag, 24. Oktober 2011 10:06 An: Appelt, Uwe Cc: galaxy-user@lists.bx.psu.edu Betreff: Re: [galaxy-user] tool.xml "&>" console redirection within command-tag On Mon, Oct 24, 2011 at 9:01 AM, Appelt, Uwe <uwe.appelt@nct-heidelberg.de> wrote:
Dear all,
that has probably been asked 10e6 times, however, I don't find any comments on how to properly escape special characters in a toolxy.xml. What I want to achieve is the following:
<command interpreter="bash">$myScript &> $logFile</command>
where myScript is dynamically created (as part of the configfiles-section) and the logFile is where stderr and stdout should go to. Now the problem is obviously the "&"-character, because it causes the following error-msg on reloading the toolxy.xml:
ExpatError: not well-formed (invalid token): line 5, column 50
where the 5/50 refers to the "&"-character. By the way: redirecting either stdout or stderr by specifying just ">" (instead of "&>") works fine.
I tried a lot of different notations in the meanwhile, but neither something like "\&>" nor "\&>" works. Any suggestions?
Many thanks in advance and Cheers! Uwe
Don't you need to escape both the & and the > characters here? i.e. &> should work. Peter
On Mon, Oct 24, 2011 at 9:42 AM, Appelt, Uwe <uwe.appelt@nct-heidelberg.de> wrote:
Hi Peter,
thanks a lot for your quick reply. However "&>" doesn't seem to work as well. Also, I do have to admit that two different "doesn't work" exist.
Notations that are not accepted during parsing of toolxy.xml: &> \&> \\&>
Notations that are accepted, but don't produce any content written to "$logFile": &> &> &>
Any other suggestions?
Cheers, Uwe
Curious. Well, at least you seem to have valid XML now (double check this with an XML validator). Does Galaxy capture anything in the "info" box (stdout)? There are some variants like 2>&1 for redirecting stderr to stdout, try combining that with > logfile. My *guess* however is the interpreter="bash" may be to blame, try this after ensuring the script is executable and on the path or fully specified: <command>$myScript &> $logFile</command> Peter
Hi again, yes, you're right. The problem source is something else. As soon as I specify one of the following <command interpreter="bash">$hisapWrapperScript &> $logFile</command> <command interpreter="sh">$hisapWrapperScript &> $logFile</command> <command>bash $hisapWrapperScript &> $logFile</command> <command>sh $hisapWrapperScript &> $logFile</command> I get a according debug-line in the galaxy.log, reporting something like: galaxy.jobs.runners.local DEBUG 2011-10-24 11:26:55,360 executing: bash /home/extuser/www/galaxy/galaxy_working/database/job_working_directory/2222/tmpZmFMfg &> /home/extuser/www/galaxy/galaxy_working/database/files/005/dataset_5410.dat So the "&>" definitely makes it through XML, Cheetah, etc. -parsers and finally to the cmd-line. However, neither BASH nor SH produce a log-file as intended. For some strange reason (Peter, you're right again) all log-lines appear in the green box. And even more, the green box turns of course red as soon as something is written to stderr. Aaaaaah, but the different redirection syntax works. So to conclude: <command interpreter="bash">$hisapWrapperScript > $logFile 2>&1</command> works like a charme - problem solved! Thanks Peter!!! Cheers, Uwe -----Ursprüngliche Nachricht----- Von: Peter Cock [mailto:p.j.a.cock@googlemail.com] Gesendet: Montag, 24. Oktober 2011 11:07 An: Appelt, Uwe Cc: galaxy-user@lists.bx.psu.edu Betreff: Re: [galaxy-user] tool.xml "&>" console redirection within command-tag On Mon, Oct 24, 2011 at 9:42 AM, Appelt, Uwe <uwe.appelt@nct-heidelberg.de> wrote:
Hi Peter,
thanks a lot for your quick reply. However "&>" doesn't seem to work as well. Also, I do have to admit that two different "doesn't work" exist.
Notations that are not accepted during parsing of toolxy.xml: &> \&> \\&>
Notations that are accepted, but don't produce any content written to "$logFile": &> &> &>
Any other suggestions?
Cheers, Uwe
Curious. Well, at least you seem to have valid XML now (double check this with an XML validator). Does Galaxy capture anything in the "info" box (stdout)? There are some variants like 2>&1 for redirecting stderr to stdout, try combining that with > logfile. My *guess* however is the interpreter="bash" may be to blame, try this after ensuring the script is executable and on the path or fully specified: <command>$myScript &> $logFile</command> Peter
Hi Uwe, On Mon, Oct 24, 2011 at 10:47 AM, Appelt, Uwe <uwe.appelt@nct-heidelberg.de> wrote:
Hi again,
yes, you're right. The problem source is something else. As soon as I specify one of the following
<command interpreter="bash">$hisapWrapperScript &> $logFile</command> <command interpreter="sh">$hisapWrapperScript &> $logFile</command> <command>bash $hisapWrapperScript &> $logFile</command> <command>sh $hisapWrapperScript &> $logFile</command>
I get a according debug-line in the galaxy.log, reporting something like:
galaxy.jobs.runners.local DEBUG 2011-10-24 11:26:55,360 executing: bash /home/extuser/www/galaxy/galaxy_working/database/job_working_directory/2222/tmpZmFMfg &> /home/extuser/www/galaxy/galaxy_working/database/files/005/dataset_5410.dat
So the "&>" definitely makes it through XML, Cheetah, etc. -parsers and finally to the cmd-line. However, neither BASH nor SH produce a log-file as intended.
You didn't try the precise combination I meant, <command>$hisapWrapperScript &> $logFile</command> where $hisapWrapperScript is marked as executable and has a Unix hashbang line saying which shell to use, e.g. #!/usr/bin/bash Apologies if I was too vague.
For some strange reason (Peter, you're right again) all log-lines appear in the green box.
To be expected, Galaxy captures any stdout (assuming nothing else captured it first) and puts it in the info field.
And even more, the green box turns of course red as soon as something is written to stderr.
Yes, it is a long standing Galaxy bug that any stderr out is treated as an error condition, rather than looking at the return code: https://bitbucket.org/galaxy/galaxy-central/issue/325/
Aaaaaah, but the different redirection syntax works. So to conclude:
<command interpreter="bash">$hisapWrapperScript > $logFile 2>&1</command>
works like a charme - problem solved! Thanks Peter!!!
Cheers, Uwe
Glad you got there in the end :) Peter By the way - this whole discussion would have been better suited to the galaxy-dev list (CC'd) since it is about developing tools for Galaxy rather than Galaxy end users.
Well, no, I didn't try to specify neither an "interpreter=" nor the actual shell-binary as part of the command-tag itself, because I'm assembling the shell-script dynamically within a configfile-tag. Thus, I do not see any chance to apply an "executable"-flag to that script that is infact filed to disk by the galaxy-job-scheduler somewhen in the future. So it's not that you were too vague, I just that didn't know how to embed your idea into xml/shell-code. But as you already mentioned, I made it with the slightly more verbose notation and that's great. Thanks again, Peter! Cheers, Uwe -----Ursprüngliche Nachricht----- Von: Peter Cock [mailto:p.j.a.cock@googlemail.com] Gesendet: Montag, 24. Oktober 2011 12:06 An: Appelt, Uwe Cc: galaxy-user@lists.bx.psu.edu; galaxy-dev@lists.bx.psu.edu Betreff: Re: [galaxy-user] tool.xml "&>" console redirection within command-tag Hi Uwe, On Mon, Oct 24, 2011 at 10:47 AM, Appelt, Uwe <uwe.appelt@nct-heidelberg.de> wrote:
Hi again,
yes, you're right. The problem source is something else. As soon as I specify one of the following
<command interpreter="bash">$hisapWrapperScript &> $logFile</command> <command interpreter="sh">$hisapWrapperScript &> $logFile</command> <command>bash $hisapWrapperScript &> $logFile</command> <command>sh $hisapWrapperScript &> $logFile</command>
I get a according debug-line in the galaxy.log, reporting something like:
galaxy.jobs.runners.local DEBUG 2011-10-24 11:26:55,360 executing: bash /home/extuser/www/galaxy/galaxy_working/database/job_working_directory/2222/tmpZmFMfg &> /home/extuser/www/galaxy/galaxy_working/database/files/005/dataset_5410.dat
So the "&>" definitely makes it through XML, Cheetah, etc. -parsers and finally to the cmd-line. However, neither BASH nor SH produce a log-file as intended.
You didn't try the precise combination I meant, <command>$hisapWrapperScript &> $logFile</command> where $hisapWrapperScript is marked as executable and has a Unix hashbang line saying which shell to use, e.g. #!/usr/bin/bash Apologies if I was too vague.
For some strange reason (Peter, you're right again) all log-lines appear in the green box.
To be expected, Galaxy captures any stdout (assuming nothing else captured it first) and puts it in the info field.
And even more, the green box turns of course red as soon as something is written to stderr.
Yes, it is a long standing Galaxy bug that any stderr out is treated as an error condition, rather than looking at the return code: https://bitbucket.org/galaxy/galaxy-central/issue/325/
Aaaaaah, but the different redirection syntax works. So to conclude:
<command interpreter="bash">$hisapWrapperScript > $logFile 2>&1</command>
works like a charme - problem solved! Thanks Peter!!!
Cheers, Uwe
Glad you got there in the end :) Peter By the way - this whole discussion would have been better suited to the galaxy-dev list (CC'd) since it is about developing tools for Galaxy rather than Galaxy end users.
participants (2)
-
Appelt, Uwe -
Peter Cock