RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Nate,
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
Yes, but only with tools from the toolshed and there I poked people to remove the trailing semicolon from the wrapper. From my observation the tool does not produce double semicolons. Only one at the end, but the Job scheduler adds one, resulting in a double one with invalid syntax.
I can't see where bowtie is producing a double semicolon. I really think SGE is adding one. At least I restructured the semicolons (attached) and it worked for us.
Thanks Nate, Bjoern
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
If the tool produces a trailing semi-colon (and without Nicola's patch the bowtie 2 tool definitely could), then I bet build_command_line() is going to be what is adding the second semi-colon. It has not check that there isn't a semi-colon at the end of the command before adding one and extending the command. It might not be the only culprit, but it is likely one.
-John
On Fri, Oct 11, 2013 at 9:02 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Hi Nate,
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
Yes, but only with tools from the toolshed and there I poked people to remove the trailing semicolon from the wrapper. From my observation the tool does not produce double semicolons. Only one at the end, but the Job scheduler adds one, resulting in a double one with invalid syntax.
I can't see where bowtie is producing a double semicolon. I really think SGE is adding one. At least I restructured the semicolons (attached) and it worked for us.
Thanks Nate, Bjoern
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Ah now I got it ... there are two build_command_line functions. lib/galaxy/tools/__init__.py and lib/galaxy/jobs/runners/__init__.py.
And one is adding a trailing ';'
So either stripping the trailing semicolon, right after creating the command line from cheetah (patch was attached) or adding more logic to only add a ';' when no one is there.
Which one is preferred, should I create a pull request?
Thanks! Bjoern
If the tool produces a trailing semi-colon (and without Nicola's patch the bowtie 2 tool definitely could), then I bet build_command_line() is going to be what is adding the second semi-colon. It has not check that there isn't a semi-colon at the end of the command before adding one and extending the command. It might not be the only culprit, but it is likely one.
-John
On Fri, Oct 11, 2013 at 9:02 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Hi Nate,
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
Yes, but only with tools from the toolshed and there I poked people to remove the trailing semicolon from the wrapper. From my observation the tool does not produce double semicolons. Only one at the end, but the Job scheduler adds one, resulting in a double one with invalid syntax.
I can't see where bowtie is producing a double semicolon. I really think SGE is adding one. At least I restructured the semicolons (attached) and it worked for us.
Thanks Nate, Bjoern
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
My plan is to refactor build_command_line out of lib/galaxy/jobs/runners/__init__.py, into its own module and add unit tests for this functionality. I'll issue a pull request and let people test it. I am also testing Nicola's PR, I will approve it shortly.
Nate yelled at me for committing code yesterday :) so I will let him figure out when to merge it. But I will approve the pull requests that I am willing to say are tested and that I will take responsibility for if something breaks in them.
-John
On Fri, Oct 11, 2013 at 9:13 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Ah now I got it ... there are two build_command_line functions. lib/galaxy/tools/__init__.py and lib/galaxy/jobs/runners/__init__.py.
And one is adding a trailing ';'
So either stripping the trailing semicolon, right after creating the command line from cheetah (patch was attached) or adding more logic to only add a ';' when no one is there.
Which one is preferred, should I create a pull request?
Thanks! Bjoern
If the tool produces a trailing semi-colon (and without Nicola's patch the bowtie 2 tool definitely could), then I bet build_command_line() is going to be what is adding the second semi-colon. It has not check that there isn't a semi-colon at the end of the command before adding one and extending the command. It might not be the only culprit, but it is likely one.
-John
On Fri, Oct 11, 2013 at 9:02 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Hi Nate,
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
Yes, but only with tools from the toolshed and there I poked people to remove the trailing semicolon from the wrapper. From my observation the tool does not produce double semicolons. Only one at the end, but the Job scheduler adds one, resulting in a double one with invalid syntax.
I can't see where bowtie is producing a double semicolon. I really think SGE is adding one. At least I restructured the semicolons (attached) and it worked for us.
Thanks Nate, Bjoern
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Am Freitag, den 11.10.2013, 09:18 -0500 schrieb John Chilton:
My plan is to refactor build_command_line out of lib/galaxy/jobs/runners/__init__.py, into its own module and add unit tests for this functionality. I'll issue a pull request and let people test it. I am also testing Nicola's PR, I will approve it shortly.
Nate yelled at me for committing code yesterday :) so I will let him figure out when to merge it. But I will approve the pull requests that I am willing to say are tested and that I will take responsibility for if something breaks in them.
Ok, sounds good!
Thanks! Bjoern
-John
On Fri, Oct 11, 2013 at 9:13 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Ah now I got it ... there are two build_command_line functions. lib/galaxy/tools/__init__.py and lib/galaxy/jobs/runners/__init__.py.
And one is adding a trailing ';'
So either stripping the trailing semicolon, right after creating the command line from cheetah (patch was attached) or adding more logic to only add a ';' when no one is there.
Which one is preferred, should I create a pull request?
Thanks! Bjoern
If the tool produces a trailing semi-colon (and without Nicola's patch the bowtie 2 tool definitely could), then I bet build_command_line() is going to be what is adding the second semi-colon. It has not check that there isn't a semi-colon at the end of the command before adding one and extending the command. It might not be the only culprit, but it is likely one.
-John
On Fri, Oct 11, 2013 at 9:02 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Hi Nate,
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
Yes, but only with tools from the toolshed and there I poked people to remove the trailing semicolon from the wrapper. From my observation the tool does not produce double semicolons. Only one at the end, but the Job scheduler adds one, resulting in a double one with invalid syntax.
I can't see where bowtie is producing a double semicolon. I really think SGE is adding one. At least I restructured the semicolons (attached) and it worked for us.
Thanks Nate, Bjoern
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote: > > Dear all, > > some SGE instances will add automatically an semicolon add the end of > each command, resulting in a disrupted job, because ';;' are not > allowed. > > The latest changes to the Bowtie2 wrapper resulting in such a case and a > crash in our instance. An easy solution would be to fix the wrapper and > omitting the trailing ';' but maybe its better to fix it once for all in > the corresponding tools code, patched attached. > > I'm not familiar with other Job schedulers so I'm seeking for > comments ... it there any disadvantage of removing trailing ';' from the > command line? > > Ciao, > Bjoern > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hey All,
Patch available with test case, let me know what I broke :).
https://bitbucket.org/jmchilton/galaxy-central-multi-input-tool-fixes-2/comm...
-John
On Fri, Oct 11, 2013 at 10:37 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Am Freitag, den 11.10.2013, 09:18 -0500 schrieb John Chilton:
My plan is to refactor build_command_line out of lib/galaxy/jobs/runners/__init__.py, into its own module and add unit tests for this functionality. I'll issue a pull request and let people test it. I am also testing Nicola's PR, I will approve it shortly.
Nate yelled at me for committing code yesterday :) so I will let him figure out when to merge it. But I will approve the pull requests that I am willing to say are tested and that I will take responsibility for if something breaks in them.
Ok, sounds good!
Thanks! Bjoern
-John
On Fri, Oct 11, 2013 at 9:13 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Ah now I got it ... there are two build_command_line functions. lib/galaxy/tools/__init__.py and lib/galaxy/jobs/runners/__init__.py.
And one is adding a trailing ';'
So either stripping the trailing semicolon, right after creating the command line from cheetah (patch was attached) or adding more logic to only add a ';' when no one is there.
Which one is preferred, should I create a pull request?
Thanks! Bjoern
If the tool produces a trailing semi-colon (and without Nicola's patch the bowtie 2 tool definitely could), then I bet build_command_line() is going to be what is adding the second semi-colon. It has not check that there isn't a semi-colon at the end of the command before adding one and extending the command. It might not be the only culprit, but it is likely one.
-John
On Fri, Oct 11, 2013 at 9:02 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Hi Nate,
Hi Björn,
Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools?
Yes, but only with tools from the toolshed and there I poked people to remove the trailing semicolon from the wrapper. From my observation the tool does not produce double semicolons. Only one at the end, but the Job scheduler adds one, resulting in a double one with invalid syntax.
I can't see where bowtie is producing a double semicolon. I really think SGE is adding one. At least I restructured the semicolons (attached) and it worked for us.
Thanks Nate, Bjoern
--nate
On Oct 11, 2013, at 8:57 AM, John Chilton wrote:
> This sounds like a good way to solve the problem, I guess I would modify > > > On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning > bjoern.gruening@pharmazie.uni-freiburg.de wrote: >> >> Dear all, >> >> some SGE instances will add automatically an semicolon add the end of >> each command, resulting in a disrupted job, because ';;' are not >> allowed. >> >> The latest changes to the Bowtie2 wrapper resulting in such a case and a >> crash in our instance. An easy solution would be to fix the wrapper and >> omitting the trailing ';' but maybe its better to fix it once for all in >> the corresponding tools code, patched attached. >> >> I'm not familiar with other Job schedulers so I'm seeking for >> comments ... it there any disadvantage of removing trailing ';' from the >> command line? >> >> Ciao, >> Bjoern >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ >
Hi John,
Is all your semi-colon fixing on the trunk? I've found another bug in this area (patch below), which is showing up with task splitting under SGE.
e.g. this job is meant to be running and merging 5 BLAST XML files, stderr:
nothing to merge for /mnt/galaxy/galaxy-central/database/files/000/dataset_498.dat (expected 5 files)
/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"'
Note the repeated semi-colon, which causes the child jobs to fail (patch below).
There is however a second bug, the merge fails yet the history entry is still green (success). The merge method is raising a ValueError yet it is being ignored.
Regards,
Peter
$ hg branch default
$ hg tip changeset: 12028:8e001dc9675c tag: tip user: John Chilton jmchilton@gmail.com date: Wed Oct 16 23:15:21 2013 -0500 summary: Dependencies: Do not attempt to install package dependencies in tool shed if the will be ignored.
$ hg diff lib/galaxy/jobs/command_factory.py diff -r 8e001dc9675c lib/galaxy/jobs/command_factory.py --- a/lib/galaxy/jobs/command_factory.py Wed Oct 16 23:15:21 2013 -0500 +++ b/lib/galaxy/jobs/command_factory.py Thu Oct 17 10:50:18 2013 +0100 @@ -58,7 +58,7 @@ if not captured_return_code: commands += capture_return_code_command captured_return_code = True - commands += "; cd %s; " % abspath( getcwd() ) + commands += "; cd %s " % abspath( getcwd() ) commands += job_wrapper.setup_external_metadata( exec_dir=abspath( getcwd() ), tmp_dir=job_wrapper.working_directory,
On Thu, Oct 17, 2013 at 10:53 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
Hi John,
Is all your semi-colon fixing on the trunk? I've found another bug in this area (patch below), which is showing up with task splitting under SGE.
...
Note the repeated semi-colon, which causes the child jobs to fail (patch below).
There is however a second bug, the merge fails yet the history entry is still green (success). The merge method is raising a ValueError yet it is being ignored.
That bug detecting failiures be a symptom of something more general, I've just had a split-BLAST job return this as stderr, yet the history entry is green: Job output not returned from cluster
Peter
I broke the TaskWrapper last week with my exit code handling "fix", the double semi-colon thing you are seeing there. Your fix would break non-task split jobs so that is probably the problem(?) Hopefully? Want to revert 8e001dc9675c and pull in the changeset I just pushed out.
Otherwise, I will test out task splitting later today.
I am very sorry.
-John
On Thu, Oct 17, 2013 at 4:53 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
Hi John,
Is all your semi-colon fixing on the trunk? I've found another bug in this area (patch below), which is showing up with task splitting under SGE.
e.g. this job is meant to be running and merging 5 BLAST XML files, stderr:
nothing to merge for /mnt/galaxy/galaxy-central/database/files/000/dataset_498.dat (expected 5 files)
/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"'
Note the repeated semi-colon, which causes the child jobs to fail (patch below).
There is however a second bug, the merge fails yet the history entry is still green (success). The merge method is raising a ValueError yet it is being ignored.
Regards,
Peter
$ hg branch default
$ hg tip changeset: 12028:8e001dc9675c tag: tip user: John Chilton jmchilton@gmail.com date: Wed Oct 16 23:15:21 2013 -0500 summary: Dependencies: Do not attempt to install package dependencies in tool shed if the will be ignored.
$ hg diff lib/galaxy/jobs/command_factory.py diff -r 8e001dc9675c lib/galaxy/jobs/command_factory.py --- a/lib/galaxy/jobs/command_factory.py Wed Oct 16 23:15:21 2013 -0500 +++ b/lib/galaxy/jobs/command_factory.py Thu Oct 17 10:50:18 2013 +0100 @@ -58,7 +58,7 @@ if not captured_return_code: commands += capture_return_code_command captured_return_code = True
commands += "; cd %s; " % abspath( getcwd() )
commands += "; cd %s " % abspath( getcwd() ) commands += job_wrapper.setup_external_metadata( exec_dir=abspath( getcwd() ), tmp_dir=job_wrapper.working_directory,
On Thu, Oct 17, 2013 at 11:36 AM, John Chilton chilton@msi.umn.edu wrote:
I broke the TaskWrapper last week with my exit code handling "fix", the double semi-colon thing you are seeing there. Your fix would break non-task split jobs so that is probably the problem(?) Hopefully? Want to revert 8e001dc9675c and pull in the changeset I just pushed out.
Otherwise, I will test out task splitting later today.
I am very sorry.
-John
Thanks John,
Those of us running with galaxy-dist expect minor breakage from time to time - I do this to avoid more pain if the problems were not spotted by the community and reached galaxy-central and thus our production Galaxy instance.
(And with the job splitting not being enabled by default, I am aware that I am in a relatively small group of Galaxy admins using it.)
I don't think my fix hurts non-task split jobs, but I will now try your fix on the default branch:
https://bitbucket.org/galaxy/galaxy-central/commits/329ea7a83af4f389a7c95ee4...
This appears to also address a metadata issue, which if I am lucky may be the fix for this issue?:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/017031.html
Thanks,
Peter
On Thu, Oct 17, 2013 at 5:49 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
On Thu, Oct 17, 2013 at 11:36 AM, John Chilton chilton@msi.umn.edu wrote:
I broke the TaskWrapper last week with my exit code handling "fix", the double semi-colon thing you are seeing there. Your fix would break non-task split jobs so that is probably the problem(?) Hopefully? Want to revert 8e001dc9675c and pull in the changeset I just pushed out.
Otherwise, I will test out task splitting later today.
I am very sorry.
-John
Thanks John,
Those of us running with galaxy-dist expect minor breakage from time to time - I do this to avoid more pain if the problems were not spotted by the community and reached galaxy-central and thus our production Galaxy instance.
Well they have given me commit access so expect a lot more "minor breakage" :).
(And with the job splitting not being enabled by default, I am aware that I am in a relatively small group of Galaxy admins using it.)
I don't think my fix hurts non-task split jobs, but I will now try your fix on the default branch:
I think it will in at least some cases if metadata is getting set externally, I don't see how it is preventing some commands from running together, I could totally be wrong though.
At any rate, I tested the version with my fix on your blast wrappers, with task splitting on and off, submitting to a DRM and using the local job runner, and they all seemed to work. Let me know if your galaxy instance is unconvinced.
https://bitbucket.org/galaxy/galaxy-central/commits/329ea7a83af4f389a7c95ee4...
This appears to also address a metadata issue, which if I am lucky may be the fix for this issue?:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/017031.html
I doubt your luck is so good. That problems looks like some sort of disk caching issue to me (galaxy process and worker node having inconsistent views of the same file system), I doubt this will fix it. Though hopefully I am wrong on both counts :).
Thanks for reporting the problem and the fix!
-John
Thanks,
Peter
On Thu, Oct 17, 2013 at 5:04 PM, John Chilton chilton@msi.umn.edu wrote:
On Thu, Oct 17, 2013 at 5:49 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
On Thu, Oct 17, 2013 at 11:36 AM, John Chilton chilton@msi.umn.edu wrote:
(And with the job splitting not being enabled by default, I am aware that I am in a relatively small group of Galaxy admins using it.)
I don't think my fix hurts non-task split jobs, but I will now try your fix on the default branch:
I think it will in at least some cases if metadata is getting set externally, I don't see how it is preventing some commands from running together, I could totally be wrong though.
At any rate, I tested the version with my fix on your blast wrappers, with task splitting on and off, submitting to a DRM and using the local job runner, and they all seemed to work. Let me know if your galaxy instance is unconvinced.
Seem fine so far :)
This appears to also address a metadata issue, which if I am lucky may be the fix for this issue?:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/017031.html
I doubt your luck is so good. That problems looks like some sort of disk caching issue to me (galaxy process and worker node having inconsistent views of the same file system), I doubt this will fix it. Though hopefully I am wrong on both counts :).
Disk caching makes sense as a root cause - I've not had this happen consistently or reproducibly yet so it may well return.
Thanks for reporting the problem and the fix!
-John
No problem, thank you for addressing it so promptly.
Peter
Opps, sent that last e-mail before I meant to, gmail is acting very buggy for me this morning. I just meant to say I would modify build_command_line in lib/galaxy/jobs/runners/__init__.py instead. It seems like there is a lot of logic in there for this kind of thing already. I will create a Trello card and you.
Also just say Nate's response, he is probably right so ignore this e-mail as well :).
-John
On Fri, Oct 11, 2013 at 7:57 AM, John Chilton chil0060@umn.edu wrote:
This sounds like a good way to solve the problem, I guess I would modify
On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
Ciao, Bjoern
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
The Bowtie2 bug may be fixed if my pull request 234 will be merged:
https://bitbucket.org/galaxy/galaxy-central/pull-request/234/bug-fixes-for-3...
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
That would be sensible too.
Ciao, Nicola
Hi Nicola,
interesting you fixed bowtie in the same way than we did. But somehow I would like to strip a trailing ';' by default in Galaxy, when it does not break anything else.
Thanks for sharing! Bjoern
Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto:
Dear all,
some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached.
The Bowtie2 bug may be fixed if my pull request 234 will be merged:
https://bitbucket.org/galaxy/galaxy-central/pull-request/234/bug-fixes-for-3...
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line?
That would be sensible too.
Ciao, Nicola
I already fixed it one week ago:
https://bitbucket.org/nsoranzo/galaxy-central-tools/commits/74b5fd2318d33b76...
but then Ross committed a similar fix to galaxy-central, I merged it in my pull request and then Ross reverted the fix! This is the reason for rebasing pull request #222 to #234.
Ciao, Nicola
Il 2013-10-11 16:05 Björn Grüning ha scritto:
Hi Nicola,
interesting you fixed bowtie in the same way than we did. But somehow I would like to strip a trailing ';' by default in Galaxy, when it does not break anything else.
Thanks for sharing! Bjoern
Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto:
Dear all,
some SGE instances will add automatically an semicolon add the end
of
each command, resulting in a disrupted job, because ';;' are not allowed.
The latest changes to the Bowtie2 wrapper resulting in such a case
and a
crash in our instance. An easy solution would be to fix the
wrapper and
omitting the trailing ';' but maybe its better to fix it once for
all in
the corresponding tools code, patched attached.
The Bowtie2 bug may be fixed if my pull request 234 will be merged:
https://bitbucket.org/galaxy/galaxy-central/pull-request/234/bug-fixes-for-3...
I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';'
from the
command line?
That would be sensible too.
Ciao, Nicola
galaxy-dev@lists.galaxyproject.org
-
Björn Grüning -
John Chilton -
John Chilton -
Nate Coraor -
Nicola Soranzo -
Peter Cock