Hi John,
Is all your semi-colon fixing on the trunk? I've found another bug in this area (patch below), which is showing up with task splitting under SGE.
e.g. this job is meant to be running and merging 5 BLAST XML files, stderr:
nothing to merge for /mnt/galaxy/galaxy-central/database/files/000/dataset_498.dat (expected 5 files)
/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4: /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh: line 13: syntax error near unexpected token `;' /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh: line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4; blastn -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_495.dat" -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001 -out "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_498.dat" -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit $return_code"'
Note the repeated semi-colon, which causes the child jobs to fail (patch below).
There is however a second bug, the merge fails yet the history entry is still green (success). The merge method is raising a ValueError yet it is being ignored.
Regards,
Peter
$ hg branch default
$ hg tip changeset: 12028:8e001dc9675c tag: tip user: John Chilton jmchilton@gmail.com date: Wed Oct 16 23:15:21 2013 -0500 summary: Dependencies: Do not attempt to install package dependencies in tool shed if the will be ignored.
$ hg diff lib/galaxy/jobs/command_factory.py diff -r 8e001dc9675c lib/galaxy/jobs/command_factory.py --- a/lib/galaxy/jobs/command_factory.py Wed Oct 16 23:15:21 2013 -0500 +++ b/lib/galaxy/jobs/command_factory.py Thu Oct 17 10:50:18 2013 +0100 @@ -58,7 +58,7 @@ if not captured_return_code: commands += capture_return_code_command captured_return_code = True - commands += "; cd %s; " % abspath( getcwd() ) + commands += "; cd %s " % abspath( getcwd() ) commands += job_wrapper.setup_external_metadata( exec_dir=abspath( getcwd() ), tmp_dir=job_wrapper.working_directory,