bug with output files_path property dataset ID when Galaxy has shared histories
Hello, I've developed a few tools for Galaxy and I think I ran into a bug that even exists in the latest version. As you know a Galaxy server maintains the same external dataset ID (i.e. viewable in the web URLs) to the filesystem internal dataset ID (i.e. names in database/files/000/) if no user in the Galaxy server has yet shared any histories (and their datasets). But once sharing starts the external dataset IDs start differing from the internal dataset ID, and they are always higher and Galaxy maintains this transparently. But this behavior seems to be broken with the output files_path property. If you have a tool which uses the output files_path property like this one I have: <command interpreter="perl">search.pl $query_list $output1 $output1.files_path</command> On my test server I've shared one history with a single dataset. So my external-internal offset is 1. The above tool then produces the following command: perl /home/hermida/galaxy/galaxy_dist/tools/omics_data_miner/search.pl /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_28.dat /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_31.dat /home/hermida/galaxy/galaxy_dist/database/tmp/dataset_32_files Galaxy is not generating the correct files_path, it should end in dataset_31_files not 32. This bug causes the tool to completely break when you are trying to view your output in the browser and I tried to circumvent the bug with symlinking magic but it can't fix the problem because symlinks start stepping on existing directories once you run the tool more than once. Thanks for any help on how to fix the problem, Leandro
Leandro, This is fixed in the latest commit. Thanks, --nate Hermida, Leandro wrote:
Hello,
I’ve developed a few tools for Galaxy and I think I ran into a bug that even exists in the latest version. As you know a Galaxy server maintains the same external dataset ID (i.e. viewable in the web URLs) to the filesystem internal dataset ID (i.e. names in database/files/000/) if no user in the Galaxy server has yet shared any histories (and their datasets). But once sharing starts the external dataset IDs start differing from the internal dataset ID, and they are always higher and Galaxy maintains this transparently. But this behavior seems to be broken with the output files_path property.
If you have a tool which uses the output files_path property like this one I have:
<command interpreter="perl">search.pl $query_list $output1 $output1.files_path</command>
On my test server I’ve shared one history with a single dataset. So my external-internal offset is 1. The above tool then produces the following command:
perl /home/hermida/galaxy/galaxy_dist/tools/omics_data_miner/search.pl /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_28.dat /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_31.dat /home/hermida/galaxy/galaxy_dist/database/tmp/dataset_32_files
Galaxy is not generating the correct files_path, it should end in dataset_31_files not 32. This bug causes the tool to completely break when you are trying to view your output in the browser and I tried to circumvent the bug with symlinking magic but it can’t fix the problem because symlinks start stepping on existing directories once you run the tool more than once.
Thanks for any help on how to fix the problem,
Leandro
------------------------------------------------------------------------
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
Hello,
I've developed a few tools for Galaxy and I think I ran into a bug
Hi Nate, Thanks for fixing so fast... has this been committed to http://www.bx.psu.edu/hg/galaxy? Best, Leandro -----Original Message----- From: Nate Coraor [mailto:nate@bx.psu.edu] Sent: Thursday, May 21, 2009 18:46 To: Hermida, Leandro Cc: galaxy-dev@bx.psu.edu Subject: Re: [galaxy-dev] bug with output files_path property dataset ID when Galaxy has shared histories Leandro, This is fixed in the latest commit. Thanks, --nate Hermida, Leandro wrote: that
even exists in the latest version. As you know a Galaxy server maintains the same external dataset ID (i.e. viewable in the web URLs)
to the filesystem internal dataset ID (i.e. names in database/files/000/) if no user in the Galaxy server has yet shared any histories (and their datasets). But once sharing starts the external dataset IDs start differing from the internal dataset ID, and they are
always higher and Galaxy maintains this transparently. But this behavior seems to be broken with the output files_path property.
If you have a tool which uses the output files_path property like this
one I have:
<command interpreter="perl">search.pl $query_list $output1 $output1.files_path</command>
On my test server I've shared one history with a single dataset. So my external-internal offset is 1. The above tool then produces the following command:
perl /home/hermida/galaxy/galaxy_dist/tools/omics_data_miner/search.pl
/home/hermida/galaxy/galaxy_dist/database/files/000/dataset_28.dat /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_31.dat /home/hermida/galaxy/galaxy_dist/database/tmp/dataset_32_files
Galaxy is not generating the correct files_path, it should end in dataset_31_files not 32. This bug causes the tool to completely break
when you are trying to view your output in the browser and I tried to circumvent the bug with symlinking magic but it can't fix the problem because symlinks start stepping on existing directories once you run the tool more than once.
Thanks for any help on how to fix the problem,
Leandro
------------------------------------------------------------------------
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
Hello,
I've developed a few tools for Galaxy and I think I ran into a bug
Sorry... I got it, forgot to pull first :) leandro -----Original Message----- From: galaxy-dev-bounces@bx.psu.edu [mailto:galaxy-dev-bounces@bx.psu.edu] On Behalf Of Hermida, Leandro Sent: Thursday, May 21, 2009 21:56 To: Nate Coraor Cc: galaxy-dev@bx.psu.edu Subject: Re: [galaxy-dev] bug with output files_path property dataset IDwhen Galaxy has shared histories Hi Nate, Thanks for fixing so fast... has this been committed to http://www.bx.psu.edu/hg/galaxy? Best, Leandro -----Original Message----- From: Nate Coraor [mailto:nate@bx.psu.edu] Sent: Thursday, May 21, 2009 18:46 To: Hermida, Leandro Cc: galaxy-dev@bx.psu.edu Subject: Re: [galaxy-dev] bug with output files_path property dataset ID when Galaxy has shared histories Leandro, This is fixed in the latest commit. Thanks, --nate Hermida, Leandro wrote: that
even exists in the latest version. As you know a Galaxy server maintains the same external dataset ID (i.e. viewable in the web URLs)
to the filesystem internal dataset ID (i.e. names in database/files/000/) if no user in the Galaxy server has yet shared any histories (and their datasets). But once sharing starts the external dataset IDs start differing from the internal dataset ID, and they are
always higher and Galaxy maintains this transparently. But this behavior seems to be broken with the output files_path property.
If you have a tool which uses the output files_path property like this
one I have:
<command interpreter="perl">search.pl $query_list $output1 $output1.files_path</command>
On my test server I've shared one history with a single dataset. So my external-internal offset is 1. The above tool then produces the following command:
perl /home/hermida/galaxy/galaxy_dist/tools/omics_data_miner/search.pl
/home/hermida/galaxy/galaxy_dist/database/files/000/dataset_28.dat /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_31.dat /home/hermida/galaxy/galaxy_dist/database/tmp/dataset_32_files
Galaxy is not generating the correct files_path, it should end in dataset_31_files not 32. This bug causes the tool to completely break
when you are trying to view your output in the browser and I tried to circumvent the bug with symlinking magic but it can't fix the problem because symlinks start stepping on existing directories once you run the tool more than once.
Thanks for any help on how to fix the problem,
Leandro
------------------------------------------------------------------------
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
Hello,
I've developed a few tools for Galaxy and I think I ran into a bug
Hi again, I checked on my search tool and the problem with files_path looks fixed. Thanks. BTW what was the reasoning for moving files_path from tmp --> job_working_directory? Cheers, Leandro -----Original Message----- From: Hermida, Leandro Sent: Thursday, May 21, 2009 22:18 To: Nate Coraor Cc: galaxy-dev@bx.psu.edu Subject: RE: [galaxy-dev] bug with output files_path property dataset IDwhen Galaxy has shared histories Sorry... I got it, forgot to pull first :) leandro -----Original Message----- From: galaxy-dev-bounces@bx.psu.edu [mailto:galaxy-dev-bounces@bx.psu.edu] On Behalf Of Hermida, Leandro Sent: Thursday, May 21, 2009 21:56 To: Nate Coraor Cc: galaxy-dev@bx.psu.edu Subject: Re: [galaxy-dev] bug with output files_path property dataset IDwhen Galaxy has shared histories Hi Nate, Thanks for fixing so fast... has this been committed to http://www.bx.psu.edu/hg/galaxy? Best, Leandro -----Original Message----- From: Nate Coraor [mailto:nate@bx.psu.edu] Sent: Thursday, May 21, 2009 18:46 To: Hermida, Leandro Cc: galaxy-dev@bx.psu.edu Subject: Re: [galaxy-dev] bug with output files_path property dataset ID when Galaxy has shared histories Leandro, This is fixed in the latest commit. Thanks, --nate Hermida, Leandro wrote: that
even exists in the latest version. As you know a Galaxy server maintains the same external dataset ID (i.e. viewable in the web URLs)
to the filesystem internal dataset ID (i.e. names in database/files/000/) if no user in the Galaxy server has yet shared any histories (and their datasets). But once sharing starts the external dataset IDs start differing from the internal dataset ID, and they are
always higher and Galaxy maintains this transparently. But this behavior seems to be broken with the output files_path property.
If you have a tool which uses the output files_path property like this
one I have:
<command interpreter="perl">search.pl $query_list $output1 $output1.files_path</command>
On my test server I've shared one history with a single dataset. So my external-internal offset is 1. The above tool then produces the following command:
perl /home/hermida/galaxy/galaxy_dist/tools/omics_data_miner/search.pl
/home/hermida/galaxy/galaxy_dist/database/files/000/dataset_28.dat /home/hermida/galaxy/galaxy_dist/database/files/000/dataset_31.dat /home/hermida/galaxy/galaxy_dist/database/tmp/dataset_32_files
Galaxy is not generating the correct files_path, it should end in dataset_31_files not 32. This bug causes the tool to completely break
when you are trying to view your output in the browser and I tried to circumvent the bug with symlinking magic but it can't fix the problem because symlinks start stepping on existing directories once you run the tool more than once.
Thanks for any help on how to fix the problem,
Leandro
------------------------------------------------------------------------
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
Hermida, Leandro wrote:
I checked on my search tool and the problem with files_path looks fixed. Thanks.
Great!
BTW what was the reasoning for moving files_path from tmp --> job_working_directory?
files_path was originally the only use for tmp, but it's since expanded to be used for other things. It seemed more appropriate to store the temporary files_path in the working directory with everything else job related than in the top level of the tmp dir. --nate
participants (2)
-
Hermida, Leandro -
Nate Coraor