Hi, I've tried using the Data Manager (Admin > Data > Manage local data (beta)) to install builds for BWA and Samtools on my local Galaxy instance. Previous to using the Data Manager, I used to add the build to tool-data/shared/ucsc/builds.txt, create the .fai indexes (for samtools) from the command line, add them to tool-data/sam_fa_indices.loc and restart Galaxy (obviously doing a similar thing for BWA and adding the build to bwa_index.loc).
I thought I'd try using the Data Manager to add builds for BWA and Samtools. The BWA builds work fine (I can map to the build), but when I try to use SAM-to-BAM I get the error "Sequences are not currently available for the specified build."
Using the Data Manager creates the directory tool-data/n_sylvestris/ which contains the sub-dirs 'seq', 'bwa_index' and 'sam_index'. 'seq' contains a symlink to the n_sylvestris.fa sequence. 'sam_index' and 'bwa_index' both contains the sub-directory 'n_sylvestris', which contains a symlink to the symlink for n_sylvestris.fa in 'seq' along with their respective n_sylvestris.fa.xxx index files.
OK - all goodŠ
In tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ there are three subdirectories: data_manager_bwa_index_builder, data_manager_sam_fa_index_builder and data_manager_fetch_genome_all_fasta All three directories contain all_fasta.loc, tool_data_table_conf.xml, tool_data_table_conf.xml.sample and (for sam and bam dirs) their pertinent index.loc file.
The data_manager_fetch_genome_all_fasta/all_fasta.loc file contains the path to the fasta symlinks.
The all_fasta.loc files in the sam and bwa data_manager_index_builder directories don't contain any uncommented lines.
The index.loc files in the sam and bwa data_manager_index_builder directories point to: tool-data/n_sylvestris/bwa_index/n_sylvestris/n_sylvestris.fa tool-data/n_sylvestris/sam_index/n_sylvestris/n_sylvestris.fa
As BWA runs fine, it's obviously reading the bwa_index.loc file from the directory: tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_ind ex_builder/fe6508204acc/bwa_index.loc
...but it's not reading the samtools indexes at: tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_ index_builder/926e50397b83/sam_fa_indices.loc
For Galaxy to find the sam indexes, I have to go to the tool-data/sam_fa_indices.loc file and manually insert into it the contents of: tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_ index_builder/926e50397b83/sam_fa_indices.loc
So, I guess my question is: other than inserting the genome builds into builds.txt, should I be doing any other configuration to get Data Manager to write and configure Galaxy to read it's newly created builds. I find it strange that the BWA builds work OK, but the Samtools ones don't.
I've done a few greps for mentions of .loc files in Galaxy and the only difference between the bwa and sam .loc files is that there is a file tool-data/tool_data_table_conf.xml (plus a .sample version) which contains:
<!-- Use the file tool_data_table_conf.xml.oldlocstyle if you don't want to update your loc files as changed in revision 4550:535d276c92bc--> <tables> <!-- Location of SAMTools indexes and other files --> <table name="sam_fa_indexes" comment_char="#"> <columns>line_type, value, path</columns> <file path="tool-data/sam_fa_indices.loc" /> </table> </tables>
Could Galaxy be reading this file and ignoring the one in tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ ??
Best wishes, Graham
Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601
Hi Graham,
Which revision of Galaxy are you currently using? Currently data managers require at least 9952:a28faa6ac188 on the default branch.
When installed from the tool shed, data managers use the shed_tool_data_table_conf.xml file, could you check the contents of that file?
Also can you check the paster.log during server start up for any errors, its most likely the case that this changeset https://bitbucket.org/galaxy/galaxy-central/commits/af20b15f7eda (10145:af20b15f7eda) creates a non-compatiible tool data table entry in the main file.
Thanks for using Galaxy,
Dan
On Jul 12, 2013, at 8:14 AM, graham etherington (TSL) wrote:
Hi, I've tried using the Data Manager (Admin > Data > Manage local data (beta)) to install builds for BWA and Samtools on my local Galaxy instance. Previous to using the Data Manager, I used to add the build to tool-data/shared/ucsc/builds.txt, create the .fai indexes (for samtools) from the command line, add them to tool-data/sam_fa_indices.loc and restart Galaxy (obviously doing a similar thing for BWA and adding the build to bwa_index.loc).
I thought I'd try using the Data Manager to add builds for BWA and Samtools. The BWA builds work fine (I can map to the build), but when I try to use SAM-to-BAM I get the error "Sequences are not currently available for the specified build."
Using the Data Manager creates the directory tool-data/n_sylvestris/ which contains the sub-dirs 'seq', 'bwa_index' and 'sam_index'. 'seq' contains a symlink to the n_sylvestris.fa sequence. 'sam_index' and 'bwa_index' both contains the sub-directory 'n_sylvestris', which contains a symlink to the symlink for n_sylvestris.fa in 'seq' along with their respective n_sylvestris.fa.xxx index files.
OK - all goodŠ
In tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ there are three subdirectories: data_manager_bwa_index_builder, data_manager_sam_fa_index_builder and data_manager_fetch_genome_all_fasta All three directories contain all_fasta.loc, tool_data_table_conf.xml, tool_data_table_conf.xml.sample and (for sam and bam dirs) their pertinent index.loc file.
The data_manager_fetch_genome_all_fasta/all_fasta.loc file contains the path to the fasta symlinks.
The all_fasta.loc files in the sam and bwa data_manager_index_builder directories don't contain any uncommented lines.
The index.loc files in the sam and bwa data_manager_index_builder directories point to: tool-data/n_sylvestris/bwa_index/n_sylvestris/n_sylvestris.fa tool-data/n_sylvestris/sam_index/n_sylvestris/n_sylvestris.fa
As BWA runs fine, it's obviously reading the bwa_index.loc file from the directory: tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_ind ex_builder/fe6508204acc/bwa_index.loc
...but it's not reading the samtools indexes at: tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_ index_builder/926e50397b83/sam_fa_indices.loc
For Galaxy to find the sam indexes, I have to go to the tool-data/sam_fa_indices.loc file and manually insert into it the contents of: tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_ index_builder/926e50397b83/sam_fa_indices.loc
So, I guess my question is: other than inserting the genome builds into builds.txt, should I be doing any other configuration to get Data Manager to write and configure Galaxy to read it's newly created builds. I find it strange that the BWA builds work OK, but the Samtools ones don't.
I've done a few greps for mentions of .loc files in Galaxy and the only difference between the bwa and sam .loc files is that there is a file tool-data/tool_data_table_conf.xml (plus a .sample version) which contains:
<!-- Use the file tool_data_table_conf.xml.oldlocstyle if you don't want to update your loc files as changed in revision 4550:535d276c92bc-->
<tables> <!-- Location of SAMTools indexes and other files --> <table name="sam_fa_indexes" comment_char="#"> <columns>line_type, value, path</columns> <file path="tool-data/sam_fa_indices.loc" /> </table> </tables>
Could Galaxy be reading this file and ignoring the one in tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ ??
Best wishes, Graham
Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Dan, Thanks for your reply. I don't handle the updates, but it was updated on 25 Jun 13. I'm pretty sure I can rule out the version number as the BWA builds, created using the Data Manager, work fine.
Here is the pertinent contents of shed_tool_data_table_conf.xml
<?xml version="1.0"?> <tables> <table comment_char="#" name="all_fasta"> <columns>value, dbkey, name, path</columns> <file path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu .edu/repos/blankenberg/data_manager_bwa_index_builder/fe6508204acc/all_fast a.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep ository_name>data_manager_bwa_index_builder</repository_name><repository_ow ner>blankenberg</repository_owner><installed_changeset_revision>fe6508204ac c</installed_changeset_revision></tool_shed_repository></table>
<table comment_char="#" name="all_fasta"> <columns>value, dbkey, name, path</columns> <file path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu .edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/all_f asta.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep ository_name>data_manager_sam_fa_index_builder</repository_name><repository _owner>blankenberg</repository_owner><installed_changeset_revision>926e5039 7b83</installed_changeset_revision></tool_shed_repository></table>
<table comment_char="#" name="sam_fa_indexes"> <columns>line_type, value, path</columns> <file path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu .edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/sam_f a_indices.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep ository_name>data_manager_sam_fa_index_builder</repository_name><repository _owner>blankenberg</repository_owner><installed_changeset_revision>926e5039 7b83</installed_changeset_revision></tool_shed_repository></table>
<table comment_char="#" name="all_fasta"> <columns>value, dbkey, name, path</columns> <file path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu .edu/repos/blankenberg/data_manager_fetch_genome_all_fasta/ca8b3709309e/all _fasta.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep ository_name>data_manager_fetch_genome_all_fasta</repository_name><reposito ry_owner>blankenberg</repository_owner><installed_changeset_revision>ca8b37 09309e</installed_changeset_revision></tool_shed_repository></table> </tables>
in ${GALAXY_HOME}/tool_data_table_conf.xml.sample the entry for sam_fa reads as so:
<!-- Location of SAMTools indexes and other files --> <table name="sam_fa_indexes" comment_char="#"> <columns>line_type, value, path</columns> <file path="tool-data/sam_fa_indices.loc" /> </table>
The file tool-data/sam_fa_new_indices.loc (and .sample) does not exist.
If I keep the manually inserted builds listed in tool-data/sam_fa_indices.loc and restart Galaxy, then I get the following (abridged) entries in the paster.log: galaxy.tools.data DEBUG 2013-07-15 09:51:11,109 Loaded tool data table 'all_fasta' galaxy.tools.data DEBUG 2013-07-15 09:51:11,115 Loaded tool data table 'bwa_indexes' galaxy.tools.data DEBUG 2013-07-15 09:51:11,116 Loaded tool data table 'bwa_indexes_color' galaxy.tools.data DEBUG 2013-07-15 09:51:11,167 Loaded tool data table 'sam_fa_indexes' ...
galaxy.tools.data DEBUG 2013-07-15 09:51:11,324 Loading another instance of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,340 Loading another instance of data table 'bwa_indexes', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 09:51:11,348 Loading another instance of data table 'bwa_indexes_color', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 09:51:11,410 Loading another instance of data table 'all_fasta', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 09:51:11,422 Loading another instance of data table 'sam_fa_indexes', attempting to merge content. galaxy.tools.data ERROR 2013-07-15 09:51:11,422 Attempted to add fields (['index', 'cfraxinea_s1v1', '/home/galaxy/software/galaxy-central/tool-data/cfraxinea_s1v1/sam_index/cf raxinea_s1v1/c_fraxinea_s1v1.fa']) to data table 'sam_fa_indexes', but this entry already exists and allow_duplicates is False. galaxy.tools.data ERROR 2013-07-15 09:51:11,422 Attempted to add fields (['index', 'b_distachyon', '/home/galaxy/software/galaxy-central/tool-data/b_distachyon/sam_index/b_di stachyon/b_distachyon.fa']) to data table 'sam_fa_indexes', but this entry already exists and allow_duplicates is False. galaxy.tools.data ERROR 2013-07-15 09:51:11,423 Attempted to add fields (['index', 'n_sylvestris', '/home/galaxy/software/galaxy-central/tool-data/n_sylvestris/sam_index/n_sy lvestris/n_sylvestris.fa']) to data table 'sam_fa_indexes', but this entry already exists and allow_duplicates is False. galaxy.tools.data ERROR 2013-07-15 09:51:11,423 Attempted to add fields (['index', 'n_tomentosiformis', '/home/galaxy/software/galaxy-central/tool-data/n_tomentosiformis/sam_index /n_tomentosiformis/n_tomentosiformis.fa']) to data table 'sam_fa_indexes', but this entry already exists and allow_duplicates is False. galaxy.tools.data DEBUG 2013-07-15 09:51:11,491 Loading another instance of data table 'all_fasta', attempting to merge content.
The builds listed above with the 'entry already exists' error, are all the ones which I attempted to load using the Data Manager.
If I comment out these in entries in tool-data/sam_fa_indices.loc and restart Galaxy, the errors disappear, but when I try to use those builds to SAM-to-BAM, I'm back to the "Sequences are not currently available for the specified build." error. paster.log entries after re-start: galaxy.tools.data DEBUG 2013-07-15 10:02:56,484 Loaded tool data table 'all_fasta' galaxy.tools.data DEBUG 2013-07-15 10:02:56,491 Loaded tool data table 'bwa_indexes' galaxy.tools.data DEBUG 2013-07-15 10:02:56,492 Loaded tool data table 'bwa_indexes_color' galaxy.tools.data DEBUG 2013-07-15 10:02:56,497 Loaded tool data table 'sam_fa_indexes' ...
galaxy.tools.data DEBUG 2013-07-15 10:02:56,508 Loading another instance of data table 'all_fasta', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 10:02:56,509 Loading another instance of data table 'bwa_indexes', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 10:02:56,510 Loading another instance of data table 'bwa_indexes_color', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 10:02:56,512 Loading another instance of data table 'all_fasta', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 10:02:56,512 Loading another instance of data table 'sam_fa_indexes', attempting to merge content. galaxy.tools.data DEBUG 2013-07-15 10:02:56,514 Loading another instance of data table 'all_fasta', attempting to merge content.
On both occasions the following lines are in the paster.log galaxy.tools.data_manager.manager DEBUG 2013-07-15 09:51:47,667 Loaded Data Manager: testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_index_builder /data_manager/bwa_index_builder/0.0.1 galaxy.tools.data_manager.manager DEBUG 2013-07-15 09:51:47,689 Loaded Data Manager: testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_index_builder /data_manager/bwa_color_space_index_builder/0.0.1 galaxy.tools.data_manager.manager DEBUG 2013-07-15 09:51:47,755 Loaded Data Manager: testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_buil der/data_manager/sam_fa_index_builder/0.0.1
Any suggestions? Cheers for now, Graham
Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601
On 12/07/2013 15:45, "Daniel Blankenberg" dan@bx.psu.edu wrote:
Hi Graham,
Which revision of Galaxy are you currently using? Currently data managers require at least 9952:a28faa6ac188 on the default branch.
When installed from the tool shed, data managers use the shed_tool_data_table_conf.xml file, could you check the contents of that file?
Also can you check the paster.log during server start up for any errors, its most likely the case that this changeset https://bitbucket.org/galaxy/galaxy-central/commits/af20b15f7eda (10145:af20b15f7eda) creates a non-compatiible tool data table entry in the main file.
Thanks for using Galaxy,
Dan
On Jul 12, 2013, at 8:14 AM, graham etherington (TSL) wrote:
Hi, I've tried using the Data Manager (Admin > Data > Manage local data (beta)) to install builds for BWA and Samtools on my local Galaxy instance. Previous to using the Data Manager, I used to add the build to tool-data/shared/ucsc/builds.txt, create the .fai indexes (for samtools) from the command line, add them to tool-data/sam_fa_indices.loc and restart Galaxy (obviously doing a similar thing for BWA and adding the build to bwa_index.loc).
I thought I'd try using the Data Manager to add builds for BWA and Samtools. The BWA builds work fine (I can map to the build), but when I try to use SAM-to-BAM I get the error "Sequences are not currently available for the specified build."
Using the Data Manager creates the directory tool-data/n_sylvestris/ which contains the sub-dirs 'seq', 'bwa_index' and 'sam_index'. 'seq' contains a symlink to the n_sylvestris.fa sequence. 'sam_index' and 'bwa_index' both contains the sub-directory 'n_sylvestris', which contains a symlink to the symlink for n_sylvestris.fa in 'seq' along with their respective n_sylvestris.fa.xxx index files.
OK - all goodŠ
In tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ there are three subdirectories: data_manager_bwa_index_builder, data_manager_sam_fa_index_builder and data_manager_fetch_genome_all_fasta All three directories contain all_fasta.loc, tool_data_table_conf.xml, tool_data_table_conf.xml.sample and (for sam and bam dirs) their pertinent index.loc file.
The data_manager_fetch_genome_all_fasta/all_fasta.loc file contains the path to the fasta symlinks.
The all_fasta.loc files in the sam and bwa data_manager_index_builder directories don't contain any uncommented lines.
The index.loc files in the sam and bwa data_manager_index_builder directories point to: tool-data/n_sylvestris/bwa_index/n_sylvestris/n_sylvestris.fa tool-data/n_sylvestris/sam_index/n_sylvestris/n_sylvestris.fa
As BWA runs fine, it's obviously reading the bwa_index.loc file from the directory:
tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_i nd ex_builder/fe6508204acc/bwa_index.loc
...but it's not reading the samtools indexes at:
tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_f a_ index_builder/926e50397b83/sam_fa_indices.loc
For Galaxy to find the sam indexes, I have to go to the tool-data/sam_fa_indices.loc file and manually insert into it the contents of:
tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_f a_ index_builder/926e50397b83/sam_fa_indices.loc
So, I guess my question is: other than inserting the genome builds into builds.txt, should I be doing any other configuration to get Data Manager to write and configure Galaxy to read it's newly created builds. I find it strange that the BWA builds work OK, but the Samtools ones don't.
I've done a few greps for mentions of .loc files in Galaxy and the only difference between the bwa and sam .loc files is that there is a file tool-data/tool_data_table_conf.xml (plus a .sample version) which contains:
<!-- Use the file tool_data_table_conf.xml.oldlocstyle if you don't want to update your loc files as changed in revision 4550:535d276c92bc-->
<tables> <!-- Location of SAMTools indexes and other files --> <table name="sam_fa_indexes" comment_char="#"> <columns>line_type, value, path</columns> <file path="tool-data/sam_fa_indices.loc" /> </table> </tables>
Could Galaxy be reading this file and ignoring the one in tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ ??
Best wishes, Graham
Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Daniel Blankenberg -
graham etherington (TSL)