Hi Ravi,
The directory structure for the installation of ToolShed tools
changed, which is why you have three directories. You perhaps had
bowtie2 installed once before, then reinstalled (without completely
removing the older version and associated data)? Or updated without
resetting the metadata? In either case, the ../shed_tools directory
is the new one. Having this as the path in your .xml configuration
files (as Bjoern suggested earlier) and moving all contents &
data to be under the same location will be the simplest global
solution ongoing. Links near the end of my reply can help explain
how-to.
For the specific reason why bowtie2 indices are not working: I
noticed that the reference genome ".fa" file is not linked from the
directory containing the indexes. This is required. Adding it in,
the same way that you did for the bowtie2 indexes, into this dir:
"/global/referenceData/databases/bowtie2/hg19" will probably solve
that part of the problem. I didn't see this posted - but I apologize
if I am duplicating advice already given.
I also tend to advise keeping all data under the same master "data"
directory - all indexes and sequence data - as symbolic links to
additional file system paths that are unknown to the 'galaxy user'
cause a different set of problems. However, that said, this doesn't
seem to be an issue in your specific case: if the bowtie2 indexes
are functioning correctly - then environment is set up so that the
other dir hierarchy where the .fa files are kept must included in
the 'galaxy' user's ENV. Symbolic links that go outside of the local
dir structure are known to cause problems unless the ENV config is
carefully set up, and to my knowledge are best avoided entirely for
certain uses such as "upload by file path" into libraries.
For reference:
NGS data set-up is described in this wiki - including expected
content for each type of index:
https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup
For examples, you can rsync a genome or two and examine the
contents. Or, our /location dir and have a look at the .loc files.
https://wiki.galaxyproject.org/Admin/DataIntegration
Tool Shed help (very detailed):
https://wiki.galaxyproject.org/Tool%20Shed
In particular, if you had previously installed repositories (this is
not clear, just suspected from the duplications), updating the
Metadata with certain distribution updates can be very important.
This has been necessary for the last few releases to update to
changes. The News Brief noted this, and included a link to this wiki
page. Also see the "Related Pages" lower down on the wiki.
https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories
This may also be useful:
https://wiki.galaxyproject.org/RepairingInstalledRepositories
Hopefully you have sorted most of this out by now, or this helps!
Jen
Galaxy team
On 3/4/14 12:21 PM, Ravi Alla wrote:
Bjoern,
This is getting frustrating.
There are three places bowtie2_indices.loc file is expected.
I really don't know which one is the one I should modify.
galaxy-dist/tool-data/bowtie2_indices.loc
What is the difference between
these 3 files?
I changed the 2nd file to include
the path to the indices and this shows up as a valid
preloaded index for tophat2 but does not for bowtie2. I
also changed the normal bowtie_indices.loc file at the
second location and this works and I can see the bowtie1
indices for that bowtie tool. Currently only the bowtie2
indices are acting up.
I wish this was easier.
Thanks
Ravi.
Hi,
can you check if the file tool_data_table_conf.xml contains
an entry about bowtie2_indexes ... or the table
shed_data_table_conf.xml ... in one of them should be one
entry with bowtie2_indexes.
That warning should not be there I think:
> galaxy.tools.parameters.dynamic_options WARNING
2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes'
is required by tool but not configured
Hope you can figure it out,
Bjoern
Am 04.03.2014 19:19, schrieb Ravi Alla:
Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the
paster.log file I see these entries for bowtie
galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded
tool data table 'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading
another instance of data table 'bowtie_indexes',
attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04
10:14:33,335 Data table named 'bowtie2_indexes' is
required by tool but not configured
And ls -l /global/referenceData/databases/bowtie2/hg19 is
drwxr-xr-x 2 ralla cgrl 2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl 2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May 2 2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May 2 2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl 3833 May 2 2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May 2 2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May 2 2012
hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May 2 2012
hg19.rev.2.bt2
and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl 2048 Mar 4 10:07 .
drwxr-xr-x 3 ralla cgrl 2048 Nov 30 2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13 2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13 2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl 3284 Nov 13 2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13 2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl 45 Aug 21 2013 hg19.fa
-> /global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13 2009
hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13 2009
hg19.rev.2.ebwt
Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <bjoern.gruening@gmail.com>
wrote:
Hi Ravi,
can you attach the loc file, do you see anything in the
Galaxy log files about bowtie2, try grepping for
"bowtie2".
Cheers,
Bjoern
Am 04.03.2014 18:07, schrieb Ravi Alla:
Hi Bjoern,
Thank you for your clarifications. I have tried
installing bowtie2 using both variations, once with
both wrapper and dependencies and once with only
wrapper (since I have dependencies installed on the
system). In either case even after editing the .loc
file I cannot see the indices in galaxy. I tried the
same edits to .loc file with bwa and the indices show
right up, but not with bowtie2 and even bowtie for
that matter. I really don't know what to do about
this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <bjoern.gruening@gmail.com>
wrote:
Hi Ravi,
Hi guys,
I am new to galaxy and am in the process of
setting it up on a cluster with some help. I am
trying to install bowtie2 to my local galaxy
through the toolshed. I have a few questions.
- Which bowtie2 should I install? When I search
for bowtie2 a bunch of results come up?
Only two, or? Please make sure you are using the
main toolshed. bowtie2 and package_bowtie2_2_1_0 are
connected together. One is the binary and the other
contains the wrapper. Use devteam respositories is
always a good choice.
- What are repository
dependencies? Why would I need these if bowtie2 is
already installed on my system?
repository dependecies containing all dependencies
of your wrappers, that can be some binaries, but
also R, Perl, python libraries that are needed to
execute your wrappers. We recommend you to use the
dependencies because then you have full control over
all your used versions (wrapper, dependencies, tool)
and that enables reproducibility. To put in other
words: With the toolshed you can have different tool
and wrappers versions with different dependencies at
the same time.
- I have been installing
bowtie2 by unchecking the repository dependencies
and tool dependencies boxes. Is this correct?
It will work, if you have a system installed
version. But you need to take care about binary
updates by your own. And reproducibility of your
results is not guaranteed.
- Finally when I install
bowtie2 this way it creates 2 bowtie2_indices.loc
files, one in galaxy/tool-data and the other in
shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2,
which one do I need to edit to point to my index
files? No matter which one I change I can't seem
to see the indices.
That is strange. Are you sure you don't have a
spelling mistake in it. It should be the
bowtie2_indices.los I think. I will CC Greg, he is
one of the Tool Shed Developers and should know more
about it.
Please do not forget to restart your Galaxy instance
once you have update the *loc files.
I hope someone on here can
help me out.
Hope it helped a little bit,
Bjoern
Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using
"reply all"
in your mail client. To manage your subscriptions
to this
and other Galaxy lists, please use the interface
at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified
search at:
http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
--
Jennifer Hillman-Jackson
http://galaxyproject.org