BWA Illumina Mapping / BWA Reference Genome
Hello Ladies and Gentlemen, I am Moritz Juchler from University Heidelberg. For my Bachelor thesis I have to choose a bioinformatic pipeline management tool to find SNP's in genomes from hcc patients. My decision was made in favor of galaxy. I have a 64-bit openSuse 11.3 server. I have installed Galaxy locally, since we have a) very large files (>30GB per patient) and b) the data is protection sensitive. I kept close to http://wiki.galaxyproject.org/Admin/Get%20Galaxy Now I would like to run this bpipe pipeline: http://pastebin.com/sZd5vfdL And the first step is to align my genome to a *hg19 reference genome* which I have locally under /genedata/human_genome_GRCh37/. trr@portalmoritz:~> ls -l /genedata/human_genome_GRCh37/
total 8486312 -rw-r--r-- 1 trr root 3199905909 2013-06-25 16:44 hg19.fa -rw-r--r-- 1 trr root 8591 2013-07-01 16:06 hg19.fa.amb -rw-r--r-- 1 trr root 4040 2013-07-01 16:06 hg19.fa.ann -rw-r--r-- 1 trr root 3137161344 2013-07-01 16:05 hg19.fa.bwt -rw-r--r-- 1 trr root 784290318 2013-07-01 16:06 hg19.fa.pac -rw-r--r-- 1 trr root 1568580688 2013-07-01 16:31 hg19.fa.sa
*bwa is installed and gives me:*
trr@portalmoritz:~> bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.5a-r405 Contact: Heng Li <lh3@sanger.ac.uk>
Then I tried to follow this guide: http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup to get the reference files and http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies. This is my *$PATH*
trr@portalmoritz:~> echo $PATH
/home/trr/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/home/trr/bpipe-0.9.8/bin:/home/trr/bwa-0.7.5a:/home/trr/samtools-0.1.19
*In the universe_wsgi.ini I changed:*
tool_dependency_dir = /home/trr/galaxy-dist/tool_dependency_dir debug = False use_interactive = True library_import_dir = /genedata/ allow_library_path_paste = True admin_users = ...
This is my *tool_dependency_dir:*
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa> ls -l total 4 drwxr-xr-x 3 trr users 4096 2013-07-16 14:28 0.7.4 lrwxrwxrwx 1 trr users 6 2013-07-16 14:17 default -> 0.7.4/
This is the* version folder of bwa:*
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa/0.7.4> ls -l total 8 drwxr-xr-x 2 trr users 4096 2013-07-16 14:18 bin -rw-r--r-- 1 trr users 47 2013-07-16 14:22 env.sh
This is the *content of env.sh:*
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa/0.7.4> cat env.sh PATH="/home/trr/bwa-0.7.5a/:$PATH" export PATH
And this is the *content of the bin folder:*
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa/0.7.4/bin> ls -l total 3896 -rw-r--r-- 1 trr users 6098 2013-07-16 14:18 bamlite.c -rw-r--r-- 1 trr users 3124 2013-07-16 14:18 bamlite.h -rw-r--r-- 1 trr users 24816 2013-07-16 14:18 bamlite.o -rw-r--r-- 1 trr users 11508 2013-07-16 14:18 bntseq.c -rw-r--r-- 1 trr users 2557 2013-07-16 14:18 bntseq.h -rw-r--r-- 1 trr users 37440 2013-07-16 14:18 bntseq.o -rwxr-xr-x 1 trr users 998217 2013-07-16 14:18 bwa -rw-r--r-- 1 trr users 24225 2013-07-16 14:18 bwa.1 -rw-r--r-- 1 trr users 9416 2013-07-16 14:18 bwa.c -rw-r--r-- 1 trr users 1381 2013-07-16 14:18 bwa.h
.... I got the xmls and .py from https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping and i didnt change them at all and put them into ~/galaxy-dist/*tools/sr_mapping * (since they were missing in this folder)
bwa_color_wrapper.xml bwa_wrapper.py bwa_wrapper.xml
I added *bwa_index_color.loc and bwa_index.loc* to ~galaxy-dist/*tool-data* (they were missing as well, there* were no* bwa_index_color.loc.sample or bwa_index.lox.sample files!!!) I only have this single line in both bwa_index_color.loc and bwa_index.loc
trr@portalmoritz:~/galaxy-dist/tool-data> cat bwa_index_color.loc #This is a sample file distributed with Galaxy that enables tools # #<unique_build_id> <dbkey> <display_name> <file_path> hg19 hg19 hg19 /genedata/human_genome_GRCh37/hg19.fa
(Spaces are actually tabs!) After all that, I neither have the "Map with BWA for Illumina<https://main.g2.bx.psu.edu/tool_runner?tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3>" in my local Galaxy version, nor do I find the reference genome. If i missed on any required, please tell me, I will answer you as soon as possible. Sincerly Yours Moritz Juchler
Hi Moritz I am struggling to follow what exactly you have done. As far as I can see, you did not use the toolshed (http://wiki.galaxyproject.org/Tool%20Shed) to install the BWA alinger tool, but did all manually? If so, have you added the following line: <tool file="sr_mapping/bwa_wrapper.xml" /> to the "tool_conf.xml" file, and restarted Galaxy? see also: http://wiki.galaxyproject.org/Admin/Tools/Add%20Tool%20Tutorial Hope this helps Hans-Rudolf On 07/16/2013 08:35 PM, Moritz Juchler wrote:
Hello Ladies and Gentlemen,
I am Moritz Juchler from University Heidelberg. For my Bachelor thesis I have to choose a bioinformatic pipeline management tool to find SNP's in genomes from hcc patients. My decision was made in favor of galaxy. I have a 64-bit openSuse 11.3 server. I have installed Galaxy locally, since we have a) very large files (>30GB per patient) and b) the data is protection sensitive. I kept close to http://wiki.galaxyproject.org/Admin/Get%20Galaxy Now I would like to run this bpipe pipeline: http://pastebin.com/sZd5vfdL And the first step is to align my genome to a _hg19 reference genome_ which I have locally under /genedata/human_genome_GRCh37/.
trr@portalmoritz:~> ls -l /genedata/human_genome_GRCh37/ total 8486312 -rw-r--r-- 1 trr root 3199905909 2013-06-25 16:44 hg19.fa -rw-r--r-- 1 trr root 8591 2013-07-01 16:06 hg19.fa.amb -rw-r--r-- 1 trr root 4040 2013-07-01 16:06 hg19.fa.ann -rw-r--r-- 1 trr root 3137161344 2013-07-01 16:05 hg19.fa.bwt -rw-r--r-- 1 trr root 784290318 2013-07-01 16:06 hg19.fa.pac -rw-r--r-- 1 trr root 1568580688 2013-07-01 16:31 hg19.fa.sa <http://hg19.fa.sa>
_bwa is installed and gives me:_
trr@portalmoritz:~> bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.5a-r405 Contact: Heng Li <lh3@sanger.ac.uk <mailto:lh3@sanger.ac.uk>>
Then I tried to follow this guide: http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup to get the reference files and http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies.
This is my _$PATH_
trr@portalmoritz:~> echo $PATH /home/trr/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/home/trr/bpipe-0.9.8/bin:/home/trr/bwa-0.7.5a:/home/trr/samtools-0.1.19
_In the universe_wsgi.ini I changed:_
tool_dependency_dir = /home/trr/galaxy-dist/tool_dependency_dir debug = False use_interactive = True library_import_dir = /genedata/ allow_library_path_paste = True admin_users = ...
This is my _tool_dependency_dir:_
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa> ls -l total 4 drwxr-xr-x 3 trr users 4096 2013-07-16 14:28 0.7.4 lrwxrwxrwx 1 trr users 6 2013-07-16 14:17 default -> 0.7.4/
This is the_version folder of bwa:_
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa/0.7.4> ls -l total 8 drwxr-xr-x 2 trr users 4096 2013-07-16 14:18 bin -rw-r--r-- 1 trr users 47 2013-07-16 14:22 env.sh
This is the _content of env.sh:_
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa/0.7.4> cat env.sh PATH="/home/trr/bwa-0.7.5a/:$PATH" export PATH
And this is the _content of the bin folder:_
trr@portalmoritz:~/galaxy-dist/tool_dependency_dir/bwa/0.7.4/bin> ls -l total 3896 -rw-r--r-- 1 trr users 6098 2013-07-16 14:18 bamlite.c -rw-r--r-- 1 trr users 3124 2013-07-16 14:18 bamlite.h -rw-r--r-- 1 trr users 24816 2013-07-16 14:18 bamlite.o -rw-r--r-- 1 trr users 11508 2013-07-16 14:18 bntseq.c -rw-r--r-- 1 trr users 2557 2013-07-16 14:18 bntseq.h -rw-r--r-- 1 trr users 37440 2013-07-16 14:18 bntseq.o -rwxr-xr-x 1 trr users 998217 2013-07-16 14:18 bwa -rw-r--r-- 1 trr users 24225 2013-07-16 14:18 bwa.1 -rw-r--r-- 1 trr users 9416 2013-07-16 14:18 bwa.c -rw-r--r-- 1 trr users 1381 2013-07-16 14:18 bwa.h
....
I got the xmls and .py from https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping and i didnt change them at all and put them into ~/galaxy-dist/_tools/sr_mapping_ (since they were missing in this folder)
bwa_color_wrapper.xml bwa_wrapper.py bwa_wrapper.xml
I added _bwa_index_color.loc and bwa_index.loc_ to ~galaxy-dist/_tool-data_ (they were missing as well, there/*were no*/ bwa_index_color.loc.sample or bwa_index.lox.sample files!!!)
I only have this single line in both bwa_index_color.loc and bwa_index.loc
trr@portalmoritz:~/galaxy-dist/tool-data> cat bwa_index_color.loc #This is a sample file distributed with Galaxy that enables tools # #<unique_build_id> <dbkey> <display_name> <file_path> hg19 hg19 hg19 /genedata/human_genome_GRCh37/hg19.fa
(Spaces are actually tabs!)
After all that, I neither have the "Map with BWA for Illumina <https://main.g2.bx.psu.edu/tool_runner?tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3>" in my local Galaxy version, nor do I find the reference genome. If i missed on any required, please tell me, I will answer you as soon as possible. Sincerly Yours Moritz Juchler
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hey, no thats correct I did not use Tool Shed. If I would use it, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle? How did you not follow my steps :) I wrote down everything clearly, at least thats what I was hoping. And no I didnt write this line " <tool file="sr_mapping/bwa_wrapper.**xml" />" Thats the first time, I am seeing this tutorial :( I will try this out right now Best Moritz On 17 July 2013 09:42, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
Hi Moritz
I am struggling to follow what exactly you have done. As far as I can see, you did not use the toolshed (http://wiki.galaxyproject.**org/Tool%20Shed<http://wiki.galaxyproject.org/Tool%20Shed>) to install the BWA alinger tool, but did all manually?
If so, have you added the following line:
<tool file="sr_mapping/bwa_wrapper.**xml" />
to the "tool_conf.xml" file, and restarted Galaxy?
Hope this helps Hans-Rudolf
On 07/16/2013 08:35 PM, Moritz Juchler wrote:
Hello Ladies and Gentlemen,
I am Moritz Juchler from University Heidelberg. For my Bachelor thesis I have to choose a bioinformatic pipeline management tool to find SNP's in genomes from hcc patients. My decision was made in favor of galaxy. I have a 64-bit openSuse 11.3 server. I have installed Galaxy locally, since we have a) very large files (>30GB per patient) and b) the data is protection sensitive. I kept close to http://wiki.galaxyproject.org/**Admin/Get%20Galaxy<http://wiki.galaxyproject.org/Admin/Get%20Galaxy> Now I would like to run this bpipe pipeline: http://pastebin.com/sZd5vfdL And the first step is to align my genome to a _hg19 reference genome_
which I have locally under /genedata/human_genome_GRCh37/**.
trr@portalmoritz:~> ls -l /genedata/human_genome_GRCh37/ total 8486312 -rw-r--r-- 1 trr root 3199905909 2013-06-25 16:44 hg19.fa -rw-r--r-- 1 trr root 8591 2013-07-01 16:06 hg19.fa.amb -rw-r--r-- 1 trr root 4040 2013-07-01 16:06 hg19.fa.ann -rw-r--r-- 1 trr root 3137161344 2013-07-01 16:05 hg19.fa.bwt -rw-r--r-- 1 trr root 784290318 2013-07-01 16:06 hg19.fa.pac -rw-r--r-- 1 trr root 1568580688 2013-07-01 16:31 hg19.fa.sa <http://hg19.fa.sa>
_bwa is installed and gives me:_
trr@portalmoritz:~> bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.5a-r405 Contact: Heng Li <lh3@sanger.ac.uk <mailto:lh3@sanger.ac.uk>>
Then I tried to follow this guide: http://wiki.galaxyproject.org/**Admin/NGS%20Local%20Setup<http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup>to get the reference files and http://wiki.galaxyproject.org/**Admin/Config/Tool%**20Dependencies<http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies> .
This is my _$PATH_
trr@portalmoritz:~> echo $PATH /home/trr/bin:/usr/local/bin:/**usr/bin:/bin:/usr/bin/X11:/** usr/X11R6/bin:/usr/games:/**home/trr/bpipe-0.9.8/bin:/** home/trr/bwa-0.7.5a:/home/trr/**samtools-0.1.19
_In the universe_wsgi.ini I changed:_
tool_dependency_dir = /home/trr/galaxy-dist/tool_**dependency_dir debug = False use_interactive = True library_import_dir = /genedata/ allow_library_path_paste = True admin_users = ...
This is my _tool_dependency_dir:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa> ls -l total 4 drwxr-xr-x 3 trr users 4096 2013-07-16 14:28 0.7.4 lrwxrwxrwx 1 trr users 6 2013-07-16 14:17 default -> 0.7.4/
This is the_version folder of bwa:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4> ls -l total 8 drwxr-xr-x 2 trr users 4096 2013-07-16 14:18 bin -rw-r--r-- 1 trr users 47 2013-07-16 14:22 env.sh
This is the _content of env.sh:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4> cat env.sh PATH="/home/trr/bwa-0.7.5a/:$**PATH" export PATH
And this is the _content of the bin folder:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4/bin> ls -l total 3896 -rw-r--r-- 1 trr users 6098 2013-07-16 14:18 bamlite.c -rw-r--r-- 1 trr users 3124 2013-07-16 14:18 bamlite.h -rw-r--r-- 1 trr users 24816 2013-07-16 14:18 bamlite.o -rw-r--r-- 1 trr users 11508 2013-07-16 14:18 bntseq.c -rw-r--r-- 1 trr users 2557 2013-07-16 14:18 bntseq.h -rw-r--r-- 1 trr users 37440 2013-07-16 14:18 bntseq.o -rwxr-xr-x 1 trr users 998217 2013-07-16 14:18 bwa -rw-r--r-- 1 trr users 24225 2013-07-16 14:18 bwa.1 -rw-r--r-- 1 trr users 9416 2013-07-16 14:18 bwa.c -rw-r--r-- 1 trr users 1381 2013-07-16 14:18 bwa.h
....
I got the xmls and .py from https://bitbucket.org/galaxy/**galaxy-dist/src/da9d740fce31/** tools/sr_mapping<https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping>and i didnt change them at all and put them into ~/galaxy-dist/_tools/sr_**mapping_ (since they were missing in this folder)
bwa_color_wrapper.xml bwa_wrapper.py bwa_wrapper.xml
I added _bwa_index_color.loc and bwa_index.loc_ to ~galaxy-dist/_tool-data_ (they were missing as well, there/*were no*/
bwa_index_color.loc.sample or bwa_index.lox.sample files!!!)
I only have this single line in both bwa_index_color.loc and bwa_index.loc
trr@portalmoritz:~/galaxy-**dist/tool-data> cat bwa_index_color.loc #This is a sample file distributed with Galaxy that enables tools # #<unique_build_id> <dbkey> <display_name> <file_path> hg19 hg19 hg19 /genedata/human_genome_GRCh37/**hg19.fa
(Spaces are actually tabs!)
After all that, I neither have the "Map with BWA for Illumina <https://main.g2.bx.psu.edu/**tool_runner?tool_id=toolshed.** g2.bx.psu.edu/repos/devteam/**bwa_wrappers/bwa_wrapper/1.2.3<https://main.g2.bx.psu.edu/tool_runner?tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3> **>"
in my local Galaxy version, nor do I find the reference genome. If i missed on any required, please tell me, I will answer you as soon as possible. Sincerly Yours Moritz Juchler
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/**search/mailinglists/<http://galaxyproject.org/search/mailinglists/>
Hey, with the "tool file" line I got it working :) Thanks a lot. Could you answer the question regarding Tool Shed: "If I would use Tool Shed, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle?" Best Moritz On 17 July 2013 09:51, Moritz Juchler <juchler@stud.uni-heidelberg.de>wrote:
Hey,
no thats correct I did not use Tool Shed. If I would use it, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle?
How did you not follow my steps :) I wrote down everything clearly, at least thats what I was hoping.
And no I didnt write this line " <tool file="sr_mapping/bwa_wrapper.**xml" />" Thats the first time, I am seeing this tutorial :( I will try this out right now
Best Moritz
On 17 July 2013 09:42, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
Hi Moritz
I am struggling to follow what exactly you have done. As far as I can see, you did not use the toolshed (http://wiki.galaxyproject.** org/Tool%20Shed <http://wiki.galaxyproject.org/Tool%20Shed>) to install the BWA alinger tool, but did all manually?
If so, have you added the following line:
<tool file="sr_mapping/bwa_wrapper.**xml" />
to the "tool_conf.xml" file, and restarted Galaxy?
Hope this helps Hans-Rudolf
On 07/16/2013 08:35 PM, Moritz Juchler wrote:
Hello Ladies and Gentlemen,
I am Moritz Juchler from University Heidelberg. For my Bachelor thesis I have to choose a bioinformatic pipeline management tool to find SNP's in genomes from hcc patients. My decision was made in favor of galaxy. I have a 64-bit openSuse 11.3 server. I have installed Galaxy locally, since we have a) very large files (>30GB per patient) and b) the data is protection sensitive. I kept close to http://wiki.galaxyproject.org/**Admin/Get%20Galaxy<http://wiki.galaxyproject.org/Admin/Get%20Galaxy> Now I would like to run this bpipe pipeline: http://pastebin.com/sZd5vfdL And the first step is to align my genome to a _hg19 reference genome_
which I have locally under /genedata/human_genome_GRCh37/**.
trr@portalmoritz:~> ls -l /genedata/human_genome_GRCh37/ total 8486312 -rw-r--r-- 1 trr root 3199905909 2013-06-25 16:44 hg19.fa -rw-r--r-- 1 trr root 8591 2013-07-01 16:06 hg19.fa.amb -rw-r--r-- 1 trr root 4040 2013-07-01 16:06 hg19.fa.ann -rw-r--r-- 1 trr root 3137161344 2013-07-01 16:05 hg19.fa.bwt -rw-r--r-- 1 trr root 784290318 2013-07-01 16:06 hg19.fa.pac -rw-r--r-- 1 trr root 1568580688 2013-07-01 16:31 hg19.fa.sa <http://hg19.fa.sa>
_bwa is installed and gives me:_
trr@portalmoritz:~> bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.5a-r405 Contact: Heng Li <lh3@sanger.ac.uk <mailto:lh3@sanger.ac.uk>>
Then I tried to follow this guide: http://wiki.galaxyproject.org/**Admin/NGS%20Local%20Setup<http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup>to get the reference files and http://wiki.galaxyproject.org/**Admin/Config/Tool%**20Dependencies<http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies> .
This is my _$PATH_
trr@portalmoritz:~> echo $PATH /home/trr/bin:/usr/local/bin:/**usr/bin:/bin:/usr/bin/X11:/** usr/X11R6/bin:/usr/games:/**home/trr/bpipe-0.9.8/bin:/** home/trr/bwa-0.7.5a:/home/trr/**samtools-0.1.19
_In the universe_wsgi.ini I changed:_
tool_dependency_dir = /home/trr/galaxy-dist/tool_**dependency_dir debug = False use_interactive = True library_import_dir = /genedata/ allow_library_path_paste = True admin_users = ...
This is my _tool_dependency_dir:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa> ls -l total 4 drwxr-xr-x 3 trr users 4096 2013-07-16 14:28 0.7.4 lrwxrwxrwx 1 trr users 6 2013-07-16 14:17 default -> 0.7.4/
This is the_version folder of bwa:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4> ls -l total 8 drwxr-xr-x 2 trr users 4096 2013-07-16 14:18 bin -rw-r--r-- 1 trr users 47 2013-07-16 14:22 env.sh
This is the _content of env.sh:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4> cat env.sh PATH="/home/trr/bwa-0.7.5a/:$**PATH" export PATH
And this is the _content of the bin folder:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4/bin> ls -l total 3896 -rw-r--r-- 1 trr users 6098 2013-07-16 14:18 bamlite.c -rw-r--r-- 1 trr users 3124 2013-07-16 14:18 bamlite.h -rw-r--r-- 1 trr users 24816 2013-07-16 14:18 bamlite.o -rw-r--r-- 1 trr users 11508 2013-07-16 14:18 bntseq.c -rw-r--r-- 1 trr users 2557 2013-07-16 14:18 bntseq.h -rw-r--r-- 1 trr users 37440 2013-07-16 14:18 bntseq.o -rwxr-xr-x 1 trr users 998217 2013-07-16 14:18 bwa -rw-r--r-- 1 trr users 24225 2013-07-16 14:18 bwa.1 -rw-r--r-- 1 trr users 9416 2013-07-16 14:18 bwa.c -rw-r--r-- 1 trr users 1381 2013-07-16 14:18 bwa.h
....
I got the xmls and .py from https://bitbucket.org/galaxy/**galaxy-dist/src/da9d740fce31/** tools/sr_mapping<https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping>and i didnt change them at all and put them into ~/galaxy-dist/_tools/sr_**mapping_ (since they were missing in this folder)
bwa_color_wrapper.xml bwa_wrapper.py bwa_wrapper.xml
I added _bwa_index_color.loc and bwa_index.loc_ to ~galaxy-dist/_tool-data_ (they were missing as well, there/*were no*/
bwa_index_color.loc.sample or bwa_index.lox.sample files!!!)
I only have this single line in both bwa_index_color.loc and bwa_index.loc
trr@portalmoritz:~/galaxy-**dist/tool-data> cat bwa_index_color.loc #This is a sample file distributed with Galaxy that enables tools # #<unique_build_id> <dbkey> <display_name> <file_path> hg19 hg19 hg19 /genedata/human_genome_GRCh37/**hg19.fa
(Spaces are actually tabs!)
After all that, I neither have the "Map with BWA for Illumina <https://main.g2.bx.psu.edu/**tool_runner?tool_id=toolshed.** g2.bx.psu.edu/repos/devteam/**bwa_wrappers/bwa_wrapper/1.2.3<https://main.g2.bx.psu.edu/tool_runner?tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3> **>"
in my local Galaxy version, nor do I find the reference genome. If i missed on any required, please tell me, I will answer you as soon as possible. Sincerly Yours Moritz Juchler
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/**search/mailinglists/<http://galaxyproject.org/search/mailinglists/>
Hey, now I am having a new problem: *Convert SAM to BAM *Tool execution generated the following error message: [samopen] SAM header is present: 93 sequences. Parse error at line 106: sequence and quality are inconsistent /bin/sh: line 1: 27934 Aborted samtools view -bS "/home/trr/galaxy-dist/database/files/000/dataset_17.dat" > "/tmp/tmp-sam_to_bam_converter-ut5Tag/unsorted.bam" The tool produced the following additional output: [bam_header_read] EOF marker is absent. The input is probably truncated. (should I make a new post out of this?) The step I did before was: 7: *Map* with BWA for Illumina on data 5 and data 3: mapped reads ~21,000 lines, 94 comments format: sam, database: hg19 BWA Version: 0.7.5a-r405 BWA run on paired-end data That one seems to work correctly. Any help appreciated :) Best Moritz On 22 July 2013 11:20, Moritz Juchler <juchler@stud.uni-heidelberg.de>wrote:
Hey,
with the "tool file" line I got it working :) Thanks a lot. Could you answer the question regarding Tool Shed: "If I would use Tool Shed, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle?"
Best Moritz
On 17 July 2013 09:51, Moritz Juchler <juchler@stud.uni-heidelberg.de>wrote:
Hey,
no thats correct I did not use Tool Shed. If I would use it, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle?
How did you not follow my steps :) I wrote down everything clearly, at least thats what I was hoping.
And no I didnt write this line " <tool file="sr_mapping/bwa_wrapper.**xml" />" Thats the first time, I am seeing this tutorial :( I will try this out right now
Best Moritz
On 17 July 2013 09:42, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
Hi Moritz
I am struggling to follow what exactly you have done. As far as I can see, you did not use the toolshed (http://wiki.galaxyproject.** org/Tool%20Shed <http://wiki.galaxyproject.org/Tool%20Shed>) to install the BWA alinger tool, but did all manually?
If so, have you added the following line:
<tool file="sr_mapping/bwa_wrapper.**xml" />
to the "tool_conf.xml" file, and restarted Galaxy?
Hope this helps Hans-Rudolf
On 07/16/2013 08:35 PM, Moritz Juchler wrote:
Hello Ladies and Gentlemen,
I am Moritz Juchler from University Heidelberg. For my Bachelor thesis I have to choose a bioinformatic pipeline management tool to find SNP's in genomes from hcc patients. My decision was made in favor of galaxy. I have a 64-bit openSuse 11.3 server. I have installed Galaxy locally, since we have a) very large files (>30GB per patient) and b) the data is protection sensitive. I kept close to http://wiki.galaxyproject.org/**Admin/Get%20Galaxy<http://wiki.galaxyproject.org/Admin/Get%20Galaxy> Now I would like to run this bpipe pipeline: http://pastebin.com/sZd5vfdL And the first step is to align my genome to a _hg19 reference genome_
which I have locally under /genedata/human_genome_GRCh37/**.
trr@portalmoritz:~> ls -l /genedata/human_genome_GRCh37/ total 8486312 -rw-r--r-- 1 trr root 3199905909 2013-06-25 16:44 hg19.fa -rw-r--r-- 1 trr root 8591 2013-07-01 16:06 hg19.fa.amb -rw-r--r-- 1 trr root 4040 2013-07-01 16:06 hg19.fa.ann -rw-r--r-- 1 trr root 3137161344 2013-07-01 16:05 hg19.fa.bwt -rw-r--r-- 1 trr root 784290318 2013-07-01 16:06 hg19.fa.pac -rw-r--r-- 1 trr root 1568580688 2013-07-01 16:31 hg19.fa.sa <http://hg19.fa.sa>
_bwa is installed and gives me:_
trr@portalmoritz:~> bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.5a-r405 Contact: Heng Li <lh3@sanger.ac.uk <mailto:lh3@sanger.ac.uk>>
Then I tried to follow this guide: http://wiki.galaxyproject.org/**Admin/NGS%20Local%20Setup<http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup>to get the reference files and http://wiki.galaxyproject.org/**Admin/Config/Tool%**20Dependencies<http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies> .
This is my _$PATH_
trr@portalmoritz:~> echo $PATH /home/trr/bin:/usr/local/bin:/**usr/bin:/bin:/usr/bin/X11:/** usr/X11R6/bin:/usr/games:/**home/trr/bpipe-0.9.8/bin:/** home/trr/bwa-0.7.5a:/home/trr/**samtools-0.1.19
_In the universe_wsgi.ini I changed:_
tool_dependency_dir = /home/trr/galaxy-dist/tool_**dependency_dir debug = False use_interactive = True library_import_dir = /genedata/ allow_library_path_paste = True admin_users = ...
This is my _tool_dependency_dir:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa> ls -l total 4 drwxr-xr-x 3 trr users 4096 2013-07-16 14:28 0.7.4 lrwxrwxrwx 1 trr users 6 2013-07-16 14:17 default -> 0.7.4/
This is the_version folder of bwa:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4> ls -l total 8 drwxr-xr-x 2 trr users 4096 2013-07-16 14:18 bin -rw-r--r-- 1 trr users 47 2013-07-16 14:22 env.sh
This is the _content of env.sh:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4> cat env.sh PATH="/home/trr/bwa-0.7.5a/:$**PATH" export PATH
And this is the _content of the bin folder:_
trr@portalmoritz:~/galaxy-**dist/tool_dependency_dir/bwa/**0.7.4/bin> ls -l total 3896 -rw-r--r-- 1 trr users 6098 2013-07-16 14:18 bamlite.c -rw-r--r-- 1 trr users 3124 2013-07-16 14:18 bamlite.h -rw-r--r-- 1 trr users 24816 2013-07-16 14:18 bamlite.o -rw-r--r-- 1 trr users 11508 2013-07-16 14:18 bntseq.c -rw-r--r-- 1 trr users 2557 2013-07-16 14:18 bntseq.h -rw-r--r-- 1 trr users 37440 2013-07-16 14:18 bntseq.o -rwxr-xr-x 1 trr users 998217 2013-07-16 14:18 bwa -rw-r--r-- 1 trr users 24225 2013-07-16 14:18 bwa.1 -rw-r--r-- 1 trr users 9416 2013-07-16 14:18 bwa.c -rw-r--r-- 1 trr users 1381 2013-07-16 14:18 bwa.h
....
I got the xmls and .py from https://bitbucket.org/galaxy/**galaxy-dist/src/da9d740fce31/** tools/sr_mapping<https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping>and i didnt change them at all and put them into ~/galaxy-dist/_tools/sr_**mapping_ (since they were missing in this folder)
bwa_color_wrapper.xml bwa_wrapper.py bwa_wrapper.xml
I added _bwa_index_color.loc and bwa_index.loc_ to ~galaxy-dist/_tool-data_ (they were missing as well, there/*were no*/
bwa_index_color.loc.sample or bwa_index.lox.sample files!!!)
I only have this single line in both bwa_index_color.loc and bwa_index.loc
trr@portalmoritz:~/galaxy-**dist/tool-data> cat bwa_index_color.loc #This is a sample file distributed with Galaxy that enables tools # #<unique_build_id> <dbkey> <display_name> <file_path> hg19 hg19 hg19 /genedata/human_genome_GRCh37/**hg19.fa
(Spaces are actually tabs!)
After all that, I neither have the "Map with BWA for Illumina <https://main.g2.bx.psu.edu/**tool_runner?tool_id=toolshed.** g2.bx.psu.edu/repos/devteam/**bwa_wrappers/bwa_wrapper/1.2.3<https://main.g2.bx.psu.edu/tool_runner?tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3> **>"
in my local Galaxy version, nor do I find the reference genome. If i missed on any required, please tell me, I will answer you as soon as possible. Sincerly Yours Moritz Juchler
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/**search/mailinglists/<http://galaxyproject.org/search/mailinglists/>
On Mon, Jul 22, 2013 at 10:40 AM, Moritz Juchler <juchler@stud.uni-heidelberg.de> wrote:
Hey,
now I am having a new problem: Convert SAM to BAM Tool execution generated the following error message:
[samopen] SAM header is present: 93 sequences. Parse error at line 106: sequence and quality are inconsistent /bin/sh: line 1: 27934 Aborted samtools view -bS "/home/trr/galaxy-dist/database/files/000/dataset_17.dat" > "/tmp/tmp-sam_to_bam_converter-ut5Tag/unsorted.bam"
That's bad.
The tool produced the following additional output:
[bam_header_read] EOF marker is absent. The input is probably truncated.
(should I make a new post out of this?)
Which version of samtools? There is a bug in the currently release where that warning is a false alarm: https://github.com/samtools/samtools/issues/18 Peter
1. Thanks, I'll try this out, after you reavealed me which version I should use :) 2. Originally I had this workflow to start with https://main.g2.bx.psu.edu/u/mj--/w/ngs , but I at the sam-to-bam conversion I get the "sequences are not currently available for specified build" error when using "locally cached" and I can't figure out how to use the reference file hg19.fa without actually uploading it to Galaxy, because I do not have enough space on the filesystem where the Galaxy distribution is placed ( /home). The genedata are all on /genedata. So my question here is: How to use the hg19.fa file placed on another filesystem then the galaxydist? 3. Thus I searched the web and found this workaround: https://main.g2.bx.psu.edu/u/mj--/w/sample-workflow-whole-exome-sequencing which runs fine ONLINE AT USE GALAXY but produces this error in my local instance Dataset generation errors *Dataset 18: Filter SAM on data 7* Tool execution generated the following error message: Traceback (most recent call last): File "/home/trr/galaxy-dist/tools/samtools/sam_bitwise_flag_filter.py", line 148, in <module> if __name__ == "__main__": main() File "/home/trr/galaxy-dist/tools/samtools/sam_bitwise_flag_filter.py", line 137, in main flags = int( fields[flag_col] ) IndexError: list index out of range So here is my 3rd question: How to solve this error? I didnt find anything online. Best Moritz On 22 July 2013 16:54, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Jul 22, 2013 at 10:40 AM, Moritz Juchler <juchler@stud.uni-heidelberg.de> wrote:
Hey,
now I am having a new problem: Convert SAM to BAM Tool execution generated the following error message:
[samopen] SAM header is present: 93 sequences. Parse error at line 106: sequence and quality are inconsistent /bin/sh: line 1: 27934 Aborted samtools view -bS "/home/trr/galaxy-dist/database/files/000/dataset_17.dat" > "/tmp/tmp-sam_to_bam_converter-ut5Tag/unsorted.bam"
That's bad.
The tool produced the following additional output:
[bam_header_read] EOF marker is absent. The input is probably truncated.
(should I make a new post out of this?)
Which version of samtools? There is a bug in the currently release where that warning is a false alarm: https://github.com/samtools/samtools/issues/18
Peter
Hi Moritz I am not a galaxy tool shed expert, but I recommend you start with reading these wiki pages: http://wiki.galaxyproject.org/Tool%20Shed http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy and then try installing BWA via the tool shed and ask (if any) specific questions (preferably with a new subject line) on the mailing list. Regards Hans-Rudolf On 07/22/2013 11:20 AM, Moritz Juchler wrote:
Hey,
with the "tool file" line I got it working :) Thanks a lot. Could you answer the question regarding Tool Shed: "If I would use Tool Shed, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle?"
Best Moritz
On 17 July 2013 09:51, Moritz Juchler <juchler@stud.uni-heidelberg.de <mailto:juchler@stud.uni-heidelberg.de>> wrote:
Hey,
no thats correct I did not use Tool Shed. If I would use it, could I skip all these manual steps? Is the installation of BWA with Tool Shed also this complicated or is it more simle?
How did you not follow my steps :) I wrote down everything clearly, at least thats what I was hoping.
And no I didnt write this line " <tool file="sr_mapping/bwa_wrapper.__xml" />" Thats the first time, I am seeing this tutorial :( I will try this out right now
Best Moritz
On 17 July 2013 09:42, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Moritz
I am struggling to follow what exactly you have done. As far as I can see, you did not use the toolshed (http://wiki.galaxyproject.__org/Tool%20Shed <http://wiki.galaxyproject.org/Tool%20Shed>) to install the BWA alinger tool, but did all manually?
If so, have you added the following line:
<tool file="sr_mapping/bwa_wrapper.__xml" />
to the "tool_conf.xml" file, and restarted Galaxy?
see also: http://wiki.galaxyproject.org/__Admin/Tools/Add%20Tool%__20Tutorial <http://wiki.galaxyproject.org/Admin/Tools/Add%20Tool%20Tutorial>
Hope this helps Hans-Rudolf
On 07/16/2013 08:35 PM, Moritz Juchler wrote:
Hello Ladies and Gentlemen,
I am Moritz Juchler from University Heidelberg. For my Bachelor thesis I have to choose a bioinformatic pipeline management tool to find SNP's in genomes from hcc patients. My decision was made in favor of galaxy. I have a 64-bit openSuse 11.3 server. I have installed Galaxy locally, since we have a) very large files (>30GB per patient) and b) the data is protection sensitive. I kept close to http://wiki.galaxyproject.org/__Admin/Get%20Galaxy <http://wiki.galaxyproject.org/Admin/Get%20Galaxy> Now I would like to run this bpipe pipeline: http://pastebin.com/sZd5vfdL And the first step is to align my genome to a _hg19 reference genome_
which I have locally under /genedata/human_genome_GRCh37/__.
trr@portalmoritz:~> ls -l /genedata/human_genome_GRCh37/ total 8486312 -rw-r--r-- 1 trr root 3199905909 2013-06-25 16:44 hg19.fa -rw-r--r-- 1 trr root 8591 2013-07-01 16:06 hg19.fa.amb -rw-r--r-- 1 trr root 4040 2013-07-01 16:06 hg19.fa.ann -rw-r--r-- 1 trr root 3137161344 2013-07-01 16:05 hg19.fa.bwt -rw-r--r-- 1 trr root 784290318 2013-07-01 16:06 hg19.fa.pac -rw-r--r-- 1 trr root 1568580688 2013-07-01 16:31 hg19.fa.sa <http://hg19.fa.sa> <http://hg19.fa.sa>
_bwa is installed and gives me:_
trr@portalmoritz:~> bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.5a-r405 Contact: Heng Li <lh3@sanger.ac.uk <mailto:lh3@sanger.ac.uk> <mailto:lh3@sanger.ac.uk <mailto:lh3@sanger.ac.uk>>>
Then I tried to follow this guide: http://wiki.galaxyproject.org/__Admin/NGS%20Local%20Setup <http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup> to get the reference files and http://wiki.galaxyproject.org/__Admin/Config/Tool%__20Dependencies <http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies>.
This is my _$PATH_
trr@portalmoritz:~> echo $PATH
/home/trr/bin:/usr/local/bin:/__usr/bin:/bin:/usr/bin/X11:/__usr/X11R6/bin:/usr/games:/__home/trr/bpipe-0.9.8/bin:/__home/trr/bwa-0.7.5a:/home/trr/__samtools-0.1.19
_In the universe_wsgi.ini I changed:_
tool_dependency_dir = /home/trr/galaxy-dist/tool___dependency_dir debug = False use_interactive = True library_import_dir = /genedata/ allow_library_path_paste = True admin_users = ...
This is my _tool_dependency_dir:_
trr@portalmoritz:~/galaxy-__dist/tool_dependency_dir/bwa> ls -l total 4 drwxr-xr-x 3 trr users 4096 2013-07-16 14:28 0.7.4 lrwxrwxrwx 1 trr users 6 2013-07-16 14:17 default -> 0.7.4/
This is the_version folder of bwa:_
trr@portalmoritz:~/galaxy-__dist/tool_dependency_dir/bwa/__0.7.4> ls -l total 8 drwxr-xr-x 2 trr users 4096 2013-07-16 14:18 bin -rw-r--r-- 1 trr users 47 2013-07-16 14:22 env.sh
This is the _content of env.sh:_
trr@portalmoritz:~/galaxy-__dist/tool_dependency_dir/bwa/__0.7.4> cat env.sh PATH="/home/trr/bwa-0.7.5a/:$__PATH" export PATH
And this is the _content of the bin folder:_
trr@portalmoritz:~/galaxy-__dist/tool_dependency_dir/bwa/__0.7.4/bin> ls -l total 3896 -rw-r--r-- 1 trr users 6098 2013-07-16 14:18 bamlite.c -rw-r--r-- 1 trr users 3124 2013-07-16 14:18 bamlite.h -rw-r--r-- 1 trr users 24816 2013-07-16 14:18 bamlite.o -rw-r--r-- 1 trr users 11508 2013-07-16 14:18 bntseq.c -rw-r--r-- 1 trr users 2557 2013-07-16 14:18 bntseq.h -rw-r--r-- 1 trr users 37440 2013-07-16 14:18 bntseq.o -rwxr-xr-x 1 trr users 998217 2013-07-16 14:18 bwa -rw-r--r-- 1 trr users 24225 2013-07-16 14:18 bwa.1 -rw-r--r-- 1 trr users 9416 2013-07-16 14:18 bwa.c -rw-r--r-- 1 trr users 1381 2013-07-16 14:18 bwa.h
....
I got the xmls and .py from https://bitbucket.org/galaxy/__galaxy-dist/src/da9d740fce31/__tools/sr_mappi... <https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping> and i didnt change them at all and put them into ~/galaxy-dist/_tools/sr___mapping_ (since they were missing in this folder)
bwa_color_wrapper.xml bwa_wrapper.py bwa_wrapper.xml
I added _bwa_index_color.loc and bwa_index.loc_ to ~galaxy-dist/_tool-data_ (they were missing as well, there/*were no*/
bwa_index_color.loc.sample or bwa_index.lox.sample files!!!)
I only have this single line in both bwa_index_color.loc and bwa_index.loc
trr@portalmoritz:~/galaxy-__dist/tool-data> cat bwa_index_color.loc #This is a sample file distributed with Galaxy that enables tools # #<unique_build_id> <dbkey> <display_name> <file_path> hg19 hg19 hg19 /genedata/human_genome_GRCh37/__hg19.fa
(Spaces are actually tabs!)
After all that, I neither have the "Map with BWA for Illumina <https://main.g2.bx.psu.edu/__tool_runner?tool_id=toolshed.__g2.bx.psu.edu/re... <https://main.g2.bx.psu.edu/tool_runner?tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3>__>"
in my local Galaxy version, nor do I find the reference genome. If i missed on any required, please tell me, I will answer you as soon as possible. Sincerly Yours Moritz Juchler
_____________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/__search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Hans-Rudolf Hotz -
Moritz Juchler -
Peter Cock