Hi All,
I was trying to setup the Galaxy tool dependencies, and I met the following problem when running sam-to-bam in samtools: Traceback (most recent call last): File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish dataset.set_meta( overwrite = False ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta return self.datatype.set_meta( self, **kwd ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$ ll total 8 drwxr-xr-x 4 ubuntu ubuntu 65 2012-04-16 19:09 ./ drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16 18:41 ../ drwxr-xr-x 2 ubuntu ubuntu 53 2012-04-16 19:08 0.1.12/ drwxr-xr-x 2 ubuntu ubuntu 53 2012-04-13 20:23 0.1.7/ lrwxrwxrwx 1 ubuntu ubuntu 38 2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$ more default/env.sh PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other programs, such as bwa and bowtie, and they worked perfectly. Only samtools could not work. I even used the full path of samtools in the wrapper file sam_to_bam.py, and it was not working neither.
I am wondering if any one met the same problem, or some one can give me some hints.
Thank you very much.
Cai
Hi Cai,
Please double check that your env.sh file contains the following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" export PATH
http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote:
Hi All,
I was trying to setup the Galaxy tool dependencies, and I met the following problem when running sam-to-bam in samtools: Traceback (most recent call last): File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish dataset.set_meta( overwrite = False ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta return self.datatype.set_meta( self, **kwd ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$ ll total 8 drwxr-xr-x 4 ubuntu ubuntu 65 2012-04-16 19:09 ./ drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16 18:41 ../ drwxr-xr-x 2 ubuntu ubuntu 53 2012-04-16 19:08 0.1.12/ drwxr-xr-x 2 ubuntu ubuntu 53 2012-04-13 20:23 0.1.7/ lrwxrwxrwx 1 ubuntu ubuntu 38 2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$ more default/env.sh PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other programs, such as bwa and bowtie, and they worked perfectly. Only samtools could not work. I even used the full path of samtools in the wrapper file sam_to_bam.py, and it was not working neither.
I am wondering if any one met the same problem, or some one can give me some hints.
Thank you very much.
Cai
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what you seggested, and the problem still existed. I am pasting more details below:
I submitted a job to convert sam to bam, and the job was running forever without outputing the result. I then checked the log, and it read: Traceback (most recent call last): File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish dataset.set_meta( overwrite = False ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta return self.datatype.set_meta( self, **kwd ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
It means that the samtools is not in the PATH. I tried to set the PATH in a couple of methods according the Galaxy documentation: 1. put the path in the env.sh in the tool directory and symbolink default to the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in /path/sge_request
none of them worked, and I got the above same problem.
Then I checked the job log file in the job_working_directory, and it read: Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH of samtools. To double check it, I added samtools index to Galaxy, and it worked well. I am very confused why SGE knows the tool path but cannot run the job correctly.
The system I am using is ubuntu on EC2 (the image id is: ami-999d49f0, which is the one used by StarCluster). I checked out the code from galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well using the same setting method(put env.sh in the tools directory to set the tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer Jackson jen@bx.psu.edu 写道：
发件人: Jennifer Jackson jen@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午2:12 Hi Cai,
Please double check that your env.sh file contains the following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" export PATH
http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote:
Hi All,
I was trying to setup the Galaxy tool dependencies, and
I met the following problem when running sam-to-bam in samtools:
Traceback (most recent call last): File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout, stderr )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish
dataset.set_meta( overwrite = False
)
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta
return self.datatype.set_meta(
self, **kwd )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta
raise Exception, "Error Setting BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$ ll total 8 drwxr-xr-x 4 ubuntu ubuntu 65
2012-04-16 19:09 ./
drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16 18:41 ../ drwxr-xr-x 2 ubuntu ubuntu 53
2012-04-16 19:08 0.1.12/
drwxr-xr-x 2 ubuntu ubuntu 53
2012-04-13 20:23 0.1.7/
lrwxrwxrwx 1 ubuntu ubuntu 38
2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$ more
default/env.sh
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other programs, such as bwa
and bowtie, and they worked perfectly. Only samtools could not work.
I even used the full path of samtools in the wrapper
file sam_to_bam.py, and it was not working neither.
I am wondering if any one met the same problem, or some
one can give me some hints.
Thank you very much.
Cai
Please keep all replies on the list by using "reply
all"
in your mail client. To manage your subscriptions
to this
and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy user's home directory) sge_request file
eg I have rlazarus@iaas1-int:~$ cat /var/lib/gridengine/default/common/sge_request -cwd -V
Without that switch, the job gets no path. With -V, it inherits the job submitter's full path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what you seggested, and the problem still existed. I am pasting more details below:
I submitted a job to convert sam to bam, and the job was running forever without outputing the result. I then checked the log, and it read: Traceback (most recent call last): File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish dataset.set_meta( overwrite = False ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta return self.datatype.set_meta( self, **kwd ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
It means that the samtools is not in the PATH. I tried to set the PATH in a couple of methods according the Galaxy documentation:
- put the path in the env.sh in the tool directory and symbolink default to the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18
- put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request
- put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in /path/sge_request
none of them worked, and I got the above same problem.
Then I checked the job log file in the job_working_directory, and it read: Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH of samtools. To double check it, I added samtools index to Galaxy, and it worked well. I am very confused why SGE knows the tool path but cannot run the job correctly.
The system I am using is ubuntu on EC2 (the image id is: ami-999d49f0, which is the one used by StarCluster). I checked out the code from galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well using the same setting method(put env.sh in the tools directory to set the tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer Jackson jen@bx.psu.edu 写道：
发件人: Jennifer Jackson jen@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午2:12 Hi Cai,
Please double check that your env.sh file contains the following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" export PATH
http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote:
Hi All,
I was trying to setup the Galaxy tool dependencies, and
I met the following problem when running sam-to-bam in samtools:
Traceback (most recent call last): File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job
drm_job_state.job_wrapper.finish(stdout, stderr )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish
dataset.set_meta( overwrite = False)
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta
return self.datatype.set_meta(self, **kwd )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta
raise Exception, "Error Setting BAMMetadata: %s" % stderr
Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$ ll total 8 drwxr-xr-x 4 ubuntu ubuntu 65
2012-04-16 19:09 ./
drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16 18:41 ../ drwxr-xr-x 2 ubuntu ubuntu 53
2012-04-16 19:08 0.1.12/
drwxr-xr-x 2 ubuntu ubuntu 53
2012-04-13 20:23 0.1.7/
lrwxrwxrwx 1 ubuntu ubuntu 38
2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$ more
default/env.sh
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other programs, such as bwa
and bowtie, and they worked perfectly. Only samtools could not work.
I even used the full path of samtools in the wrapper
file sam_to_bam.py, and it was not working neither.
I am wondering if any one met the same problem, or some
one can give me some hints.
Thank you very much.
Cai
Please keep all replies on the list by using "reply
all"
in your mail client. To manage your subscriptions
to this
and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other tools works fine.
samtools was compiled from source correctly. If I install samtools using "sudo apt-get install samtools", this problem is gone, but this is not a solution for since it does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午4:59 Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy user's home directory) sge_request file
eg I have rlazarus@iaas1-int:~$ cat /var/lib/gridengine/default/common/sge_request -cwd -V
Without that switch, the job gets no path. With -V, it inherits the job submitter's full path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what you seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to bam, and the job
was running forever without outputing the result. I then checked the log, and it read:
Traceback (most recent call last): File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job
drm_job_state.job_wrapper.finish( stdout,
stderr )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish
dataset.set_meta( overwrite = False ) File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta
return self.datatype.set_meta( self, **kwd
)
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta
raise Exception, "Error Setting BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
It means that the samtools is not in the PATH. I tried
to set the PATH in a couple of methods according the Galaxy documentation:
- put the path in the env.sh in the tool directory and
symbolink default to the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18
- put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
- put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the above same problem.
Then I checked the job log file in the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH of samtools. To
double check it, I added samtools index to Galaxy, and it worked well. I am very confused why SGE knows the tool path but cannot run the job correctly.
The system I am using is ubuntu on EC2 (the image id
is: ami-999d49f0, which is the one used by StarCluster). I checked out the code from galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well using the same setting method(put env.sh in the tools directory to set the tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer Jackson jen@bx.psu.edu
写道：
发件人: Jennifer Jackson jen@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午2:12 Hi Cai,
Please double check that your env.sh file contains
the
following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
export PATH
http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote:
Hi All,
I was trying to setup the Galaxy tool
dependencies, and
I met the following problem when running sam-to-bam
in
samtools:
Traceback (most recent call last): File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout, stderr )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta(
overwrite = False
)
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return
self.datatype.set_meta(
self, **kwd )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception, "Error
Setting BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata:
/bin/sh:
samtools: not found
I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll
total 8 drwxr-xr-x 4 ubuntu
ubuntu 65
2012-04-16 19:09 ./
drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16
18:41 ../
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-16 19:08 0.1.12/
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-13 20:23 0.1.7/
lrwxrwxrwx 1 ubuntu
ubuntu 38
2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more
default/env.sh
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other programs,
such as bwa
and bowtie, and they worked perfectly. Only
samtools could
not work.
I even used the full path of samtools in the
wrapper
file sam_to_bam.py, and it was not working
neither.
I am wondering if any one met the same
problem, or some
one can give me some hints.
Thank you very much.
Cai
Please keep all replies on the list by using
"reply
all"
in your mail client. To manage your
subscriptions
to this
and other Galaxy lists, please use the
interface at:
-- Jennifer Jackson http://galaxyproject.org
Please keep all replies on the list by using "reply
all"
in your mail client. To manage your subscriptions
to this
and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps in /data/apps/bin
To share the compiled libraries and binaries together, remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to the galaxy user's path so the nodes running jobs inherit those paths because of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other tools works fine.
samtools was compiled from source correctly. If I install samtools using "sudo apt-get install samtools", this problem is gone, but this is not a solution for since it does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午4:59 Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy user's home directory) sge_request file
eg I have rlazarus@iaas1-int:~$ cat /var/lib/gridengine/default/common/sge_request -cwd -V
Without that switch, the job gets no path. With -V, it inherits the job submitter's full path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what you seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to bam, and the job
was running forever without outputing the result. I then checked the log, and it read:
Traceback (most recent call last): File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job
drm_job_state.job_wrapper.finish( stdout,
stderr )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish
dataset.set_meta( overwrite = False ) File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta
return self.datatype.set_meta( self, **kwd
)
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta
raise Exception, "Error Setting BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
It means that the samtools is not in the PATH. I tried
to set the PATH in a couple of methods according the Galaxy documentation:
- put the path in the env.sh in the tool directory and
symbolink default to the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18
- put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
- put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the above same problem.
Then I checked the job log file in the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH of samtools. To
double check it, I added samtools index to Galaxy, and it worked well. I am very confused why SGE knows the tool path but cannot run the job correctly.
The system I am using is ubuntu on EC2 (the image id
is: ami-999d49f0, which is the one used by StarCluster). I checked out the code from galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well using the same setting method(put env.sh in the tools directory to set the tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer Jackson jen@bx.psu.edu
写道：
发件人: Jennifer Jackson jen@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午2:12 Hi Cai,
Please double check that your env.sh file contains
the
following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
export PATHhttp://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote:
Hi All,
I was trying to setup the Galaxy tool
dependencies, and
I met the following problem when running sam-to-bam
in
samtools:
Traceback (most recent call last): File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout, stderr )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta(overwrite = False
)
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
returnself.datatype.set_meta(
self, **kwd )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception, "ErrorSetting BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata:
/bin/sh:
samtools: not found
I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll
total 8 drwxr-xr-x 4 ubuntu
ubuntu 65
2012-04-16 19:09 ./
drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16
18:41 ../
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-16 19:08 0.1.12/
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-13 20:23 0.1.7/
lrwxrwxrwx 1 ubuntu
ubuntu 38
2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more
default/env.sh
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other programs,
such as bwa
and bowtie, and they worked perfectly. Only
samtools could
not work.
I even used the full path of samtools in the
wrapper
file sam_to_bam.py, and it was not working
neither.
I am wondering if any one met the same
problem, or some
one can give me some hints.
Thank you very much.
Cai
Please keep all replies on the list by using
"reply
all"
in your mail client. To manage your
subscriptions
to this
and other Galaxy lists, please use the
interface at:
-- Jennifer Jackson http://galaxyproject.org
Please keep all replies on the list by using "reply
all"
in your mail client. To manage your subscriptions
to this
and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Hi Ross,
Thank you for your reply. It still did not work. I think SGE knows the path of samtools. Here is a test case I added to Galaxy, and it creates the index for fasta file. I pasted the samtools faidx wrapper and xml files below: ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.pl #!/usr/bin/perl use strict; use warnings;
if (@ARGV < 1) { die "Usage: perl $0 <input fasta file>\n"; } `samtools faidx $ARGV[0]`; `cp ${ARGV[0]}.fai $ARGV[1]`; exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.xml <tool id="samtools_faidx" name="samtools faidx" version="1.0.3"> <requirements> <requirement type="package">samtools</requirement> </requirements> <description>creates fasta index</description> <command interpreter="perl"> samtools_faidx.pl $input1 $output1 </command> <inputs> <param name="input1" type="data" format="fasta" label="fasta file to be indexed" /> </inputs> <outputs> <data format="fai" name="output1" label="${tool.name} on ${on_string}: fasta" /> </outputs> </tool>
It works, which shows that SGE knows the path of samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午5:18 Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps in /data/apps/bin
To share the compiled libraries and binaries together, remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to the galaxy user's path so the nodes running jobs inherit those paths because of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other tools
works fine.
samtools was compiled from source correctly. If I
install samtools using "sudo apt-get install samtools", this problem is gone, but this is not a solution for since it does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午4:59 Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy user's home
directory)
sge_request file
eg I have rlazarus@iaas1-int:~$ cat /var/lib/gridengine/default/common/sge_request -cwd -V
Without that switch, the job gets no path. With -V, it inherits the job submitter's full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai
wrote:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what you
seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to bam, and
the job
was running forever without outputing the result. I
then
checked the log, and it read:
Traceback (most recent call last): File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout,
stderr )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta( overwrite =
False )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return self.datatype.set_meta(
self, **kwd
)
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception, "Error Setting
BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata:
/bin/sh:
samtools: not found
It means that the samtools is not in the PATH.
I tried
to set the PATH in a couple of methods according
the Galaxy
documentation:
- put the path in the env.sh in the tool
directory and
symbolink default to the tool directory, e.g.
default ->
=/mnt/galaxyTools/tools/samtools/0.1.18
- put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
- put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the above same
problem.
Then I checked the job log file in the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH of
samtools. To
double check it, I added samtools index to Galaxy,
and it
worked well. I am very confused why SGE knows the
tool path
but cannot run the job correctly.
The system I am using is ubuntu on EC2 (the
image id
is: ami-999d49f0, which is the one used by
StarCluster). I
checked out the code from galaxy-dist on bitbucket.
Other
tools such as bwa and bowtie worked well using the
same
setting method(put env.sh in the tools directory to
set the
tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer Jackson
写道：
发件人: Jennifer Jackson jen@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: galaxy-dev@lists.bx.psu.edu 日期:
2012年4月18日,周三,上午2:12
Hi Cai,
Please double check that your env.sh file
contains
the
following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
export PATH
http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote:
Hi All,
I was trying to setup the Galaxy
tool
dependencies, and
I met the following problem when running
sam-to-bam
in
samtools:
Traceback (most recent call last): File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout, stderr )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta(
overwrite = False
)
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return
self.datatype.set_meta(
self, **kwd )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception,
"Error
Setting BAM
Metadata: %s" % stderr
Exception: Error Setting BAM
Metadata:
/bin/sh:
samtools: not found
I already set the path of samtools Below I pasted how I set the path:
ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll
total 8 drwxr-xr-x 4 ubuntu
ubuntu 65
2012-04-16 19:09 ./
drwxr-xr-x 38 ubuntu ubuntu 4096
2012-04-16
18:41 ../
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-16 19:08 0.1.12/
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-13 20:23 0.1.7/
lrwxrwxrwx 1 ubuntu
ubuntu 38
2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more
default/env.sh
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set other
programs,
such as bwa
and bowtie, and they worked perfectly.
Only
samtools could
not work.
I even used the full path of samtools
in the
wrapper
file sam_to_bam.py, and it was not
working
neither.
I am wondering if any one met the
same
problem, or some
one can give me some hints.
Thank you very much.
Cai
Please keep all replies on the list
by using
"reply
all"
in your mail client. To manage
your
subscriptions
to this
and other Galaxy lists, please use
the
interface at:
-- Jennifer Jackson http://galaxyproject.org
Please keep all replies on the list by using
"reply
all"
in your mail client. To manage your
subscriptions
to this
and other Galaxy lists, please use the
interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the --prefix configure option and tested that all the samtools based Galaxy wrappers work if you use the local job runner? That means that both the appropriate lib and bin directories are on the Galaxy user's path ready to be inherited by SGE with the -V SGE option.
If the local job runner works, that seems to isolate the problem to SGE configuration and operation - the steps I've suggested seem to work for us so I've run out of ideas on why it's not working at your shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. It still did not work. I think SGE knows the path of samtools. Here is a test case I added to Galaxy, and it creates the index for fasta file. I pasted the samtools faidx wrapper and xml files below: ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.pl #!/usr/bin/perl use strict; use warnings;
if (@ARGV < 1) { die "Usage: perl $0 <input fasta file>\n"; } `samtools faidx $ARGV[0]`; `cp ${ARGV[0]}.fai $ARGV[1]`; exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.xml
<tool id="samtools_faidx" name="samtools faidx" version="1.0.3"> <requirements> <requirement type="package">samtools</requirement> </requirements> <description>creates fasta index</description> <command interpreter="perl"> samtools_faidx.pl $input1 $output1 </command> <inputs> <param name="input1" type="data" format="fasta" label="fasta file to be indexed" /> </inputs> <outputs> <data format="fai" name="output1" label="${tool.name} on ${on_string}: fasta" /> </outputs> </tool>
It works, which shows that SGE knows the path of samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,上午5:18 Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps in /data/apps/bin
To share the compiled libraries and binaries together, remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to the galaxy user's path so the nodes running jobs inherit those paths because of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other tools
works fine.
samtools was compiled from source correctly. If I
install samtools using "sudo apt-get install samtools", this problem is gone, but this is not a solution for since it does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午4:59 Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy user's home
directory)
sge_request file
eg I have rlazarus@iaas1-int:~$ cat /var/lib/gridengine/default/common/sge_request -cwd -V
Without that switch, the job gets no path. With -V, it inherits the job submitter's full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai
wrote:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what you
seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to bam, and
the job
was running forever without outputing the result. I
then
checked the log, and it read:
Traceback (most recent call last): File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout,
stderr )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta( overwrite =
False )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return self.datatype.set_meta(
self, **kwd
)
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception, "Error Setting
BAM
Metadata: %s" % stderr
Exception: Error Setting BAM Metadata:
/bin/sh:
samtools: not found
It means that the samtools is not in the PATH.
I tried
to set the PATH in a couple of methods according
the Galaxy
documentation:
- put the path in the env.sh in the tool
directory and
symbolink default to the tool directory, e.g.
default ->
=/mnt/galaxyTools/tools/samtools/0.1.18
- put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
- put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the above same
problem.
Then I checked the job log file in the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH of
samtools. To
double check it, I added samtools index to Galaxy,
and it
worked well. I am very confused why SGE knows the
tool path
but cannot run the job correctly.
The system I am using is ubuntu on EC2 (the
image id
is: ami-999d49f0, which is the one used by
StarCluster). I
checked out the code from galaxy-dist on bitbucket.
Other
tools such as bwa and bowtie worked well using the
same
setting method(put env.sh in the tools directory to
set the
tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer Jackson
写道：
发件人: Jennifer Jackson jen@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: galaxy-dev@lists.bx.psu.edu 日期:
2012年4月18日,周三,上午2:12
Hi Cai,
Please double check that your env.sh file
contains
the
following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
export PATHhttp://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai wrote: > Hi All, > > I was trying to setup the Galaxy
tool
dependencies, and
I met the following problem when running
sam-to-bam
in
samtools: > Traceback (most recent call last): > File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job >
drm_job_state.job_wrapper.finish(
stdout, stderr ) > File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish >
dataset.set_meta(
overwrite = False
) > File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta > return
self.datatype.set_meta(
self, **kwd ) > File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta > raise Exception,
"Error
Setting BAM
Metadata: %s" % stderr > Exception: Error Setting BAM
Metadata:
/bin/sh:
samtools: not found > > > I already set the path of samtools > Below I pasted how I set the path: >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll
> total 8 > drwxr-xr-x 4 ubuntu
ubuntu 65
2012-04-16 19:09 ./ > drwxr-xr-x 38 ubuntu ubuntu 4096
2012-04-16
18:41 ../
> drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-16 19:08 0.1.12/ > drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-13 20:23 0.1.7/ > lrwxrwxrwx 1 ubuntu
ubuntu 38
2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/ > >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more
default/env.sh >
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
> > I used the same way to set other
programs,
such as bwa
and bowtie, and they worked perfectly.
Only
samtools could
not work. > I even used the full path of samtools
in the
wrapper
file sam_to_bam.py, and it was not
working
neither.
> > I am wondering if any one met the
same
problem, or some
one can give me some hints. > > Thank you very much. > > Cai > >
> Please keep all replies on the list
by using
"reply
all" > in your mail client. To manage
your
subscriptions
to this > and other Galaxy lists, please use
the
interface at:
-- Jennifer Jackson http://galaxyproject.org
Please keep all replies on the list by using
"reply
all"
in your mail client. To manage your
subscriptions
to this
and other Galaxy lists, please use the
interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with Galaxy AMI image, and I copied it to the same path on my EC2. How to compile samtools using --prefix? I just typed 'make', and there is not a configuration file. In the cloudman version of Galaxy (it is working well for me), only two executables (samtools and maq2sam-long) in the directory /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is the only compiled executable 'samtools' fully functional for the cluster(it is on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,下午11:07 Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the --prefix configure option and tested that all the samtools based Galaxy wrappers work if you use the local job runner? That means that both the appropriate lib and bin directories are on the Galaxy user's path ready to be inherited by SGE with the -V SGE option.
If the local job runner works, that seems to isolate the problem to SGE configuration and operation - the steps I've suggested seem to work for us so I've run out of ideas on why it's not working at your shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. It still did not work. I think SGE knows the path of samtools. Here is a test
case I added to Galaxy, and it creates the index for fasta file. I pasted the samtools faidx wrapper and xml files below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.pl
#!/usr/bin/perl use strict; use warnings;
if (@ARGV < 1) { die "Usage:
perl $0 <input fasta file>\n";
} `samtools faidx $ARGV[0]`; `cp ${ARGV[0]}.fai $ARGV[1]`; exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.xml
<tool id="samtools_faidx" name="samtools faidx"
version="1.0.3">
<requirements> <requirement
type="package">samtools</requirement>
</requirements> <description>creates fasta
index</description>
<command interpreter="perl"> samtools_faidx.pl $input1 $output1 </command> <inputs> <param name="input1" type="data"
format="fasta" label="fasta file to be indexed" />
</inputs> <outputs> <data format="fai" name="output1"
label="${tool.name} on ${on_string}: fasta" />
</outputs>
</tool>
It works, which shows that SGE knows the path of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18 Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps in /data/apps/bin
To share the compiled libraries and binaries
together,
remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to
the
galaxy user's path so the nodes running jobs inherit those paths
because
of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other
tools
works fine.
samtools was compiled from source correctly.
If I
install samtools using "sudo apt-get install
samtools", this
problem is gone, but this is not a solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午4:59
Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy
user's home
directory)
sge_request file
eg I have rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd -V
Without that switch, the job gets no
path.
With -V, it inherits the job submitter's
full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu
cai
wrote:
Hi Jen,
Thank you for your quick response. I changed the env.sh file to what
you
seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to
bam, and
the job
was running forever without outputing the
result. I
then
checked the log, and it read:
Traceback (most recent call last): File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout,
stderr )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta(
overwrite =
False )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return
self.datatype.set_meta(
self, **kwd
)
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception, "Error
Setting
BAM
Metadata: %s" % stderr
Exception: Error Setting BAM
Metadata:
/bin/sh:
samtools: not found
It means that the samtools is not in
the PATH.
I tried
to set the PATH in a couple of methods
according
the Galaxy
documentation:
- put the path in the env.sh in the
tool
directory and
symbolink default to the tool directory,
e.g.
default ->
=/mnt/galaxyTools/tools/samtools/0.1.18
- put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
- put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the
above same
problem.
Then I checked the job log file in
the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM
which shows that sge knows the PATH
of
samtools. To
double check it, I added samtools index to
Galaxy,
and it
worked well. I am very confused why SGE
knows the
tool path
but cannot run the job correctly.
The system I am using is ubuntu on
EC2 (the
image id
is: ami-999d49f0, which is the one used
by
StarCluster). I
checked out the code from galaxy-dist on
bitbucket.
Other
tools such as bwa and bowtie worked well
using the
same
setting method(put env.sh in the tools
directory to
set the
tool path)
Thanks you very much,
Cai --- 12年4月18日，周三, Jennifer
Jackson
写道：
> 发件人: Jennifer Jackson
> 主题: Re: [galaxy-dev]
Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not
found
> 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn > 抄送: galaxy-dev@lists.bx.psu.edu > 日期:
2012年4月18日,周三,上午2:12
> Hi Cai, > > Please double check that your
env.sh file
contains
the
> following: > > > >
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
> export PATH > > > http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies > > > Hopefully this helps, > > Jen > Galaxy team > > On 4/16/12 12:43 PM, zhengqiu cai
wrote:
> > Hi All, > > > > I was trying to setup the
Galaxy
tool
dependencies, and
> I met the following problem when
running
sam-to-bam
in
> samtools: > > Traceback (most recent call
last):
> > File >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
> line 336, in finish_job > >
drm_job_state.job_wrapper.finish(
> stdout, stderr ) > > File >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
> line 637, in finish > >
dataset.set_meta(
overwrite = False
> ) > > File >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
> line 875, in set_meta > > return
self.datatype.set_meta(
> self, **kwd ) > > File >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
> line 179, in set_meta > > raise
Exception,
"Error
Setting BAM
> Metadata: %s" % stderr > > Exception: Error Setting
BAM
Metadata:
/bin/sh:
> samtools: not found > > > > > > I already set the path of
samtools
> > Below I pasted how I set the
path:
> >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll
> > total 8 > > drwxr-xr-x 4 ubuntu
ubuntu 65
> 2012-04-16 19:09 ./ > > drwxr-xr-x 38 ubuntu ubuntu
4096
2012-04-16
18:41 ../
> > drwxr-xr-x 2 ubuntu
ubuntu 53
> 2012-04-16 19:08 0.1.12/ > > drwxr-xr-x 2 ubuntu
ubuntu 53
> 2012-04-13 20:23 0.1.7/ > > lrwxrwxrwx 1 ubuntu
ubuntu 38
> 2012-04-16 19:09 default -> >
/mnt/galaxyTools/tools/samtools/0.1.12/
> > > >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more
> default/env.sh > >
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
> > > > I used the same way to set
other
programs,
such as bwa
> and bowtie, and they worked
perfectly.
Only
samtools could
> not work. > > I even used the full path of
samtools
in the
wrapper
> file sam_to_bam.py, and it was
not
working
neither.
> > > > I am wondering if any one
met the
same
problem, or some
> one can give me some hints. > > > > Thank you very much. > > > > Cai > > > > >
> > Please keep all replies on
the list
by using
"reply
> all" > > in your mail client.
To manage
your
subscriptions
> to this > > and other Galaxy lists,
please use
the
interface at:
> > > > http://lists.bx.psu.edu/ > > -- > Jennifer Jackson > http://galaxyproject.org >
Please keep all replies on the list
by using
"reply
all"
in your mail client. To manage
your
subscriptions
to this
and other Galaxy lists, please use
the
interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical
School;
Head, Medical Bioinformatics, BakerIDI;
Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Hi, Cai. Apologies but I had not noticed any reference to the cloud distribution before - that changes everything - I was talking about a normal cluster/SGE install - you should definitely not need to do anything at all to the AMI - maybe someone with cloudman experience can be more helpful than I can.
On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with Galaxy AMI image, and I copied it to the same path on my EC2. How to compile samtools using --prefix? I just typed 'make', and there is not a configuration file. In the cloudman version of Galaxy (it is working well for me), only two executables (samtools and maq2sam-long) in the directory /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is the only compiled executable 'samtools' fully functional for the cluster(it is on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,下午11:07 Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the --prefix configure option and tested that all the samtools based Galaxy wrappers work if you use the local job runner? That means that both the appropriate lib and bin directories are on the Galaxy user's path ready to be inherited by SGE with the -V SGE option.
If the local job runner works, that seems to isolate the problem to SGE configuration and operation - the steps I've suggested seem to work for us so I've run out of ideas on why it's not working at your shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. It still did not work. I think SGE knows the path of samtools. Here is a test
case I added to Galaxy, and it creates the index for fasta file. I pasted the samtools faidx wrapper and xml files below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.pl
#!/usr/bin/perl use strict; use warnings;
if (@ARGV < 1) { die "Usage:
perl $0 <input fasta file>\n";
} `samtools faidx $ARGV[0]`; `cp ${ARGV[0]}.fai $ARGV[1]`; exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.xml
<tool id="samtools_faidx" name="samtools faidx"
version="1.0.3">
<requirements> <requirement
type="package">samtools</requirement>
</requirements> <description>creates fasta
index</description>
<command interpreter="perl"> samtools_faidx.pl $input1 $output1 </command> <inputs> <param name="input1" type="data"
format="fasta" label="fasta file to be indexed" />
</inputs> <outputs> <data format="fai" name="output1"
label="${tool.name} on ${on_string}: fasta" />
</outputs> </tool>
It works, which shows that SGE knows the path of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18 Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps in /data/apps/bin
To share the compiled libraries and binaries
together,
remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to
the
galaxy user's path so the nodes running jobs inherit those paths
because
of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other
tools
works fine.
samtools was compiled from source correctly.
If I
install samtools using "sudo apt-get install
samtools", this
problem is gone, but this is not a solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午4:59
Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy
user's home
directory)
sge_request file
eg I have rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd -V
Without that switch, the job gets no
path.
With -V, it inherits the job submitter's
full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu
cai
wrote: > Hi Jen, > > Thank you for your quick response. > I changed the env.sh file to what
you
seggested, and
the problem still existed. > I am pasting more details below: > > I submitted a job to convert sam to
bam, and
the job
was running forever without outputing the
result. I
then
checked the log, and it read: > Traceback (most recent call last): > File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job >
drm_job_state.job_wrapper.finish(
stdout,
stderr ) > File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish > dataset.set_meta(
overwrite =
False )
> File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta > return
self.datatype.set_meta(
self, **kwd
) > File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta > raise Exception, "Error
Setting
BAM
Metadata: %s" % stderr > Exception: Error Setting BAM
Metadata:
/bin/sh:
samtools: not found > > It means that the samtools is not in
the PATH.
I tried
to set the PATH in a couple of methods
according
the Galaxy
documentation: > 1. put the path in the env.sh in the
tool
directory and
symbolink default to the tool directory,
e.g.
default ->
=/mnt/galaxyTools/tools/samtools/0.1.18 > 2. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request > 3. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request > > none of them worked, and I got the
above same
problem.
> > Then I checked the job log file in
the
job_working_directory, and it read: > Samtools Version: 0.1.18 (r982:295) > SAM file converted to BAM > > which shows that sge knows the PATH
of
samtools. To
double check it, I added samtools index to
Galaxy,
and it
worked well. I am very confused why SGE
knows the
tool path
but cannot run the job correctly. > > The system I am using is ubuntu on
EC2 (the
image id
is: ami-999d49f0, which is the one used
by
StarCluster). I
checked out the code from galaxy-dist on
bitbucket.
Other
tools such as bwa and bowtie worked well
using the
same
setting method(put env.sh in the tools
directory to
set the
tool path) > > > Thanks you very much, > > Cai > --- 12年4月18日，周三, Jennifer
Jackson
写道： > >> 发件人: Jennifer Jackson
>> 主题: Re: [galaxy-dev]
Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not
found
>> 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn >> 抄送: galaxy-dev@lists.bx.psu.edu >> 日期:
2012年4月18日,周三,上午2:12
>> Hi Cai, >> >> Please double check that your
env.sh file
contains
the >> following: >> >> >> >>
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
>> export PATH >> >> >> http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies >> >> >> Hopefully this helps, >> >> Jen >> Galaxy team >> >> On 4/16/12 12:43 PM, zhengqiu cai
wrote:
>> > Hi All, >> > >> > I was trying to setup the
Galaxy
tool
dependencies, and >> I met the following problem when
running
sam-to-bam
in >> samtools: >> > Traceback (most recent call
last):
>> > File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
>> line 336, in finish_job >> > drm_job_state.job_wrapper.finish( >> stdout, stderr ) >> > File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
>> line 637, in finish >> >
dataset.set_meta(
overwrite = False >> ) >> > File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
>> line 875, in set_meta >> > return self.datatype.set_meta( >> self, **kwd ) >> > File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
>> line 179, in set_meta >> > raise
Exception,
"Error
Setting BAM >> Metadata: %s" % stderr >> > Exception: Error Setting
BAM
Metadata:
/bin/sh: >> samtools: not found >> > >> > >> > I already set the path of
samtools
>> > Below I pasted how I set the
path:
>> >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll >> > total 8 >> > drwxr-xr-x 4 ubuntu ubuntu 65 >> 2012-04-16 19:09 ./ >> > drwxr-xr-x 38 ubuntu ubuntu
4096
2012-04-16
18:41 ../ >> > drwxr-xr-x 2 ubuntu ubuntu 53 >> 2012-04-16 19:08 0.1.12/ >> > drwxr-xr-x 2 ubuntu ubuntu 53 >> 2012-04-13 20:23 0.1.7/ >> > lrwxrwxrwx 1 ubuntu ubuntu 38 >> 2012-04-16 19:09 default -> >>
/mnt/galaxyTools/tools/samtools/0.1.12/
>> > >> >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more >> default/env.sh >> >
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
>> > >> > I used the same way to set
other
programs,
such as bwa >> and bowtie, and they worked
perfectly.
Only
samtools could >> not work. >> > I even used the full path of
samtools
in the
wrapper >> file sam_to_bam.py, and it was
not
working
neither. >> > >> > I am wondering if any one
met the
same
problem, or some >> one can give me some hints. >> > >> > Thank you very much. >> > >> > Cai >> > >> > >>
>> > Please keep all replies on
the list
by using
"reply >> all" >> > in your mail client.
To manage
your
subscriptions >> to this >> > and other Galaxy lists,
please use
the
interface at: >> > >> > http://lists.bx.psu.edu/ >> >> -- >> Jennifer Jackson >> http://galaxyproject.org >> > >
> Please keep all replies on the list
by using
"reply
all" > in your mail client. To manage
your
subscriptions
to this > and other Galaxy lists, please use
the
interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical
School;
Head, Medical Bioinformatics, BakerIDI;
Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Hi Ross,
I have a cloudman version of Galaxy working well. What I am doing now is trying to install Galaxy on starcluster, which is a SGE cluster. I should followed the install method for the normal cluster/SGE install, not for the cloud. Thank you for your quick reply. Have a nice day.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,下午11:46 Hi, Cai. Apologies but I had not noticed any reference to the cloud distribution before - that changes everything - I was talking about a normal cluster/SGE install - you should definitely not need to do anything at all to the AMI - maybe someone with cloudman experience can be more helpful than I can.
On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
I did not compile samtools using the --prefix. The
samtools I used comes with Galaxy AMI image, and I copied it to the same path on my EC2. How to compile samtools using --prefix? I just typed 'make', and there is not a configuration file. In the cloudman version of Galaxy (it is working well for me), only two executables (samtools and maq2sam-long) in the directory /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is the only compiled executable 'samtools' fully functional for the cluster(it is on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,下午11:07 Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the
--prefix
configure option and tested that all the samtools based Galaxy
wrappers work
if you use the local job runner? That means that both the
appropriate
lib and bin directories are on the Galaxy user's path ready to
be
inherited by SGE with the -V SGE option.
If the local job runner works, that seems to
isolate the
problem to SGE configuration and operation - the steps I've
suggested
seem to work for us so I've run out of ideas on why it's
not working
at your shop and hope somebody with more experience can
help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai
wrote:
Hi Ross,
Thank you for your reply. It still did not work. I think SGE knows the path of samtools. Here
is a test
case I added to Galaxy, and it creates the index
for fasta
file. I pasted the samtools faidx wrapper and xml
files
below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.pl
#!/usr/bin/perl use strict; use warnings;
if (@ARGV < 1) { die "Usage:
perl $0 <input fasta file>\n";
} `samtools faidx $ARGV[0]`; `cp ${ARGV[0]}.fai $ARGV[1]`; exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.xml
<tool id="samtools_faidx" name="samtools
faidx"
version="1.0.3">
<requirements> <requirement
type="package">samtools</requirement>
</requirements> <description>creates fasta
index</description>
<command interpreter="perl"> samtools_faidx.pl $input1
$output1
</command> <inputs> <param name="input1"
type="data"
format="fasta" label="fasta file to be indexed"
/>
</inputs> <outputs> <data format="fai"
name="output1"
label="${tool.name} on ${on_string}: fasta" />
</outputs>
</tool>
It works, which shows that SGE knows the path
of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午5:18
Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps
in
/data/apps/bin
To share the compiled libraries and
binaries
together,
remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib
path to
the
galaxy user's path so the nodes running jobs inherit
those paths
because
of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu
cai
wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not
working.
Only samtools has this problem, and
the other
tools
works fine.
samtools was compiled from source
correctly.
If I
install samtools using "sudo apt-get
install
samtools", this
problem is gone, but this is not a
solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross
写道：
> 发件人: Ross ross.lazarus@gmail.com > 主题: Re: [galaxy-dev]
Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not
found
> 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn > 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
> 日期:
2012年4月18日,周三,上午4:59
> Cai - if you are using SGE you > probably want to pass the Galaxy
user's
> path to the job using the > -V > switch in the system wide (or
Galaxy
user's home
directory)
> sge_request file > > eg I have > rlazarus@iaas1-int:~$ cat >
/var/lib/gridengine/default/common/sge_request
> -cwd > -V > > Without that switch, the job gets
no
path.
> With -V, it inherits the job
submitter's
full
path.
> > On Tue, Apr 17, 2012 at 4:26 PM,
zhengqiu
cai
> wrote: > > Hi Jen, > > > > Thank you for your quick
response.
> > I changed the env.sh file to
what
you
seggested, and
> the problem still existed. > > I am pasting more details
below:
> > > > I submitted a job to convert
sam to
bam, and
the job
> was running forever without
outputing the
result. I
then
> checked the log, and it read: > > Traceback (most recent call
last):
> > File >
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
> line 336, in finish_job > >
drm_job_state.job_wrapper.finish(
stdout,
> stderr ) > > File >
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
> line 637, in finish > >
dataset.set_meta(
overwrite =
False )
> > File >
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
> line 875, in set_meta > > return
self.datatype.set_meta(
self, **kwd
> ) > > File >
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
> line 179, in set_meta > > raise
Exception, "Error
Setting
BAM
> Metadata: %s" % stderr > > Exception: Error Setting
BAM
Metadata:
/bin/sh:
> samtools: not found > > > > It means that the samtools
is not in
the PATH.
I tried
> to set the PATH in a couple of
methods
according
the Galaxy
> documentation: > > 1. put the path in the
env.sh in the
tool
directory and
> symbolink default to the tool
directory,
e.g.
default ->
>
=/mnt/galaxyTools/tools/samtools/0.1.18
> > 2. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
> in ~/.sge_request > > 3. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
> in /path/sge_request > > > > none of them worked, and I
got the
above same
problem.
> > > > Then I checked the job log
file in
the
> job_working_directory, and it
read:
> > Samtools Version: 0.1.18
(r982:295)
> > SAM file converted to BAM > > > > which shows that sge knows
the PATH
of
samtools. To
> double check it, I added samtools
index to
Galaxy,
and it
> worked well. I am very confused
why SGE
knows the
tool path
> but cannot run the job
correctly.
> > > > The system I am using is
ubuntu on
EC2 (the
image id
> is: ami-999d49f0, which is the
one used
by
StarCluster). I
> checked out the code from
galaxy-dist on
bitbucket.
Other
> tools such as bwa and bowtie
worked well
using the
same
> setting method(put env.sh in the
tools
directory to
set the
> tool path) > > > > > > Thanks you very much, > > > > Cai > > --- 12年4月18日，周三,
Jennifer
Jackson
> 写道： > > > >> 发件人: Jennifer
Jackson
> >> 主题: Re:
[galaxy-dev]
Exception: Error
Setting
> BAM Metadata: /bin/sh: samtools:
not
found
> >> 收件人: "zhengqiu
> >> 抄送: galaxy-dev@lists.bx.psu.edu > >> 日期:
2012年4月18日,周三,上午2:12
> >> Hi Cai, > >> > >> Please double check that
your
env.sh file
contains
> the > >> following: > >> > >> > >> > >> >
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
> >>
export PATH
> >> > >> > >> http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies > >> > >> > >> Hopefully this helps, > >> > >> Jen > >> Galaxy team > >> > >> On 4/16/12 12:43 PM,
zhengqiu cai
wrote:
> >> > Hi All, > >> > > >> > I was trying to
setup the
Galaxy
tool
> dependencies, and > >> I met the following
problem when
running
sam-to-bam
> in > >> samtools: > >> > Traceback (most
recent call
last):
> >> > File > >> >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
> >> line 336, in finish_job > >> > >
drm_job_state.job_wrapper.finish(
> >> stdout, stderr ) > >> > File > >> >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
> >> line 637, in finish > >> >
dataset.set_meta(
> overwrite = False > >> ) > >> > File > >> >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
> >> line 875, in set_meta > >> >
return
> self.datatype.set_meta( > >> self, **kwd ) > >> > File > >> >
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
> >> line 179, in set_meta > >> >
raise
Exception,
"Error
> Setting BAM > >> Metadata: %s" % stderr > >> > Exception: Error
Setting
BAM
Metadata:
> /bin/sh: > >> samtools: not found > >> > > >> > > >> > I already set the
path of
samtools
> >> > Below I pasted how
I set the
path:
> >> >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
> ll > >> > total 8 > >> > drwxr-xr-x 4
ubuntu
> ubuntu 65 > >> 2012-04-16 19:09 ./ > >> > drwxr-xr-x 38
ubuntu ubuntu
4096
2012-04-16
> 18:41 ../ > >> > drwxr-xr-x 2
ubuntu
> ubuntu 53 > >> 2012-04-16 19:08
0.1.12/
> >> > drwxr-xr-x 2
ubuntu
> ubuntu 53 > >> 2012-04-13 20:23 0.1.7/ > >> > lrwxrwxrwx 1
ubuntu
> ubuntu 38 > >> 2012-04-16 19:09 default
->
> >>
/mnt/galaxyTools/tools/samtools/0.1.12/
> >> > > >> >
ubuntu@master:/mnt/galaxyTools/tools/samtools$
> more > >> default/env.sh > >> > >
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
> >> > > >> > I used the same way
to set
other
programs,
> such as bwa > >> and bowtie, and they
worked
perfectly.
Only
> samtools could > >> not work. > >> > I even used the
full path of
samtools
in the
> wrapper > >> file sam_to_bam.py, and
it was
not
working
> neither. > >> > > >> > I am wondering if
any one
met the
same
> problem, or some > >> one can give me some
hints.
> >> > > >> > Thank you very
much.
> >> > > >> > Cai > >> > > >> > > >> >
> >> > Please keep all
replies on
the list
by using
> "reply > >> all" > >> > in your mail
client.
To manage
your
> subscriptions > >> to this > >> > and other Galaxy
lists,
please use
the
> interface at: > >> > > >> > http://lists.bx.psu.edu/ > >> > >> -- > >> Jennifer Jackson > >> http://galaxyproject.org > >> > > > > >
> > Please keep all replies on
the list
by using
"reply
> all" > > in your mail client.
To manage
your
subscriptions
> to this > > and other Galaxy lists,
please use
the
interface at:
> > > > http://lists.bx.psu.edu/ > > > > -- > Ross Lazarus MBBS MPH; > Associate Professor, Harvard
Medical
School;
> Head, Medical Bioinformatics,
BakerIDI;
Tel: +61
385321444;
>
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical
School;
Head, Medical Bioinformatics, BakerIDI;
Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Hi Cai,
Can you check that you have 'set_metadata_externally = True' in your universe_wsgi.ini file? This allows the set_meta process to occur on the nodes as part of the job. Otherwise, you will need to make sure that samtools is installed for the headnode.
Thanks for using Galaxy,
Dan
On Apr 18, 2012, at 11:39 AM, zhengqiu cai wrote:
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with Galaxy AMI image, and I copied it to the same path on my EC2. How to compile samtools using --prefix? I just typed 'make', and there is not a configuration file. In the cloudman version of Galaxy (it is working well for me), only two executables (samtools and maq2sam-long) in the directory /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is the only compiled executable 'samtools' fully functional for the cluster(it is on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,下午11:07 Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the --prefix configure option and tested that all the samtools based Galaxy wrappers work if you use the local job runner? That means that both the appropriate lib and bin directories are on the Galaxy user's path ready to be inherited by SGE with the -V SGE option.
If the local job runner works, that seems to isolate the problem to SGE configuration and operation - the steps I've suggested seem to work for us so I've run out of ideas on why it's not working at your shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2005@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply. It still did not work. I think SGE knows the path of samtools. Here is a test
case I added to Galaxy, and it creates the index for fasta file. I pasted the samtools faidx wrapper and xml files below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.pl
#!/usr/bin/perl use strict; use warnings;
if (@ARGV < 1) { die "Usage:
perl $0 <input fasta file>\n";
} `samtools faidx $ARGV[0]`; `cp ${ARGV[0]}.fai $ARGV[1]`; exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more samtools_faidx.xml
<tool id="samtools_faidx" name="samtools faidx"
version="1.0.3">
<requirements> <requirement
type="package">samtools</requirement>
</requirements> <description>creates fasta
index</description>
<command interpreter="perl"> samtools_faidx.pl $input1 $output1 </command> <inputs> <param name="input1" type="data"
format="fasta" label="fasta file to be indexed" />
</inputs> <outputs> <data format="fai" name="output1"
label="${tool.name} on ${on_string}: fasta" />
</outputs> </tool>
It works, which shows that SGE knows the path of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18 Ah. Then you probably have a library path problem. Here's how we do it FWIW:
Assume your want your shared cluster apps in /data/apps/bin
To share the compiled libraries and binaries
together,
remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to
the
galaxy user's path so the nodes running jobs inherit those paths
because
of the -V switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
wrote:
Hi Ross,
Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other
tools
works fine.
samtools was compiled from source correctly.
If I
install samtools using "sudo apt-get install
samtools", this
problem is gone, but this is not a solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com 主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午4:59
Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy
user's home
directory)
sge_request file
eg I have rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd -V
Without that switch, the job gets no
path.
With -V, it inherits the job submitter's
full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu
cai
wrote: > Hi Jen, > > Thank you for your quick response. > I changed the env.sh file to what
you
seggested, and
the problem still existed. > I am pasting more details below: > > I submitted a job to convert sam to
bam, and
the job
was running forever without outputing the
result. I
then
checked the log, and it read: > Traceback (most recent call last): > File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job >
drm_job_state.job_wrapper.finish(
stdout,
stderr ) > File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish > dataset.set_meta(
overwrite =
False )
> File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta > return
self.datatype.set_meta(
self, **kwd
) > File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta > raise Exception, "Error
Setting
BAM
Metadata: %s" % stderr > Exception: Error Setting BAM
Metadata:
/bin/sh:
samtools: not found > > It means that the samtools is not in
the PATH.
I tried
to set the PATH in a couple of methods
according
the Galaxy
documentation: > 1. put the path in the env.sh in the
tool
directory and
symbolink default to the tool directory,
e.g.
default ->
=/mnt/galaxyTools/tools/samtools/0.1.18 > 2. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request > 3. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request > > none of them worked, and I got the
above same
problem.
> > Then I checked the job log file in
the
job_working_directory, and it read: > Samtools Version: 0.1.18 (r982:295) > SAM file converted to BAM > > which shows that sge knows the PATH
of
samtools. To
double check it, I added samtools index to
Galaxy,
and it
worked well. I am very confused why SGE
knows the
tool path
but cannot run the job correctly. > > The system I am using is ubuntu on
EC2 (the
image id
is: ami-999d49f0, which is the one used
by
StarCluster). I
checked out the code from galaxy-dist on
bitbucket.
Other
tools such as bwa and bowtie worked well
using the
same
setting method(put env.sh in the tools
directory to
set the
tool path) > > > Thanks you very much, > > Cai > --- 12年4月18日，周三, Jennifer
Jackson
写道： > >> 发件人: Jennifer Jackson
>> 主题: Re: [galaxy-dev]
Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not
found
>> 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn >> 抄送: galaxy-dev@lists.bx.psu.edu >> 日期:
2012年4月18日,周三,上午2:12
>> Hi Cai, >> >> Please double check that your
env.sh file
contains
the >> following: >> >> >> >>
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
>> export PATH >> >> >> http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies >> >> >> Hopefully this helps, >> >> Jen >> Galaxy team >> >> On 4/16/12 12:43 PM, zhengqiu cai
wrote:
>>> Hi All, >>> >>> I was trying to setup the
Galaxy
tool
dependencies, and >> I met the following problem when
running
sam-to-bam
in >> samtools: >>> Traceback (most recent call
last):
>>> File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
>> line 336, in finish_job >>> drm_job_state.job_wrapper.finish( >> stdout, stderr ) >>> File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
>> line 637, in finish >>>
dataset.set_meta(
overwrite = False >> ) >>> File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
>> line 875, in set_meta >>> return self.datatype.set_meta( >> self, **kwd ) >>> File >>
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
>> line 179, in set_meta >>> raise
Exception,
"Error
Setting BAM >> Metadata: %s" % stderr >>> Exception: Error Setting
BAM
Metadata:
/bin/sh: >> samtools: not found >>> >>> >>> I already set the path of
samtools
>>> Below I pasted how I set the
path:
>>>
ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll >>> total 8 >>> drwxr-xr-x 4 ubuntu ubuntu 65 >> 2012-04-16 19:09 ./ >>> drwxr-xr-x 38 ubuntu ubuntu
4096
2012-04-16
18:41 ../ >>> drwxr-xr-x 2 ubuntu ubuntu 53 >> 2012-04-16 19:08 0.1.12/ >>> drwxr-xr-x 2 ubuntu ubuntu 53 >> 2012-04-13 20:23 0.1.7/ >>> lrwxrwxrwx 1 ubuntu ubuntu 38 >> 2012-04-16 19:09 default -> >>
/mnt/galaxyTools/tools/samtools/0.1.12/
>>> >>>
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more >> default/env.sh >>>
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
>>> >>> I used the same way to set
other
programs,
such as bwa >> and bowtie, and they worked
perfectly.
Only
samtools could >> not work. >>> I even used the full path of
samtools
in the
wrapper >> file sam_to_bam.py, and it was
not
working
neither. >>> >>> I am wondering if any one
met the
same
problem, or some >> one can give me some hints. >>> >>> Thank you very much. >>> >>> Cai >>> >>> >>
>>> Please keep all replies on
the list
by using
"reply >> all" >>> in your mail client.
To manage
your
subscriptions >> to this >>> and other Galaxy lists,
please use
the
interface at: >>> >>> http://lists.bx.psu.edu/ >> >> -- >> Jennifer Jackson >> http://galaxyproject.org >> > >
> Please keep all replies on the list
by using
"reply
all" > in your mail client. To manage
your
subscriptions
to this > and other Galaxy lists, please use
the
interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical
School;
Head, Medical Bioinformatics, BakerIDI;
Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Dan, This is exactly the problem. By default, it was set to False. It works by uncommenting this line and changing it to True. Thank you very much. Cai
--- 12年4月18日，周三, Daniel Blankenberg dan@bx.psu.edu 写道：
发件人: Daniel Blankenberg dan@bx.psu.edu 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found 收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn 抄送: "Ross" ross.lazarus@gmail.com, galaxy-dev@lists.bx.psu.edu 日期: 2012年4月18日,周三,下午11:47
Hi Cai,
Can you check that you have 'set_metadata_externally = True' in your universe_wsgi.ini file? This allows the set_meta process to occur on the nodes as part of the job. Otherwise, you will need to make sure that samtools is installed for the headnode.
Thanks for using Galaxy,
Dan
On Apr 18, 2012, at 11:39 AM, zhengqiu cai wrote:
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with Galaxy AMI image, and I copied it to the same path on my EC2. How to compile samtools using --prefix? I just typed 'make', and there is not a configuration file. In the cloudman version of Galaxy (it is working well for me), only two executables (samtools and maq2sam-long) in the directory /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is the only compiled executable 'samtools' fully functional for the cluster(it is on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com 写道：
发件人: Ross ross.lazarus@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn
抄送: "Jennifer Jackson" jen@bx.psu.edu, galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,下午11:07
Hi, Cai - sorry to hear it's still
not working right.
You've compiled/installed samtools using the --prefix
configure option
and tested that all the samtools based Galaxy wrappers work
if you use
the local job runner? That means that both the appropriate
lib and bin
directories are on the Galaxy user's path ready to be
inherited by SGE
with the -V SGE option.
If the local job runner works, that seems to isolate the
problem to
SGE configuration and operation - the steps I've suggested
seem to
work for us so I've run out of ideas on why it's not working
at your
shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2005@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here is a test
case I added to Galaxy, and it creates the index for fasta
file. I pasted the samtools faidx wrapper and xml files
below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV < 1) {
die "Usage:
perl $0 <input fasta file>\n";
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.xml
<tool id="samtools_faidx" name="samtools faidx"
version="1.0.3">
<requirements>
<requirement
type="package">samtools</requirement>
</requirements>
<description>creates fasta
index</description>
<command interpreter="perl">
samtools_faidx.pl $input1 $output1
</command>
<inputs>
<param name="input1" type="data"
format="fasta" label="fasta file to be indexed" />
</inputs>
<outputs>
<data format="fai" name="output1"
label="${tool.name} on ${on_string}: fasta" />
</outputs>
</tool>
It works, which shows that SGE knows the path of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn
抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18
Ah. Then you probably have a library
path problem.
Here's how we do it FWIW:
Assume your want your shared cluster apps in
/data/apps/bin
To share the compiled libraries and binaries
together,
remember to
compile apps like samtools using the
--prefix /data/app
flag so then you can add the /data/app/lib path to
the
galaxy user's
path so the nodes running jobs inherit those paths
because
of the -V
switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
wrote:
Hi Ross,
Thank you for your reply.
I tried it, and it was still not working.
Only samtools has this problem, and the other
tools
works fine.
samtools was compiled from source correctly.
If I
install samtools using "sudo apt-get install
samtools", this
problem is gone, but this is not a solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.lazarus@gmail.com
写道：
发件人: Ross ross.lazarus@gmail.com
主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn
抄送: "Jennifer Jackson" jen@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午4:59
Cai - if you are using SGE you
probably want to pass the Galaxy user's
path to the job using the
-V
switch in the system wide (or Galaxy
user's home
directory)
sge_request file
eg I have
rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd
-V
Without that switch, the job gets no
path.
With -V, it inherits the job submitter's
full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu
cai
wrote:
Hi Jen,
Thank you for your quick response.
I changed the env.sh file to what
you
seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to
bam, and
the job
was running forever without outputing the
result. I
then
checked the log, and it read:
Traceback (most recent call last):
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout,
stderr )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta(
overwrite =
False )
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return
self.datatype.set_meta(
self, **kwd
)
File
"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise Exception, "Error
Setting
BAM
Metadata: %s" % stderr
Exception: Error Setting BAM
Metadata:
/bin/sh:
samtools: not found
It means that the samtools is not in
the PATH.
I tried
to set the PATH in a couple of methods
according
the Galaxy
documentation:
1. put the path in the env.sh in the
tool
directory and
symbolink default to the tool directory,
e.g.
default ->
=/mnt/galaxyTools/tools/samtools/0.1.18
2. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
3. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the
above same
problem.
Then I checked the job log file in
the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295)
SAM file converted to BAM
which shows that sge knows the PATH
of
samtools. To
double check it, I added samtools index to
Galaxy,
and it
worked well. I am very confused why SGE
knows the
tool path
but cannot run the job correctly.
The system I am using is ubuntu on
EC2 (the
image id
is: ami-999d49f0, which is the one used
by
StarCluster). I
checked out the code from galaxy-dist on
bitbucket.
Other
tools such as bwa and bowtie worked well
using the
same
setting method(put env.sh in the tools
directory to
set the
tool path)
Thanks you very much,
Cai
--- 12年4月18日，周三, Jennifer
Jackson
写道：
发件人: Jennifer Jackson
主题: Re: [galaxy-dev]
Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not
found
收件人: "zhengqiu cai" caizhq2005@yahoo.com.cn
抄送: galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午2:12
Hi Cai,
Please double check that your
env.sh file
contains
the
following:
PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
export PATH
http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
Hopefully this helps,
Jen
Galaxy team
On 4/16/12 12:43 PM, zhengqiu cai
wrote:
Hi All,
I was trying to setup the
Galaxy
tool
dependencies, and
I met the following problem when
running
sam-to-bam
in
samtools:
Traceback (most recent call
last):
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout, stderr )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
line 637, in finish
dataset.set_meta(
overwrite = False
)
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
line 875, in set_meta
return
self.datatype.set_meta(
self, **kwd )
File
"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
line 179, in set_meta
raise
Exception,
"Error
Setting BAM
Metadata: %s" % stderr
Exception: Error Setting
BAM
Metadata:
/bin/sh:
samtools: not found
I already set the path of
samtools
Below I pasted how I set the
path:
ubuntu@master:/mnt/galaxyTools/tools/samtools$
ll
total 8
drwxr-xr-x 4 ubuntu
ubuntu 65
2012-04-16 19:09 ./
drwxr-xr-x 38 ubuntu ubuntu
4096
2012-04-16
18:41 ../
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-16 19:08 0.1.12/
drwxr-xr-x 2 ubuntu
ubuntu 53
2012-04-13 20:23 0.1.7/
lrwxrwxrwx 1 ubuntu
ubuntu 38
2012-04-16 19:09 default ->
/mnt/galaxyTools/tools/samtools/0.1.12/
ubuntu@master:/mnt/galaxyTools/tools/samtools$
more
default/env.sh
PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
I used the same way to set
other
programs,
such as bwa
and bowtie, and they worked
perfectly.
Only
samtools could
not work.
I even used the full path of
samtools
in the
wrapper
file sam_to_bam.py, and it was
not
working
neither.
I am wondering if any one
met the
same
problem, or some
one can give me some hints.
Thank you very much.
Cai
___________________________________________________________
Please keep all replies on
the list
by using
"reply
all"
in your mail client.
To manage
your
subscriptions
to this
and other Galaxy lists,
please use
the
interface at:
--
Jennifer Jackson
___________________________________________________________
Please keep all replies on the list
by using
"reply
all"
in your mail client. To manage
your
subscriptions
to this
and other Galaxy lists, please use
the
interface at:
--
Ross Lazarus MBBS MPH;
Associate Professor, Harvard Medical
School;
Head, Medical Bioinformatics, BakerIDI;
Tel: +61
385321444;
--
Ross Lazarus MBBS MPH;
Associate Professor, Harvard Medical School;
Head, Medical Bioinformatics, BakerIDI; Tel: +61
385321444;
galaxy-dev@lists.galaxyproject.org
-
Daniel Blankenberg -
Jennifer Jackson -
Ross -
zhengqiu cai