Hi Cai,

Can you check that you have 'set_metadata_externally = True' in your universe_wsgi.ini file? This allows the set_meta process to occur on the nodes as part of the job. Otherwise, you will need to make sure that samtools is installed for the headnode.

Thanks for using Galaxy,

Dan

On Apr 18, 2012, at 11:39 AM, zhengqiu cai wrote:

Hi Ross,

I did not compile samtools using the --prefix. The samtools I used comes with Galaxy AMI image, and I copied it to the same path on my EC2. How to compile samtools using --prefix? I just typed 'make', and there is not a configuration file. In the cloudman version of Galaxy (it is working well for me), only two executables (samtools and maq2sam-long) in the directory /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is the only compiled executable 'samtools' fully functional for the cluster(it is on the local machine)? or should I set some other libraries?

Thank you,

Cai

--- 12年4月18日，周三, Ross <ross.lazarus@gmail.com> 写道：

发件人: Ross <ross.lazarus@gmail.com>

主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found

收件人: "zhengqiu cai" <caizhq2005@yahoo.com.cn>

抄送: "Jennifer Jackson" <jen@bx.psu.edu>, galaxy-dev@lists.bx.psu.edu

日期: 2012年4月18日,周三,下午11:07

Hi, Cai - sorry to hear it's still

not working right.

You've compiled/installed samtools using the --prefix

configure option

and tested that all the samtools based Galaxy wrappers work

if you use

the local job runner? That means that both the appropriate

lib and bin

directories are on the Galaxy user's path ready to be

inherited by SGE

with the -V SGE option.

If the local job runner works, that seems to isolate the

problem to

SGE configuration and operation - the steps I've suggested

seem to

work for us so I've run out of ideas on why it's not working

at your

shop and hope somebody with more experience can help out.

On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai <caizhq2005@yahoo.com.cn>

wrote:

Hi Ross,

Thank you for your reply.

It still did not work.

I think SGE knows the path of samtools. Here is a test

case I added to Galaxy, and it creates the index for fasta

file. I pasted the samtools faidx wrapper and xml files

below:

ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$

more samtools_faidx.pl

#!/usr/bin/perl

use strict;

use warnings;

if (@ARGV < 1) {

die "Usage:

   perl $0 <input fasta file>\n";

}

`samtools faidx $ARGV[0]`;

`cp ${ARGV[0]}.fai $ARGV[1]`;

exit;

ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$

more samtools_faidx.xml

<tool id="samtools_faidx" name="samtools faidx"

version="1.0.3">

<requirements>

<requirement

type="package">samtools</requirement>

</requirements>

<description>creates fasta

index</description>

<command interpreter="perl">

samtools_faidx.pl $input1 $output1

</command>

<inputs>

<param name="input1" type="data"

format="fasta" label="fasta file to be indexed" />

</inputs>

<outputs>

<data format="fai" name="output1"

label="${tool.name} on ${on_string}: fasta" />

</outputs>

</tool>

It works, which shows that SGE knows the path of

samtools.

Cai

--- 12年4月18日，周三, Ross <ross.lazarus@gmail.com>

写道：

发件人: Ross <ross.lazarus@gmail.com>

主题: Re: [galaxy-dev] Exception: Error Setting

BAM Metadata: /bin/sh: samtools: not found

收件人: "zhengqiu cai" <caizhq2005@yahoo.com.cn>

抄送: "Jennifer Jackson" <jen@bx.psu.edu>,

galaxy-dev@lists.bx.psu.edu

日期: 2012年4月18日,周三,上午5:18

Ah. Then you probably have a library

path problem.

Here's how we do it FWIW:

Assume your want your shared cluster apps in

/data/apps/bin

To share the compiled libraries and binaries

together,

remember to

compile apps like samtools using the

--prefix /data/app

flag so then you can add the /data/app/lib path to

the

galaxy user's

path so the nodes running jobs inherit those paths

because

of the -V

switch.

I hope this works!

On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai

<caizhq2005@yahoo.com.cn>

wrote:

Hi Ross,

Thank you for your reply.

I tried it, and it was still not working.

Only samtools has this problem, and the other

tools

works fine.

samtools was compiled from source correctly.

If I

install samtools using "sudo apt-get install

samtools", this

problem is gone, but this is not a solution for

since it

does not work across the cluster.

Cai

--- 12年4月18日，周三, Ross <ross.lazarus@gmail.com>

写道：

发件人: Ross <ross.lazarus@gmail.com>

主题: Re: [galaxy-dev] Exception: Error

Setting

BAM Metadata: /bin/sh: samtools: not found

收件人: "zhengqiu cai" <caizhq2005@yahoo.com.cn>

抄送: "Jennifer Jackson" <jen@bx.psu.edu>,

galaxy-dev@lists.bx.psu.edu

日期:

2012年4月18日,周三,上午4:59

Cai - if you are using SGE you

probably want to pass the Galaxy user's

path to the job using the

-V

switch in the system wide (or Galaxy

user's home

directory)

sge_request file

eg I have

rlazarus@iaas1-int:~$ cat

/var/lib/gridengine/default/common/sge_request

-cwd

-V

Without that switch, the job gets no

path.

With -V, it inherits the job submitter's

full

path.

On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu

cai

<caizhq2005@yahoo.com.cn>

wrote:

Hi Jen,

Thank you for your quick response.

I changed the env.sh file to what

you

seggested, and

the problem still existed.

I am pasting more details below:

I submitted a job to convert sam to

bam, and

the job

was running forever without outputing the

result. I

then

checked the log, and it read:

Traceback (most recent call last):

File

"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",

line 336, in finish_job

drm_job_state.job_wrapper.finish(

stdout,

stderr )

File

"/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",

line 637, in finish

dataset.set_meta(

overwrite =

False )

File

"/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",

line 875, in set_meta

return

self.datatype.set_meta(

self, **kwd

)

File

"/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",

line 179, in set_meta

raise Exception, "Error

Setting

BAM

Metadata: %s" % stderr

Exception: Error Setting BAM

Metadata:

/bin/sh:

samtools: not found

It means that the samtools is not in

the PATH.

I tried

to set the PATH in a couple of methods

according

the Galaxy

documentation:

1. put the path in the env.sh in the

tool

directory and

symbolink default to the tool directory,

e.g.

default ->

=/mnt/galaxyTools/tools/samtools/0.1.18

2. put -v

PATH=/mnt/galaxyTools/tools/samtools/0.1.18

in ~/.sge_request

3. put -v

PATH=/mnt/galaxyTools/tools/samtools/0.1.18

in /path/sge_request

none of them worked, and I got the

above same

problem.

Then I checked the job log file in

the

job_working_directory, and it read:

Samtools Version: 0.1.18 (r982:295)

SAM file converted to BAM

which shows that sge knows the PATH

of

samtools. To

double check it, I added samtools index to

Galaxy,

and it

worked well. I am very confused why SGE

knows the

tool path

but cannot run the job correctly.

The system I am using is ubuntu on

EC2 (the

image id

is: ami-999d49f0, which is the one used

by

StarCluster). I

checked out the code from galaxy-dist on

bitbucket.

Other

tools such as bwa and bowtie worked well

using the

same

setting method(put env.sh in the tools

directory to

set the

tool path)

Thanks you very much,

Cai

--- 12年4月18日，周三, Jennifer

Jackson

<jen@bx.psu.edu>

写道：

发件人: Jennifer Jackson

<jen@bx.psu.edu>

主题: Re: [galaxy-dev]

Exception: Error

Setting

BAM Metadata: /bin/sh: samtools: not

found

收件人: "zhengqiu cai" <caizhq2005@yahoo.com.cn>

抄送: galaxy-dev@lists.bx.psu.edu

日期:

2012年4月18日,周三,上午2:12

Hi Cai,

Please double check that your

env.sh file

contains

the

following:

PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"

export PATH

http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies

Hopefully this helps,

Jen

Galaxy team

On 4/16/12 12:43 PM, zhengqiu cai

wrote:

Hi All,

I was trying to setup the

Galaxy

tool

dependencies, and

I met the following problem when

running

sam-to-bam

in

samtools:

Traceback (most recent call

last):

File

"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",

line 336, in finish_job

drm_job_state.job_wrapper.finish(

stdout, stderr )

File

"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",

line 637, in finish

dataset.set_meta(

overwrite = False

)

File

"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",

line 875, in set_meta

return

self.datatype.set_meta(

self, **kwd )

File

"/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",

line 179, in set_meta

raise

Exception,

"Error

Setting BAM

Metadata: %s" % stderr

Exception: Error Setting

BAM

Metadata:

/bin/sh:

samtools: not found

I already set the path of

samtools

Below I pasted how I set the

path:

ubuntu@master:/mnt/galaxyTools/tools/samtools$

ll

total 8

drwxr-xr-x 4 ubuntu

ubuntu   65

2012-04-16 19:09 ./

drwxr-xr-x 38 ubuntu ubuntu

4096

2012-04-16

18:41 ../

drwxr-xr-x 2 ubuntu

ubuntu   53

2012-04-16 19:08 0.1.12/

drwxr-xr-x 2 ubuntu

ubuntu   53

2012-04-13 20:23 0.1.7/

lrwxrwxrwx 1 ubuntu

ubuntu   38

2012-04-16 19:09 default ->

/mnt/galaxyTools/tools/samtools/0.1.12/

ubuntu@master:/mnt/galaxyTools/tools/samtools$

more

default/env.sh

PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH

I used the same way to set

other

programs,

such as bwa

and bowtie, and they worked

perfectly.

Only

samtools could

not work.

I even used the full path of

samtools

in the

wrapper

file sam_to_bam.py, and it was

not

working

neither.

I am wondering if any one

met the

same

problem, or some

one can give me some hints.

Thank you very much.

Cai

___________________________________________________________

Please keep all replies on

the list

by using

"reply

all"

in your mail client.

To manage

your

subscriptions

to this

and other Galaxy lists,

please use

the

interface at:

http://lists.bx.psu.edu/

--

Jennifer Jackson

http://galaxyproject.org

___________________________________________________________

Please keep all replies on the list

by using

"reply

all"

in your mail client. To manage

your

subscriptions

to this

and other Galaxy lists, please use

the

interface at:

http://lists.bx.psu.edu/

--

Ross Lazarus MBBS MPH;

Associate Professor, Harvard Medical

School;

Head, Medical Bioinformatics, BakerIDI;

Tel: +61

385321444;

--

Ross Lazarus MBBS MPH;

Associate Professor, Harvard Medical School;

Head, Medical Bioinformatics, BakerIDI; Tel: +61

385321444;

--

Ross Lazarus MBBS MPH;

Associate Professor, Harvard Medical School;

Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/