On Wed, Jan 11, 2012 at 11:16 AM, Nate Coraor <nate@bx.psu.edu> wrote:

On Jan 11, 2012, at 10:56 AM, Ryan Golhar wrote:

> it is also set to True:
> > [galaxy@bic galaxy-dist]$ grep Ryan *.log
> > runner0.log:galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True
> >
> > Clearly something else is going on here. On my last import of BAM file, even after samtools finished indexing the BAM files, Galaxy never registered those jobs as completed. So this problem is still there as well.
>
> Thanks for checking these. So, what does your server log for runner0 look like?
>
>
> > it is also set to True:
> > [galaxy@bic galaxy-dist]$ grep Ryan *.log
> > runner0.log:galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True

Sorry, I should have been more specific. What are the contents of the log from job start to finish?

--nate

>

I think this is it. The rest looks like stuff for jobs I started today. I wonder if Galaxy is running the jobs on the head node because its unable to change the permissions of the BAM files. The BAM files are located on a read-only NFS mount.

serving on http://127.0.0.1:8079

galaxy.jobs DEBUG 2012-01-10 22:16:54,322 dispatching job 62 to pbs runner

galaxy.jobs INFO 2012-01-10 22:16:54,436 job 62 dispatched

galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:54,595 (62) submitting file /home/galaxy/galaxy-dist-9/database/pbs/62.sh

galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:54,596 (62) command is: python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml /home/galaxy/galaxy-dist-9/database/tmp/tmpaEn2

jR 181:/home/galaxy/galaxy-dist-9/database/job_working_directory/62/dataset_181_files:None 182:/home/galaxy/galaxy-dist-9/database/job_working_directory/62/dataset_182_files:None 183:/home/galaxy/galaxy-dist-9/database/job_workin

g_directory/62/dataset_183_files:None 184:/home/galaxy/galaxy-dist-9/database/job_working_directory/62/dataset_184_files:None; cd /home/galaxy/galaxy-dist-9; /home/galaxy/galaxy-dist-9/set_metadata.sh ./database/files ./database/tmp . datatypes_

conf.xml ./database/job_working_directory/62/galaxy.json

galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:54,597 (62) queued in default queue as 73.localhost.localdomain

galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:55,179 (62/73.localhost.localdomain) PBS job state changed from N to R

galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:17:26,324 (62/73.localhost.localdomain) PBS job state changed from R to C

galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:17:26,324 (62/73.localhost.localdomain) PBS job has completed successfully

galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True

galaxy.jobs WARNING 2012-01-11 01:29:53,110 Ryan Golhar - self.set_metadata_externally = True

galaxy.jobs WARNING 2012-01-11 04:30:06,043 Ryan Golhar - self.set_metadata_externally = True

galaxy.jobs WARNING 2012-01-11 07:41:02,917 Ryan Golhar - self.set_metadata_externally = True

galaxy.util WARNING 2012-01-11 10:22:40,046 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mn

t/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam'

galaxy.util WARNING 2012-01-11 10:22:40,066 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam, group remains

grp.struct_group(gr_name='cag_lab2', gr_passwd='x', gr_gid=10726, gr_mem=['ryang', 'lifeng', 'galaxy']), error was: [Errno 30] Read-only file system: '/mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam'

galaxy.util WARNING 2012-01-11 10:22:40,069 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500_4_BK_WTS_03_20111222_1_04-1-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mnt/isi

lon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_04-1-1.bam'

galaxy.util WARNING 2012-01-11 10:22:40,069 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_04-1-1.bam, group remains grp.

struct_group(gr_name='cag_lab2', gr_passwd='x', gr_gid=10726, gr_mem=['ryang', 'lifeng', 'galaxy']), error was: [Errno 30] Read-only file system: '/mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_04-1-1.bam'

galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mn

t/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam'

galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam, group remains

galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500_4_BK_WTS_03_20111222_1_06-2-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mnt/isi

lon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_06-2-1.bam'

galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_06-2-1.bam, group remains grp.

galaxy.jobs DEBUG 2012-01-11 10:22:40,161 job 62 ended

galaxy.datatypes.metadata DEBUG 2012-01-11 10:22:40,162 Cleaning up external metadata files