On Wed, Jan 11, 2012 at 11:16 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Jan 11, 2012, at 10:56 AM, Ryan Golhar wrote:
it is also set to True:
[galaxy@bic galaxy-dist]$ grep Ryan *.log runner0.log:galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True
Clearly something else is going on here. On my last import of BAM file, even after samtools finished indexing the BAM files, Galaxy never registered those jobs as completed. So this problem is still there as well.
Thanks for checking these. So, what does your server log for runner0 look like?
it is also set to True: [galaxy@bic galaxy-dist]$ grep Ryan *.log runner0.log:galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True
Sorry, I should have been more specific. What are the contents of the log from job start to finish?
--nate
I think this is it. The rest looks like stuff for jobs I started today. I wonder if Galaxy is running the jobs on the head node because its unable to change the permissions of the BAM files. The BAM files are located on a read-only NFS mount.
serving on http://127.0.0.1:8079 galaxy.jobs DEBUG 2012-01-10 22:16:54,322 dispatching job 62 to pbs runner galaxy.jobs INFO 2012-01-10 22:16:54,436 job 62 dispatched galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:54,595 (62) submitting file /home/galaxy/galaxy-dist-9/database/pbs/62.sh galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:54,596 (62) command is: python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml /home/galaxy/galaxy-dist-9/database/tmp/tmpaEn2 jR 181:/home/galaxy/galaxy-dist-9/database/job_working_directory/62/dataset_181_files:None 182:/home/galaxy/galaxy-dist-9/database/job_working_directory/62/dataset_182_files:None 183:/home/galaxy/galaxy-dist-9/database/job_workin g_directory/62/dataset_183_files:None 184:/home/galaxy/galaxy-dist-9/database/job_working_directory/62/dataset_184_files:None; cd /home/galaxy/galaxy-dist-9; /home/galaxy/galaxy-dist-9/set_metadata.sh ./database/files ./database/tmp . datatypes_ conf.xml ./database/job_working_directory/62/galaxy.json galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:54,597 (62) queued in default queue as 73.localhost.localdomain galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:16:55,179 (62/73.localhost.localdomain) PBS job state changed from N to R galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:17:26,324 (62/73.localhost.localdomain) PBS job state changed from R to C galaxy.jobs.runners.pbs DEBUG 2012-01-10 22:17:26,324 (62/73.localhost.localdomain) PBS job has completed successfully galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True galaxy.jobs WARNING 2012-01-11 01:29:53,110 Ryan Golhar - self.set_metadata_externally = True galaxy.jobs WARNING 2012-01-11 04:30:06,043 Ryan Golhar - self.set_metadata_externally = True galaxy.jobs WARNING 2012-01-11 07:41:02,917 Ryan Golhar - self.set_metadata_externally = True galaxy.util WARNING 2012-01-11 10:22:40,046 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mn t/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,066 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam, group remains grp.struct_group(gr_name='cag_lab2', gr_passwd='x', gr_gid=10726, gr_mem=['ryang', 'lifeng', 'galaxy']), error was: [Errno 30] Read-only file system: '/mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_06-2-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,069 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500_4_BK_WTS_03_20111222_1_04-1-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mnt/isi lon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_04-1-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,069 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_04-1-1.bam, group remains grp. struct_group(gr_name='cag_lab2', gr_passwd='x', gr_gid=10726, gr_mem=['ryang', 'lifeng', 'galaxy']), error was: [Errno 30] Read-only file system: '/mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_04-1-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mn t/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam, group remains grp.struct_group(gr_name='cag_lab2', gr_passwd='x', gr_gid=10726, gr_mem=['ryang', 'lifeng', 'galaxy']), error was: [Errno 30] Read-only file system: '/mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500xl_5_2011_BK_WTS_03_122211_1_04-1-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor umask (02) for /mnt/isilon/cag/ngs/hiseq/golharr/Bookman/5500_4_BK_WTS_03_20111222_1_06-2-1.bam, tried to set: 0664 but mode remains 0771, error was: [Errno 30] Read-only file system: '/mnt/isi lon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_06-2-1.bam' galaxy.util WARNING 2012-01-11 10:22:40,072 Unable to honor primary group (grp.struct_group(gr_name='galaxy', gr_passwd='x', gr_gid=503, gr_mem=[])) for /mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_06-2-1.bam, group remains grp. struct_group(gr_name='cag_lab2', gr_passwd='x', gr_gid=10726, gr_mem=['ryang', 'lifeng', 'galaxy']), error was: [Errno 30] Read-only file system: '/mnt/isilon/cag/ngs/hiseq/golharr/Bk/5500_4_BK_WTS_03_20111222_1_06-2-1.bam' galaxy.jobs DEBUG 2012-01-11 10:22:40,161 job 62 ended galaxy.datatypes.metadata DEBUG 2012-01-11 10:22:40,162 Cleaning up external metadata files