Hi all,
after an update to the following changeset(14859:7ba05957588a, stable,
05.12.14), our bam files that are uploaded(linked) to a data library, are
no longer indexed. The metadata_xxx.dat is created, but it stays empty.
The following error message appears in the log, although the state of the
dataset is 'ok':
galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing /g/K/K27.bam,
will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam'
galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam,
will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'
All file permissions are correct (i.e. galaxy owns them). Furthermore,
executing samtools index, just works on those files:
samtools index /g/K/K2.bam
/g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat
When uploading the file - "copy files into galaxy" - the samtools index
just works.
==============
Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and
samtools installed globally and with the bam file sorted, I get the
following situation:
When I try to upload this bam to a data library by linking the following
error is shown on the dataset (note: here the dataset is set in error
state, which does not happen on our server)
Uploaded by: scholtal(a)embl.de
Date uploaded: Mon Dec 8 17:42:39 2014 (UTC)
File size: 2.6 GB
UUID: d23cf11a-0372-41cb-939a-7c8761d78b73
Data type: auto
Build: ?
Miscellaneous information: Traceback (most recent call last): File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in __main__() File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 396, in _
Job Standard Error
Traceback (most recent call last):
File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in
__main__()
File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 396, in __main__
add_file( dataset, registry, json_file, output_path )
File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 294, in add_file
if datatype.dataset_content_needs_grooming( dataset.path ):
File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 147, in dataset_content_needs_grooming
version = self._get_samtools_version()
File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 129, in _get_samtools_version
output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
stdout=subprocess.PIPE ).communicate()[1]
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 711, in __init__
errread, errwrite)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
error
Database/Build: ?
Number of data lines: None
Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam
===============================
When uploading the bam without linking, I see the following processes:
Upload->set meta->samtools index->'error state'
Miscellaneous information: uploaded bam file Traceback (most recent call
last): File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in __main__() File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p
Job Standard Error
Traceback (most recent call last):
File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in
__main__()
File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 396, in __main__
add_file( dataset, registry, json_file, output_path )
File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 324, in add_file
if link_data_only == 'copy_files' and
datatype.dataset_content_needs_grooming( output_path ):
File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 147, in dataset_content_needs_grooming
version = self._get_samtools_version()
File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 129, in _get_samtools_version
output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
stdout=subprocess.PIPE ).communicate()[1]
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 711, in __init__
errread, errwrite)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
error
Database/Build: ?
Number of data lines: None
Disk file:
/Users/scholtalbers/workspace/galaxy-dist-new/database/files/000/dataset_6.dat
=================================
Although the error messages are different, it might be related?
Cheers,
Jelle