Hi all, after an update to the following changeset(14859:7ba05957588a, stable, 05.12.14), our bam files that are uploaded(linked) to a data library, are no longer indexed. The metadata_xxx.dat is created, but it stays empty. The following error message appears in the log, although the state of the dataset is 'ok': galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing /g/K/K27.bam, will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam' galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam, will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam' All file permissions are correct (i.e. galaxy owns them). Furthermore, executing samtools index, just works on those files: samtools index /g/K/K2.bam /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat When uploading the file - "copy files into galaxy" - the samtools index just works. ============== Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and samtools installed globally and with the bam file sorted, I get the following situation: When I try to upload this bam to a data library by linking the following error is shown on the dataset (note: here the dataset is set in error state, which does not happen on our server) Uploaded by: scholtal@embl.de Date uploaded: Mon Dec 8 17:42:39 2014 (UTC) File size: 2.6 GB UUID: d23cf11a-0372-41cb-939a-7c8761d78b73 Data type: auto Build: ? Miscellaneous information: Traceback (most recent call last): File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 396, in _ Job Standard Error Traceback (most recent call last): File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 396, in __main__ add_file( dataset, registry, json_file, output_path ) File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 294, in add_file if datatype.dataset_content_needs_grooming( dataset.path ): File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 147, in dataset_content_needs_grooming version = self._get_samtools_version() File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 129, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory error Database/Build: ? Number of data lines: None Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam =============================== When uploading the bam without linking, I see the following processes: Upload->set meta->samtools index->'error state' Miscellaneous information: uploaded bam file Traceback (most recent call last): File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p Job Standard Error Traceback (most recent call last): File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 396, in __main__ add_file( dataset, registry, json_file, output_path ) File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 324, in add_file if link_data_only == 'copy_files' and datatype.dataset_content_needs_grooming( output_path ): File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 147, in dataset_content_needs_grooming version = self._get_samtools_version() File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 129, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory error Database/Build: ? Number of data lines: None Disk file: /Users/scholtalbers/workspace/galaxy-dist-new/database/files/000/dataset_6.dat ================================= Although the error messages are different, it might be related? Cheers, Jelle