Hi all,

after an update to the following changeset(14859:7ba05957588a, stable, 05.12.14), our bam files that are uploaded(linked) to a data library, are no longer indexed. The metadata_xxx.dat is created, but it stays empty.
The following error message appears in the log, although the state of the dataset is 'ok':

galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing /g/K/K27.bam, will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam'
galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam, will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'

All file permissions are correct (i.e. galaxy owns them). Furthermore, executing samtools index, just works on those files:
samtools index /g/K/K2.bam /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat

When uploading the file - "copy files into galaxy" - the samtools index just works.

==============

Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and samtools installed globally and with the bam file sorted, I get the following situation:
When I try to upload this bam to a data library by linking the following error is shown on the dataset (note: here the dataset is set in error state, which does not happen on our server)

Uploaded by: scholtal@embl.de
Date uploaded: Mon Dec 8 17:42:39 2014 (UTC)
File size: 2.6 GB
UUID: d23cf11a-0372-41cb-939a-7c8761d78b73
Data type: auto
Build: ?
Miscellaneous information: Traceback (most recent call last): File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 396, in _
Job Standard Error

Traceback (most recent call last):
  File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in
    __main__()
  File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 396, in __main__
    add_file( dataset, registry, json_file, output_path )
  File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 294, in add_file
    if datatype.dataset_content_needs_grooming( dataset.path ):
  File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 147, in dataset_content_needs_grooming
    version = self._get_samtools_version()
  File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 129, in _get_samtools_version
    output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

error
Database/Build: ?
Number of data lines: None
Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam

===============================
When uploading the bam without linking, I see the following processes:
Upload->set meta->samtools index->'error state'

 Miscellaneous information: uploaded bam file Traceback (most recent call last): File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p
Job Standard Error

Traceback (most recent call last):
  File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 407, in
    __main__()
  File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 396, in __main__
    add_file( dataset, registry, json_file, output_path )
  File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py", line 324, in add_file
    if link_data_only == 'copy_files' and datatype.dataset_content_needs_grooming( output_path ):
  File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 147, in dataset_content_needs_grooming
    version = self._get_samtools_version()
  File "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py", line 129, in _get_samtools_version
    output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

error
Database/Build: ?
Number of data lines: None
Disk file: /Users/scholtalbers/workspace/galaxy-dist-new/database/files/000/dataset_6.dat

=================================

Although the error messages are different, it might be related?

Cheers,

Jelle