Hi John and others,

The "OSError: [Errno 2] No such file or directory" is solved when putting the "shell=True" in the mentioned locations - as written in the Trello card.

However, even after fixing that, linked BAM files where not indexed, or at least the "files/_metadata_files/000/metadata_XXX.dat" is empty on job completion. After spending too much time on this, I nailed down the problem to this setting:
outputs_to_working_directory = True

With this option set, the BAM index file is never moved to the right location (or is is not being created.)
Setting this option to False (I think the setting was a left over from a failed attempt to use "real user job submission") this now just works.

- Jelle


On Mon, Dec 15, 2014 at 8:31 PM, John Chilton <jmchilton@gmail.com> wrote:
Hello,

  Sorry we haven't made progress on this - and thanks for creating a
Trello card (https://trello.com/c/tw75nq1U).

  It looks like samtools is not on your path, can you use it from the
command-line? If not you should probably install it - I would suggest
install samtools 0.1.19 with homebrew (looks like you are a on Mac).
If you can run samtools from the command-line - can you do a which
`samtools` from your command-line and see where it is coming from and
then add that directory explicitly to your Galaxy PATH - say at the
top of run.sh in your Galaxy root (let me know if you need more
details on that).

-John

On Tue, Dec 9, 2014 at 3:02 AM, Jelle Scholtalbers
<j.scholtalbers@gmail.com> wrote:
> Hi all,
>
> after an update to the following changeset(14859:7ba05957588a, stable,
> 05.12.14), our bam files that are uploaded(linked) to a data library, are no
> longer indexed. The metadata_xxx.dat is created, but it stays empty.
> The following error message appears in the log, although the state of the
> dataset is 'ok':
>
> galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing /g/K/K27.bam,
> will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam'
> galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam,
> will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'
>
> All file permissions are correct (i.e. galaxy owns them). Furthermore,
> executing samtools index, just works on those files:
> samtools index /g/K/K2.bam
> /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat
>
> When uploading the file - "copy files into galaxy" - the samtools index just
> works.
>
> ==============
>
> Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and
> samtools installed globally and with the bam file sorted, I get the
> following situation:
> When I try to upload this bam to a data library by linking the following
> error is shown on the dataset (note: here the dataset is set in error state,
> which does not happen on our server)
>
> Uploaded by: scholtal@embl.de
> Date uploaded: Mon Dec 8 17:42:39 2014 (UTC)
> File size: 2.6 GB
> UUID: d23cf11a-0372-41cb-939a-7c8761d78b73
> Data type: auto
> Build: ?
> Miscellaneous information: Traceback (most recent call last): File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in __main__() File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 396, in _
> Job Standard Error
>
> Traceback (most recent call last):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in
>     __main__()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 396, in __main__
>     add_file( dataset, registry, json_file, output_path )
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 294, in add_file
>     if datatype.dataset_content_needs_grooming( dataset.path ):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 147, in dataset_content_needs_grooming
>     version = self._get_samtools_version()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 129, in _get_samtools_version
>     output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
> stdout=subprocess.PIPE ).communicate()[1]
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 711, in __init__
>     errread, errwrite)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 1308, in _execute_child
>     raise child_exception
> OSError: [Errno 2] No such file or directory
>
> error
> Database/Build: ?
> Number of data lines: None
> Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam
>
> ===============================
> When uploading the bam without linking, I see the following processes:
> Upload->set meta->samtools index->'error state'
>
>  Miscellaneous information: uploaded bam file Traceback (most recent call
> last): File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in __main__() File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p
> Job Standard Error
>
> Traceback (most recent call last):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in
>     __main__()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 396, in __main__
>     add_file( dataset, registry, json_file, output_path )
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 324, in add_file
>     if link_data_only == 'copy_files' and
> datatype.dataset_content_needs_grooming( output_path ):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 147, in dataset_content_needs_grooming
>     version = self._get_samtools_version()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 129, in _get_samtools_version
>     output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
> stdout=subprocess.PIPE ).communicate()[1]
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 711, in __init__
>     errread, errwrite)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 1308, in _execute_child
>     raise child_exception
> OSError: [Errno 2] No such file or directory
>
> error
> Database/Build: ?
> Number of data lines: None
> Disk file:
> /Users/scholtalbers/workspace/galaxy-dist-new/database/files/000/dataset_6.dat
>
> =================================
>
> Although the error messages are different, it might be related?
>
> Cheers,
>
> Jelle
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/