Hi John,

with this in my run.sh:
echo $PATH
which samtools
python ./scripts/paster.py serve $GALAXY_CONFIG_FILE $@

I get the following output:
/Users/scholtal/python-envs/galaxy-dist-new/bin:/Users/scholtal/perl5/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:~/bin:/Users/scholtal/workspace/installed_tools/meme/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/local/MacGPG2/bin
/Users/scholtal/bin/samtools
Entering daemon mode

I do not know what happens after that with the path..

- Jelle

On Thu, Dec 18, 2014 at 6:21 PM, John Chilton <jmchilton@gmail.com> wrote:
So I have been able to reproduce this problem with
outputs_to_working_directory - though I am not sure how to address it
I have marked the Trello card as a confirmed bug.

The problem with shell=True is still confusing me - is not a normal
executable on your PATH? I feel like modifying your PATH to make sure
samtools is properly on it in someway is a better fix than sticking
shell=True in there. I'll think about it though - certainly a lot of
people get tripped up by the samtools requirement and Galaxy isn't
great a reporting the nature of the problem.

-John

On Thu, Dec 18, 2014 at 7:51 AM, Jelle Scholtalbers
<j.scholtalbers@gmail.com> wrote:
> Hi John and others,
>
> The "OSError: [Errno 2] No such file or directory" is solved when putting
> the "shell=True" in the mentioned locations - as written in the Trello card.
>
> However, even after fixing that, linked BAM files where not indexed, or at
> least the "files/_metadata_files/000/metadata_XXX.dat" is empty on job
> completion. After spending too much time on this, I nailed down the problem
> to this setting:
> outputs_to_working_directory = True
>
> With this option set, the BAM index file is never moved to the right
> location (or is is not being created.)
> Setting this option to False (I think the setting was a left over from a
> failed attempt to use "real user job submission") this now just works.
>
> - Jelle
>
>
> On Mon, Dec 15, 2014 at 8:31 PM, John Chilton <jmchilton@gmail.com> wrote:
>>
>> Hello,
>>
>>   Sorry we haven't made progress on this - and thanks for creating a
>> Trello card (https://trello.com/c/tw75nq1U).
>>
>>   It looks like samtools is not on your path, can you use it from the
>> command-line? If not you should probably install it - I would suggest
>> install samtools 0.1.19 with homebrew (looks like you are a on Mac).
>> If you can run samtools from the command-line - can you do a which
>> `samtools` from your command-line and see where it is coming from and
>> then add that directory explicitly to your Galaxy PATH - say at the
>> top of run.sh in your Galaxy root (let me know if you need more
>> details on that).
>>
>> -John
>>
>> On Tue, Dec 9, 2014 at 3:02 AM, Jelle Scholtalbers
>> <j.scholtalbers@gmail.com> wrote:
>> > Hi all,
>> >
>> > after an update to the following changeset(14859:7ba05957588a, stable,
>> > 05.12.14), our bam files that are uploaded(linked) to a data library,
>> > are no
>> > longer indexed. The metadata_xxx.dat is created, but it stays empty.
>> > The following error message appears in the log, although the state of
>> > the
>> > dataset is 'ok':
>> >
>> > galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing
>> > /g/K/K27.bam,
>> > will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam'
>> > galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam,
>> > will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'
>> >
>> > All file permissions are correct (i.e. galaxy owns them). Furthermore,
>> > executing samtools index, just works on those files:
>> > samtools index /g/K/K2.bam
>> > /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat
>> >
>> > When uploading the file - "copy files into galaxy" - the samtools index
>> > just
>> > works.
>> >
>> > ==============
>> >
>> > Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and
>> > samtools installed globally and with the bam file sorted, I get the
>> > following situation:
>> > When I try to upload this bam to a data library by linking the following
>> > error is shown on the dataset (note: here the dataset is set in error
>> > state,
>> > which does not happen on our server)
>> >
>> > Uploaded by: scholtal@embl.de
>> > Date uploaded: Mon Dec 8 17:42:39 2014 (UTC)
>> > File size: 2.6 GB
>> > UUID: d23cf11a-0372-41cb-939a-7c8761d78b73
>> > Data type: auto
>> > Build: ?
>> > Miscellaneous information: Traceback (most recent call last): File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 407, in __main__() File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 396, in _
>> > Job Standard Error
>> >
>> > Traceback (most recent call last):
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 407, in
>> >     __main__()
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 396, in __main__
>> >     add_file( dataset, registry, json_file, output_path )
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 294, in add_file
>> >     if datatype.dataset_content_needs_grooming( dataset.path ):
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
>> > line 147, in dataset_content_needs_grooming
>> >     version = self._get_samtools_version()
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
>> > line 129, in _get_samtools_version
>> >     output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
>> > stdout=subprocess.PIPE ).communicate()[1]
>> >   File
>> >
>> > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
>> > line 711, in __init__
>> >     errread, errwrite)
>> >   File
>> >
>> > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
>> > line 1308, in _execute_child
>> >     raise child_exception
>> > OSError: [Errno 2] No such file or directory
>> >
>> > error
>> > Database/Build: ?
>> > Number of data lines: None
>> > Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam
>> >
>> > ===============================
>> > When uploading the bam without linking, I see the following processes:
>> > Upload->set meta->samtools index->'error state'
>> >
>> >  Miscellaneous information: uploaded bam file Traceback (most recent
>> > call
>> > last): File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 407, in __main__() File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p
>> > Job Standard Error
>> >
>> > Traceback (most recent call last):
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 407, in
>> >     __main__()
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 396, in __main__
>> >     add_file( dataset, registry, json_file, output_path )
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 324, in add_file
>> >     if link_data_only == 'copy_files' and
>> > datatype.dataset_content_needs_grooming( output_path ):
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
>> > line 147, in dataset_content_needs_grooming
>> >     version = self._get_samtools_version()
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
>> > line 129, in _get_samtools_version
>> >     output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
>> > stdout=subprocess.PIPE ).communicate()[1]
>> >   File
>> >
>> > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
>> > line 711, in __init__
>> >     errread, errwrite)
>> >   File
>> >
>> > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
>> > line 1308, in _execute_child
>> >     raise child_exception
>> > OSError: [Errno 2] No such file or directory
>> >
>> > error
>> > Database/Build: ?
>> > Number of data lines: None
>> > Disk file:
>> >
>> > /Users/scholtalbers/workspace/galaxy-dist-new/database/files/000/dataset_6.dat
>> >
>> > =================================
>> >
>> > Although the error messages are different, it might be related?
>> >
>> > Cheers,
>> >
>> > Jelle
>> >
>> > ___________________________________________________________
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/