Hi John and others, The "OSError: [Errno 2] No such file or directory" is solved when putting the "shell=True" in the mentioned locations - as written in the Trello card. However, even after fixing that, linked BAM files where not indexed, or at least the "files/_metadata_files/000/metadata_XXX.dat" is empty on job completion. After spending too much time on this, I nailed down the problem to this setting: outputs_to_working_directory = True With this option set, the BAM index file is never moved to the right location (or is is not being created.) Setting this option to False (I think the setting was a left over from a failed attempt to use "real user job submission") this now just works. - Jelle On Mon, Dec 15, 2014 at 8:31 PM, John Chilton <jmchilton@gmail.com> wrote:
Hello,
Sorry we haven't made progress on this - and thanks for creating a Trello card (https://trello.com/c/tw75nq1U).
It looks like samtools is not on your path, can you use it from the command-line? If not you should probably install it - I would suggest install samtools 0.1.19 with homebrew (looks like you are a on Mac). If you can run samtools from the command-line - can you do a which `samtools` from your command-line and see where it is coming from and then add that directory explicitly to your Galaxy PATH - say at the top of run.sh in your Galaxy root (let me know if you need more details on that).
-John
On Tue, Dec 9, 2014 at 3:02 AM, Jelle Scholtalbers <j.scholtalbers@gmail.com> wrote:
Hi all,
after an update to the following changeset(14859:7ba05957588a, stable, 05.12.14), our bam files that are uploaded(linked) to a data library, are no longer indexed. The metadata_xxx.dat is created, but it stays empty. The following error message appears in the log, although the state of the dataset is 'ok':
galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing /g/K/K27.bam, will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam' galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam, will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'
All file permissions are correct (i.e. galaxy owns them). Furthermore, executing samtools index, just works on those files: samtools index /g/K/K2.bam /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat
When uploading the file - "copy files into galaxy" - the samtools index just works.
==============
Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and samtools installed globally and with the bam file sorted, I get the following situation: When I try to upload this bam to a data library by linking the following error is shown on the dataset (note: here the dataset is set in error state, which does not happen on our server)
Uploaded by: scholtal@embl.de Date uploaded: Mon Dec 8 17:42:39 2014 (UTC) File size: 2.6 GB UUID: d23cf11a-0372-41cb-939a-7c8761d78b73 Data type: auto Build: ? Miscellaneous information: Traceback (most recent call last): File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in __main__() File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 396, in _ Job Standard Error
Traceback (most recent call last): File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in __main__() File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 396, in __main__ add_file( dataset, registry, json_file, output_path ) File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 294, in add_file if datatype.dataset_content_needs_grooming( dataset.path ): File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 147, in dataset_content_needs_grooming version = self._get_samtools_version() File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 129, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 711, in __init__ errread, errwrite) File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 1308, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
error Database/Build: ? Number of data lines: None Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam
=============================== When uploading the bam without linking, I see the following processes: Upload->set meta->samtools index->'error state'
Miscellaneous information: uploaded bam file Traceback (most recent call last): File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in __main__() File "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p Job Standard Error
Traceback (most recent call last): File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 407, in __main__() File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 396, in __main__ add_file( dataset, registry, json_file, output_path ) File
"/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
line 324, in add_file if link_data_only == 'copy_files' and datatype.dataset_content_needs_grooming( output_path ): File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 147, in dataset_content_needs_grooming version = self._get_samtools_version() File
"/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
line 129, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 711, in __init__ errread, errwrite) File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 1308, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
error Database/Build: ? Number of data lines: None Disk file:
/Users/scholtalbers/workspace/galaxy-dist-new/database/files/000/dataset_6.dat
=================================
Although the error messages are different, it might be related?
Cheers,
Jelle
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/