June 2015 - galaxy-dev - lists.galaxyproject.org

Object-Store, setting filetypes crashes Galaxy
by bjoern.gruening＠googlemail.com 21 Jul '15

21 Jul '15

Hi, I have configured to use the hierarchical object store but as soon as I try to reset the filetpye of a dataset Galaxy is crashing with: galaxy.objectstore DEBUG 2014-06-12 14:39:21,180 Using preferred backend 'files3' for creation of MetadataFile 5963 132.230.153.57 - - [12/Jun/2014:14:39:20 +0200] "POST /datasets/966f24627ef70c12/edit HTTP/1.1" 500 - " http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit" "Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Fire fox/29.0" Error - <type 'exceptions.OSError'>: [Errno 2] No such file or directory: 'database/tmp/metadata_temp_file_1xnGcE' URL: http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/usr/local/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/dataset.py', line 295 in edit trans.app.datatypes_registry.set_external_metadata_tool.tool_action.execute( trans.app.datatypes_registry.set_external_metadata_tool, trans, incoming = { 'input1':data }, overwrite = False ) #overwrite is False as per existi ng behavior File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py', line 18 in execute overwrite, history, job_params ) File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py', line 79 in execute_via_app kwds = { 'overwrite' : overwrite } ) File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py', line 717 in setup_external_metadata shutil.copy( dataset.metadata.get( meta_key, None ).file_name, metadata_temp.file_name ) File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py', line 575 in file_name self._filename = abspath( tempfile.NamedTemporaryFile( dir = self.tmp_dir, prefix = "metadata_temp_file_" ).name ) File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 454 in NamedTemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 235 in _mkstemp_inner fd = _os.open(file, flags, 0600) OSError: [Errno 2] No such file or directory: 'database/tmp/metadata_temp_file_1xnGcE' I have attached my object_store_conf.xml file. Thanks, Bjoern

5 7

Error sorting or uploading Bam File
by julie dubois 18 Jul '15

18 Jul '15

Hi all, My galaxy is the last update and I work with samtools 1.2. I have a bug with the sorting and the upload of bam file. This bug appears only when I try to upload bam sorted by name or when I try to sort by name. When I work with bam sorted by coordinate, there is no problem. So, when I try to sort a bam file by name, this error appears under the "bug icon" : Tool execution generated the following error message: [bam_sort_core] merging from 20 files... *** Error in `python': double free or corruption (!prev): 0x0000000002897990 *** Aborted (core dumped) The tool produced the following additional output: [E::hts_idx_push] chromosome blocks not continuous Galaxy attempted to build the BAM index with samtools 1.0+ but failed: [Errno 2] No such file or directory: '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai' And the same type of error appears when I try to upload bam sorted by name. What I understand is that the bam can be sorted properly (The command line to sort is ok in a terminal with no bug and I can follow this step in the job_working_directory during the run) but Galaxy seems to fail to create a bai. For our analysis pipeline, I think the bai is not crucial, is it possible to avoid this creation of bai during uploading and sorting of bam file by name ? Is there another problem than this creation of bai ? Thanks. Julie

5 4

Download data by data manager with non generic ftp-proxy
by Tobias Zimmermann 17 Jul '15

17 Jul '15

Dear All, I have some problems with the download of genome files by the data manager. I tried to download some genome files from the UCSC server with the data manager of galaxy ( name: data_manager_fetch_genome_all_fasta, revision: cca219f2b212), but it fails because of a timeout. Our galaxy server is in an internal network without connection to the Internet. To access the Internet we are required to go trough a http-proxy before any web-services can be accessed. The file /shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/data_manager_fetch_genome_all_fasta/data_manager/data_manager_fetch_genome_all_fasta.py manages the download of the file. Line 183 shows the problem. Here the FTP item from the python ftplib module tries to establishes the connection to the UCSC ftp server. This connection fails. So here is my question: How can i download data by the data manager with a http-proxy? Regards Tobias

2 1

problem with view of list collection in history view
by Alexander Vowinkel 09 Jul '15

09 Jul '15

Hi Team, my tool creates dynamically 96 datasets bundled into a list. In the history I can see the number 96 in the top as hidden datasets (6 shown, 96 hidden) When I open the list, I just can see 64 items. Now I run the job again and I have 96 more hidden items. I open the new list and can see 66 items in that new list. What is going on here? Is that "just" a visual bug? Or are my datasets affected? Thanks, Alexander PS: I use postgres

2 2

Parallelism using metadata
by Marco Albuquerque 06 Jul '15

06 Jul '15

Hello Galaxy Dev, I have a question regarding parallelism on a BAM file. I have currently implemented 3 split options for the BAM datatype 1) by_rname -> splits the bam into files based on the chromosome 2) by_interval -> splits the bam into files based on a defined bp length, and does so across the entire genome present in the BAM file 3) by_read -> splits the bam into files based on the number of reads encountered (if multiple files, all other files match the interval as the first) Now, as you can imagine, reading and writing large BAM files is a pain, and I personally think this is not the best solution for Galaxy. What I was hoping to implement (but don't know how) is to create a new metadata option in bam (bam.metadata.bam_interval) which would generate the interval without creating a new file (essentially, I would create a symbolic link to the old large file, and then update the metadata.bam_interval, this would contain some string of the form chrom:start-end which could then be used in a variety of tools which accept an interval as an option (for example samtools view)) This would be far more efficient then my first implementation, but the thing I don't know how to do is specify some kind of metadata at the split level. I was hoping maybe you could direct me to an example that does this? I have added the following to my metadata.py file: class IntervalParameter( MetadataParamter ) def __init__( self, spec ): MetadataParamter.__init__( self, spec ): self.rname = self.spec.get( "rname" ) self.start = self.spec.get( "start" ) self.end = self.spec.get( "end" ) def to_string(self): if self.rname = 'all': return '' else: return ''.join([self.rname, ':', self.start, '-', self.end]) And the following to my binary.py file: ### UNDER THE BAM CLASS MetadataElement( name="bam_interval", desc="BAM Interval", param=metadata.IntervalParameter, rname="all", start="", end="", visible=False, optional=True) I somehow want rname="all" to be the default, but upon parallelism, I want to be able to adjust this parameter in the split functions. So, <parallelism method="multi" split_inputs="normal,tumour" split_mode="by_interval" split_size="50000000" merge_outputs="output"/> Would actually change the metadata of each file, and not create sub-bams. PLEASE HELP!!! Marco

3 4

cloudman persist filesystem error
by Alexander Vowinkel 02 Jul '15

02 Jul '15

Hi, from IRC this question/bug report: (18:06:13) avowinkel: some cloudman guy here? enis? (18:06:30) avowinkel: AttributeError: 'NoneType' object has no attribute 'id' (18:06:50) avowinkel: when persisting galaxyIndices (18:07:43) avowinkel: snapshot is created, but it crashes when ... somewhen well after that (18:11:08) avowinkel: http://pastebin.com/SDp1p96K (18:11:10) mrscribe: Title: paster.log - Pastebin.com (at pastebin.com) Best, Alexander

2 2

output filter bugs
by Alexander Vowinkel 30 Jun '15

30 Jun '15

Hi, I have a tool with filtered outputs (<data [..]><filter>[...]</data>). A) As I understood the filters are pure python. But if I put it in "<![CDATA[", it doesn't work. I think this should work even better ;) B) The workflow generator doesn't interpret the filters. When I put the tool in a workflow, all possible outputs are shown. This doesn't change when I change an option that would change the output choice. Can someone fix this? ;) Best, Alexander

2 2

Job still waiting - Command Line None - SGE
by Gildas Le Corguille 30 Jun '15

30 Jun '15

Hi, We have a problem on our production instances since one or two years (never found the time to write it and we still hope to find by ourself). Some jobs stay in "waiting" status (grey) indefinitely. Job ID User Last Update Tool State Inputs Command Line Job Runner PID/Cluster ID 63094 foo(a)bar.fr 10 hours ago toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.0.4 new 123250 setting_metadata None None None 63093 foo(a)bar.fr 10 hours ago toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.0.4 new 123249 setting_metadata None None None 63088 foo(a)bar.fr 14 hours ago testtoolshed.g2.bx.psu.edu/repos/jjohnson/trinityrnaseq/trinityrnaseq_norm/… new 123250 setting_metadata, 123249 setting_metadata None None None Our scheduler is SGE. We tried to found the origin of this issue : - there is not relevant clues in the log files - sometimes, a restart make start those jobs - sometimes, kill a job make the others start - sometimes, there is no choice than resubmit Is anyone can propose an idea to fix this behaviour ? Thanks Gildas

2 2

planemo shed_update - error
by Alexander Vowinkel 29 Jun '15

29 Jun '15

Hi, I'm running: planemo shed_update --check_diff --shed_target toolshed and get a python error message (see below). shed_diff and shed_lint are working fine. planemo, version 0.12.2 Best, Alexander Traceback (most recent call last): > File "/usr/local/bin/planemo", line 11, in <module> > sys.exit(planemo()) > File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 664, > in __call__ > return self.main(*args, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 644, > in main > rv = self.invoke(ctx) > File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 991, > in invoke > return _process_result(sub_ctx.command.invoke(sub_ctx)) > File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 837, > in invoke > return ctx.invoke(self.callback, **ctx.params) > File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 464, > in invoke > return callback(*args, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line > 64, in new_func > return ctx.invoke(f, obj, *args[1:], **kwargs) > File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 464, > in invoke > return callback(*args, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/planemo/commands/cmd_shed_update.py", > line 52, in cli > tsi = shed.tool_shed_client(ctx, **kwds) > File "/usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py", > line 331, in tool_shed_client > key = prop("key") > File "/usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py", > line 323, in prop > return kwds.get("shed_%s" % key, None) or shed_config.get(key, None) > AttributeError: 'NoneType' object has no attribute 'get'

3 6

Galaxy API to upload files to data library
by Ryan G 29 Jun '15

29 Jun '15

Hi all - I'm trying to figure out how to script the ability to upload data (fastq) files to Galaxy via the API. I can get the Library's root_folder_id, but am unsure of what the API call is to upload a file. I'm constructing this dictionary: data = {} data['folder_id'] = galaxyFolderId data['create_type'] = 'file' data['file_type'] = 'fastq' data['upload_option'] = 'upload_paths' data['filesystem_paths'] = fileToUpload data['link_data_only'] = 'link_to_files' I assume this is the correct format based on other code I've seen, but I'm not sure what the actual submit call should be. 'api/libraries/' or 'api/folders'????

2 7