April 2015 - galaxy-dev - lists.galaxyproject.org

anybody seen this class of error before?
by Langhorst, Brad 20 Apr '15

20 Apr '15

InvalidRequestError: One or more mappers failed to initialize - can't proceed with initialization of other mappers. Original exception was: Class 'kombu.transport.sqlalchemy.Message' is not mapped galaxy.jobs.handler ERROR 2015-04-19 14:46:00,794 Exception in monitor_step Traceback (most recent call last): File "/mnt/galaxy/data/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 177, in __monitor self.__monitor_step() File "/mnt/galaxy/data/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 229, in __monitor_step jobs_to_check = self.sa_session.query(model.Job).enable_eagerloads(False) \ File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/scoping.py", line 150, in do return getattr(self.registry(), name)(*args, **kwargs) File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py", line 1165, in query return self._query_cls(entities, self, **kwargs) File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 108, in __init__ self._set_entities(entities) File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 118, in _set_entities self._set_entity_selectables(self._entities) File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 151, in _set_entity_selectables ent.setup_entity(*d[entity]) File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 3036, in setup_entity self._with_polymorphic = ext_info.with_polymorphic_mappers File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/mapper.py", line 1877, in _with_polymorphic_mappers configure_mappers() File "/mnt/galaxy/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/mapper.py", line 2586, in configure_mappers raise e Once this error occurs, no more jobs execute on the cluster. Restarting the handler seems to get things moving again, but I’d like to find the source of the failure. Any ideas? Brad -- Brad Langhorst, Ph.D. Applications and Product Development Scientist

3 2

Error generating Picard indexes when using data_manager_gatk_picard_index_builder from the main tool shed
by Paulo Silva 17 Apr '15

17 Apr '15

Hello all Galaxy devs, I’ve come across (what seems to be) a Galaxy bug (or rather, tool-shed bug) when I run the data manager data_manager_gatk_picard_index_builder (rev. a15709ad26ea) from the Galaxy main tool shed repo on a clean Galaxy install. Installation goes fine - when I install this data manager, all dependencies are correctly installed: package_picard_1_56_0 (rev. 99a28567c3a3) and package_samtools_0_1_18 (rev. 171cd8bc208d). After installation, I go to Admin -> Manage local data (beta) -> Run Data Manager Tools, click “Generate GATK-sorted Picard indexes” and the data manager form is displayed. I select the reference genome (either hg19 or hg38 - installed with data_manager_fetch_genome_dbkeys_all_fasta from the main tool shed, rev. bca4c608408c - working fine with other tools including bowtie2 and trackster), enter “hg19” both “Name of sequence” and “ID for sequence” fields, and click “Execute”. The indexing job is added to the queue, but when I go to view data manager jobs for this data manager, I see it stopped with the following error: Traceback (most recent call last): File "/home/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea/data_manager_gatk_picard_index_builder/data_manager/data_manager_gatk_picard_index_builder.py", line 179, in <module> if __name__ == "__main__": main() File "/home/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea/data_manager_gatk_picard_index_builder/data_manager/data_manager_gatk_picard_index_builder.py", line 162, in main raise Exception( '"%s" is not a valid dbkey. You must specify a valid dbkey.' % ( options.fasta_dbkey ) ) Exception: "None" is not a valid dbkey. You must specify a valid dbkey. Inspecting galaxy.log doesn’t reveal anything relevant (at least for me - which is why I need your help!): galaxy.jobs DEBUG 2015-04-17 11:13:23,031 (63) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/63 galaxy.jobs.handler DEBUG 2015-04-17 11:13:23,044 (63) Dispatching to local runner 192.168.2.74 - - [17/Apr/2015:11:13:22 +0100] "POST /api/tools HTTP/1.1" 200 - "http://192.168.2.221:8080/tool_runner?tool_id=toolshed.g2.bx.psu.edu%2Frepo…" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4 Safari/600.4.10" galaxy.jobs DEBUG 2015-04-17 11:13:23,168 (63) Persisting job destination (destination id: local) galaxy.jobs.handler INFO 2015-04-17 11:13:23,212 (63) Job dispatched galaxy.tools.deps DEBUG 2015-04-17 11:13:23,393 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2015-04-17 11:13:23,395 Building dependency shell command for dependency 'picard' galaxy.jobs.runners DEBUG 2015-04-17 11:13:23,475 (63) command is: PACKAGE_BASE=/home/galaxy/tool-deps/samtools/0.1.18/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea; export PACKAGE_BASE; . /home/galaxy/tool-deps/samtools/0.1.18/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea/env.sh; PACKAGE_BASE=/home/galaxy/tool-deps/picard/1.56.0/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea; export PACKAGE_BASE; . /home/galaxy/tool-deps/picard/1.56.0/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea/env.sh; python /home/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_gatk_picard_index_builder/a15709ad26ea/data_manager_gatk_picard_index_builder/data_manager/data_manager_gatk_picard_index_builder.py "/home/galaxy/galaxy-dist/database/files/000/dataset_70.dat" \ --jar "$JAVA_JAR_PATH/CreateSequenceDictionary.jar" \ --fasta_filename "/home/galaxy/galaxy-dist/tool-data/hg19/seq/hg19.fa" \ --fasta_dbkey "hg19" \ --fasta_description "Human, Feb. 2009, GRCh37/hg19" \ --data_table_name "gatk_picard_indexes"; return_code=$?; python /home/galaxy/galaxy-dist/database/job_working_directory/000/63/set_metadata_Ts4xSd.py /home/galaxy/galaxy-dist/database/tmp/tmpfomc7m /home/galaxy/galaxy-dist/database/job_working_directory/000/63/galaxy.json /home/galaxy/galaxy-dist/database/job_working_directory/000/63/metadata_in_HistoryDatasetAssociation_70_Lkhynl,/home/galaxy/galaxy-dist/database/job_working_directory/000/63/metadata_kwds_HistoryDatasetAssociation_70_IuPBQs,/home/galaxy/galaxy-dist/database/job_working_directory/000/63/metadata_out_HistoryDatasetAssociation_70_uuRnIn,/home/galaxy/galaxy-dist/database/job_working_directory/000/63/metadata_results_HistoryDatasetAssociation_70_aVp4XH,/home/galaxy/galaxy-dist/database/files/000/dataset_70.dat,/home/galaxy/galaxy-dist/database/job_working_directory/000/63/metadata_override_HistoryDatasetAssociation_70_hXaBQB; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-04-17 11:13:23,476 (63) executing job script: /home/galaxy/galaxy-dist/database/job_working_directory/000/63/galaxy_63.sh galaxy.jobs DEBUG 2015-04-17 11:13:23,526 (63) Persisting job destination (destination id: local) galaxy.jobs.runners.local DEBUG 2015-04-17 11:13:24,976 execution finished: /home/galaxy/galaxy-dist/database/job_working_directory/000/63/galaxy_63.sh galaxy.jobs DEBUG 2015-04-17 11:13:25,054 setting dataset state to ERROR galaxy.jobs DEBUG 2015-04-17 11:13:25,122 job 63 ended galaxy.datatypes.metadata DEBUG 2015-04-17 11:13:25,123 Cleaning up external metadata files Some more system details: Galaxy version: latest_15.03, running on Ubuntu 14.04.2 LTS Java: 1.8.0_45-b14 from the webupd8 Ubuntu repo. Using Postresql locally, everything else local too. Cloned from the Bitbucket repo (in case this matters). Can anyone shed some light on this? In the traceback log above it seems options.fasta_dbkey is being set to “None” rather than “hg19” but I am not sure how to solve this, or if this is indeed the problem. Thank you in advance! All the best, -- Paulo ____________________________________________________________ Paulo Filipe Silva | PhD, Biology | MSc, Bioinformatics Bioinformatician @ CGPP - Centre for Predictive and Preventive Genetics http://www.cgpp.eu <http://www.cgpp.eu/> IBMC - Institute for Molecular and Cell Biology Rua do Campo Alegre, 823 4150-180 Porto, Portugal E-mail: paulo.silva(a)ibmc.up.pt <mailto:paulo.silva@ibmc.up.pt> Telephone: +351 22 607 49 42 ext 2811 Fax: +351 22 600 29 23

1 0

metadata in parallelization
by Roberto Alonso CIPF 17 Apr '15

17 Apr '15

Hello, I am writing some code to enable parallelization for some tool wrappers. First, I did it for simple bwa wrapper, but now I am modifying toolshed.g2.bx.psu.edu/repos/devteam/bwa/c71dd035971e/bwa/bwa-mem.xml to check if the code would work with this wrapper. So, I wrote some code that I thing was necessary in order to merge some bam and I added the parallelism tag (in bold) to the config file: <tool id="bwa_mem" name="BWA-MEM" version="0.1"> <macros> <import>bwa_macros.xml</import> </macros> <requirements> <requirement type="package" version="0.7.10.039ea20639">bwa</requirement> <requirement type="package" version="1.1">samtools</requirement> </requirements> <description>- map medium and long reads (> 100 bp) against reference genome</description> <parallelism method="multi" split_size="3" shared_inputs="ref_file" split_mode="number_of_parts" merge_outputs="bam_output" split_inputs="fastq_input1,fastq_input2" ></parallelism> <command> ... So, everything works well, and the resulting bam from parallelization mode and without the parallelization mode is the same but the Galaxy log throws an error regarding metadata, it says something like this: galaxy.jobs.splitters.multi DEBUG 2015-04-17 09:54:58,335 merge finished: /home/ralonso/galaxy/database/files/000/dataset_198.dat galaxy.jobs.runners.tasks DEBUG 2015-04-17 09:54:58,473 executing external set_meta script for job 200: python /home/ralonso/galaxy/database/tmp/set_metadata_E5fGIE.py /home/ralonso/galaxy/database/tmp/tmpHS8Byo /home/ralonso/galaxy/database/job_working_directory/000/200/galaxy.json /home/ralonso/galaxy/database/tmp/metadata_in_HistoryDatasetAssociation_198_yOGiQG,/home/ralonso/galaxy/database/tmp/metadata_kwds_HistoryDatasetAssociation_198_nAsQoq,/home/ralonso/galaxy/database/tmp/metadata_out_HistoryDatasetAssociation_198_I_cLs4,/home/ralonso/galaxy/database/tmp/metadata_results_HistoryDatasetAssociation_198_qhjzoV,/home/ralonso/galaxy/database/files/000/dataset_198.dat,/home/ralonso/galaxy/database/tmp/metadata_override_HistoryDatasetAssociation_198_ScKLqH Traceback (most recent call last): File "/home/ralonso/galaxy/database/tmp/set_metadata_E5fGIE.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() ImportError: No module named galaxy_ext.metadata.set_metadata galaxy.jobs.runners.tasks DEBUG 2015-04-17 09:54:58,624 execution of external set_meta finished for job 200 galaxy.datatypes.metadata DEBUG 2015-04-17 09:54:58,714 setting metadata externally failed for HistoryDatasetAssociation 198: External set_meta() not called When using no parallelization mode, there is no problem, also because Galaxy doesn't go through this part of code, I mean it doesn't execute this. I see that Galaxy have to do something with metada attributes, but what is t trying to do? is there any way to solve this? Thank you very much Regards, -- Roberto Alonso Functional Genomics Unit Bioinformatics and Genomics Department Prince Felipe Research Center (CIPF) C./Eduardo Primo Yúfera (Científic), nº 3 (junto Oceanografico) 46012 Valencia, Spain Tel: +34 963289680 Ext. 1021 Fax: +34 963289574 E-Mail: ralonso(a)cipf.es

1 0

Difference between GitHub and Mercurial Galaxy download...
by Asma Riyaz 16 Apr '15

16 Apr '15

Hi, Is there a difference between the github and mercurial galaxy downloads? Is there one preferred over another? The galaxy website asks to do a Github download and many other places including the GMOD tutorial held last summer on Galaxy suggested a mercurial one? Kindly advice. Thank you, Asma

3 2

substitution of special characters in arguments of galaxy tools
by Wang, Yu 16 Apr '15

16 Apr '15

Dear galaxy developers, Galaxy seems to substitute special characters when it passes arguments to its tools (e.g. the content of input fields of type 'text'). In general, this is useful. However, '%' is substituted by 'X', i.e. the substitution is not invertible which can be a real show stopper. Is there some way to change this substitution or switch it of completely? Cheers, Yu di29her wang(a)lrz.de<mailto:wang@lrz.de>

2 1

Older galaxy I'm keeping alive by drip feed, has problem restarting when I turn debug off.
by Dooley, Damion 16 Apr '15

16 Apr '15

I have an OLDER Galaxy install I was going to fade out. In meantime I saw that its debug flag should be turned off, so I did. But when it restarted it complained in the history tab with a server error. I put debug back into config and it runs fine. I haven't changed anything recently on it. Wondering if there is a simple remedy? Details below. Thanks, Damion ********************************************************* Server Error An error occurred. See the error logs for more information. (Turn debug on to display exception reports here) And indeed, after the startup log message... serving on http://127.0.0.1:8080 There is an error that seems to be triggered when A GET /galaxy/history command is sent. ................ - - [15/Apr/2015:13:03:43 -0700] "GET /galaxy/history HTTP/1.1" 200 - "http://salk.bccdc.med.ubc.ca/galaxy/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36" ................ - - [15/Apr/2015:13:03:43 -0700] "GET /galaxy/history HTTP/1.1" 500 - "http://salk.bccdc.med.ubc.ca/galaxy/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36" Error - <type 'exceptions.TypeError'>: 'NoneType' object is not callable ................ - - [15/Apr/2015:13:03:43 -0700] "GET /galaxy/root/tool_menu HTTP/1.1" 200 - "http://salk.bccdc.med.ubc.ca/galaxy/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36" URL: http://salk.bccdc.med.ubc.ca/galaxy/history File '/usr/lib/python2.6/site-packages/paste/exceptions/errormiddleware.py', line 144 in __call__ app_iter = self.application(environ, sr_checker) File '/usr/lib/python2.6/site-packages/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/usr/lib/python2.6/site-packages/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/usr/local/galaxy/production/galaxy-dist/lib/galaxy/web/framework/base.py', line 169 in __call__ return body( environ, start_response ) File '/usr/local/galaxy/production/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 759 in render template.render_context( context ) File '/usr/local/galaxy/production/galaxy-dist/eggs/Mako-0.4.1-py2.6.egg/mako/template.py', line 319 in render_context **kwargs) File '/usr/local/galaxy/production/galaxy-dist/eggs/Mako-0.4.1-py2.6.egg/mako/runtime.py', line 692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/usr/local/galaxy/production/galaxy-dist/eggs/Mako-0.4.1-py2.6.egg/mako/runtime.py', line 718 in _exec_template callable_(context, *args, **kwargs) File '/usr/local/galaxy/production/galaxy-dist/database/compiled_templates/root/history.mako.py', line 61 in render_body __M_writer(u'\n\n') File '/usr/local/galaxy/production/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 756 in write response_write( d.encode( 'utf-8' ) ) TypeError: 'NoneType' object is not callable Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

3 6

Using synchronous method to upload multiple files
by Daniele P Colobraro 16 Apr '15

16 Apr '15

Hi all, I've set up a simple tool using the synchronous method to upload a file on galaxy. I'm able to visit the external link, select and upload a file by the list. The script that I use is data_source.py and the 'command' part in the xml configuration is the same as the ucsc xml config (ucsc_tablebrowser.xml). But my purpose is to import multiple files. Have you any idea how I can set a way to import/upload multiple files by using the synchronous method? Thank in advance, Best regards Daniele

3 5

April GalaxyAdmins Meetup: April 16
by Dave Clements 16 Apr '15

16 Apr '15

Hello all, The April 2015 GalaxyAdmins meetup <https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_04_16> will be held online on Thursday, 16 April (a week from today). We'll meet at 5pm Central European Summer Time (see your local time <http://bit.ly/1EFUf0M>). Carrie Ganote <http://www.researchgate.net/profile/Carrie_Ganote> from the National Center for Genome Analysis Support (NCGAS) <http://ncgas.org/> and Pervasive Technology Institute <http://pti.iu.edu/> at Indiana University will talk about her Galaxy work with Trinity, IU Galaxy, and the Open Science Grid. And Sam Guerler <https://wiki.galaxyproject.org/guerler> of the Galaxy Team will also discuss recent work to update Galaxy's tool forms. GalaxyAdmins <https://wiki.galaxyproject.org/Community/GalaxyAdmins> is a discussion group for Galaxy community members who are responsible for Galaxy installations. Please see the meetup page ( http://bit.ly/gxyadm201504) for more details. Hope to see (hear?) you there, Dave C (channeling Hans-Rudolf Hotz) -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 2

Geting integer value from InputValueWrapper in plugin xml file
by Christian Brenninkmeijer 16 Apr '15

16 Apr '15

Hi, I am a new galaxy developer playing with writing my first plugins In my plugin xml configaration file: I want to be able to increase the value of one parameter based on another using <command interpreter="python"> #if $rows.option =="header_ignore" #set $data_start = $rows.ignore + 1 but I get: unsupported operand type(s) for +: 'InputValueWrapper' and 'int' A hack workaround I found was: #set $data_start = int(str($rows.ignore)) + 1 I just wondered if there was a cleaner way to get the integer value from a InputValueWrapper object. Which is set to integer by: <param name="ignore" size="4" type="integer" value="1" .... If documentation exist somwere please send me an RTFM link. (I didn't find it in https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax) Thanks in advance Christian Brenninkmeijer University of Manchester Ps. If you want to see exactly what I am playing with see https://github.com/Christian-B/galaxy_tools/tree/master/summary (Note that is subject to change as I learn)

3 2

pushing toolshed directories to our own repo
by Nikhil Joshi 16 Apr '15

16 Apr '15

Hi all, So I am trying to push our toolshed directories to our own repo and I am running into problems. Basically, since the toolshed tools are subrepos, I can't figure out how to push those changes into *our* repo. Also, I have had to change some of the toolshed tools because they don't work properly with our galaxy. So, when I try to do the push, it aborts giving me this error: abort: push creates new remote head b9bd905700fe! (in subrepo toolshed.g2.bx.psu.edu/repos/devteam/tophat2/da1f39fe14bc/tophat2) And, of course, I can't push to that repo, nor do I want to. All I want to do is to push our shed tools directory to our repo, but at the same time keep the mercurial information so that when we clone our repo into a new galaxy install, the toolshed will still work from galaxy. How can I do this? Am I even thinking about this the right way? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 1

question about error output
by Roberto Alonso CIPF 16 Apr '15

16 Apr '15

Hello, I am defining some tools for my own galaxy installation and I would like to know if there is a more elegant solution for redirecting error output than the one in the next config file: <tool id="gatk_mio" name="calling with gatk"> <description>calling with gatk</description> <command> ln -s "${input}" input.bam ; samtools index input.bam; java -jar /home/ralonso/software/GenomeAnalysisTK-3.3-0/GenomeAnalysisTK.jar -T UnifiedGenotyper -R /home/ralonso/BiB/Galaxy/data/Cclementina_v1.0_scaffolds.fa -I input.bam -o $output *2>/dev/null;* </command> <inputs> <param format="bam" name="input" type="data" label="bam"/> </inputs> <outputs> <data format="vcf" name="output" /> </outputs> <help> bwa </help> </tool> Best regards many thanks -- Roberto Alonso Functional Genomics Unit Bioinformatics and Genomics Department Prince Felipe Research Center (CIPF) C./Eduardo Primo Yúfera (Científic), nº 3 (junto Oceanografico) 46012 Valencia, Spain Tel: +34 963289680 Ext. 1021 Fax: +34 963289574 E-Mail: ralonso(a)cipf.es

2 1

$REMOTE_USER
by Wang, Yu 16 Apr '15

16 Apr '15

Hi, Is there a way to query the actual name of the user, who is logged into galaxy and submitted the workflow ($REMOTE_USER), from inside a galaxy tool when galaxy is configured for external authentication using an apache server? It seems like galaxy passes this variable to scripts like 'drmaa_external_runner.py' but I could not find it in the environment of a tool which has been started via e.g. python ( using <command interpreter="python"> in the xml-file of the tool)." Cheers, Yu di29her wang(a)lrz.de<mailto:wang@lrz.de> On 13 Apr 2015, at 18:00, galaxy-dev-request(a)lists.galaxyproject.org<mailto:galaxy-dev-request@lists.galaxyproject.org> wrote: Send galaxy-dev mailing list submissions to galaxy-dev(a)lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> To subscribe or unsubscribe via the World Wide Web, visit https://lists.galaxyproject.org/listinfo/galaxy-dev or, via email, send a message with subject or body 'help' to galaxy-dev-request(a)lists.galaxyproject.org You can reach the person managing the list at galaxy-dev-owner(a)lists.galaxyproject.org When replying, please edit your Subject line so it is more specific than "Re: Contents of galaxy-dev digest..." HEY! This is important! If you reply to a thread in a digest, please 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. 2. Strip out everything else in the digest that is not part of the thread you are responding to. Why? 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative. Today's Topics: 1. Re: Strange repository dependency behavior with toolshed-installed naive_variant_caller (Martin Čech) ---------------------------------------------------------------------- Message: 1 Date: Mon, 13 Apr 2015 14:27:10 +0000 From: Martin Čech <marten(a)bx.psu.edu> To: Brian Claywell <bclaywel(a)fredhutch.org> Cc: galaxy-dev(a)lists.galaxyproject.org Subject: Re: [galaxy-dev] Strange repository dependency behavior with toolshed-installed naive_variant_caller Message-ID: <CAHKBquFw66cXyss7KjExM4-mpdK0P_TLT_xtCOBEK4pLs3OYOg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello Brian, we have made some changes in the past month that might address the issue you are having. Are you still encountering problems? If so could you please try re-installing the tools again? thank you Martin On Tue, Feb 24, 2015 at 5:15 PM Brian Claywell <bclaywel(a)fredhutch.org> wrote: Hi Martin, Thanks for the prompt reply! My responses are inline below. On Tue, Feb 24, 2015 at 8:26 AM, Martin Čech <marten(a)bx.psu.edu> wrote: Are you using Admin UI for installation or API scripts? The admin UI exclusively. Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these? Yes -- the iuc package_numpy_1_7 revision a4bdc17eed4a is listed twice, with both showing the green "Installed" status. Interestingly the installed repo page shows that they're installed in different locations: (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/ef12a3a11d5b/package_num… (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/a4bdc17eed4a/package_num… At one point during the troubleshooting steps I described in my original email, package_pybamparser_0_0_1 and package_pybamtools_0_0_1 showed up in the same way, with two records each having the same revision and a green "Installed" status. If you check on the filesystem in the {{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication? I didn't see anything that struck me as out of the ordinary. Here's a gist of the directory structures for the affected packages under both shed_tools and tool_deps: https://gist.github.com/bcclaywell/23c0b274f2d4bd0d659f How big is your Galaxy in terms of installed tools, complexity of setup and traffic? I don't think it's terribly big -- we have a couple of large tool suites like BEDTools, the gops interval operations suites, GATK 1 and 2, etc, and then maybe 15-20 other tools and their dependencies? Our setup isn't very complex, either; Galaxy is running with one web process and one handler process behind an nginx reverse proxy, all running on a single machine. Jobs are submitted to our slurm cluster via the SSH CLI runner. Traffic is pretty low -- we're just now rolling out of beta. Cheers, Brian -- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150413/6…> ------------------------------ Subject: Digest Footer _______________________________________________ galaxy-dev mailing list galaxy-dev(a)lists.galaxyproject.org https://lists.galaxyproject.org/listinfo/galaxy-dev To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ------------------------------ End of galaxy-dev Digest, Vol 106, Issue 12 *******************************************

2 2

Automated Tool Tests w/ Data Tables
by Jeremy Liu 15 Apr '15

15 Apr '15

Galaxy Devs, I've been following this documentation to write auto tests for my tools: https://wiki.galaxyproject.org/Admin/Tools/WritingTests?action=show&redirec… Straightforward enough, but I am not sure how to incorporate input parameters from data tables into this. My tools currently use as input parameters entries/files from a data table. These files were installed using a data manager. Ideally, the tests should also reference the same data table files as the tools do. How can I do this? Thanks, Jeremy Liu jeremy.liu(a)yale.edu

3 2

problems wit upgrading to "latest_15.03" - 'UniverseApplication' object has no attribute 'control_worker'
by Hans-Rudolf Hotz 14 Apr '15

14 Apr '15

Hi I am in the process of updating our development server (from "latest_2014.10.06", including the security fix from January to "latest_15.03" ). and when I restart the server, I get the following error: ... galaxy.jobs.handler INFO 2015-04-14 15:46:05,180 sending stop signal to worker thread galaxy.jobs.handler INFO 2015-04-14 15:46:05,181 job handler stop queue stopped Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/path_to_galaxy/galaxy_dist/lib/galaxy/app.py", line 167, in shutdown if self.control_worker: AttributeError: 'UniverseApplication' object has no attribute 'control_worker' Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/path_to_galaxy/galaxy_dist/lib/galaxy/app.py", line 167, in shutdown if self.control_worker: AttributeError: 'UniverseApplication' object has no attribute 'control_worker' Any help appreciated Thank you very much Hans-Rudolf -- Hans-Rudolf Hotz, PhD Bioinformatics Support Friedrich Miescher Institute for Biomedical Research Maulbeerstrasse 66 4058 Basel/Switzerland

2 2

problem with revision updates for repositories.
by Anna Terry 14 Apr '15

14 Apr '15

Hi, I have a few installed tool shed repositories which are showing that updates are available. When I click get repository updates, instead of the usual upate page, I get a page that simply says Found The resource was found at http://toolshed.g2.bx.psu.edu/repository/check_for_updates?galaxy_url=https…; you should be redirected automatically. This is for bam_to_sam, but same problem for blast_datatypes, pileup_interval and sam_to_bam too. Anyone have any ideas why I'm seeing this? Cheers Anna

2 9

MySQL InnoDB - Execution Order
by McCully, Dwayne (NIH/NIAMS) [C] 10 Apr '15

10 Apr '15

Hello Everyone, The running Galaxy in a production environment documentation states: "If you are using MySQL<http://dev.mysql.com/> with MyISAM<http://dev.mysql.com/doc/refman/5.1/en/myisam-storage-engine.html> table engine when Galaxy is in multiprocess configuration, workflow steps will get executed out of order<http://dev.list.galaxyproject.org/Job-execution-order-mixed-up-tt4662488.ht…> and fail. Use InnoDB<http://dev.mysql.com/doc/refman/5.1/en/innodb-storage-engine.html> engine instead or switch to PostgreSQL<http://www.postgresql.org/>." I've enabled InnoDB for my MySQL database via the my.cnf file but the default database and database to log all database transactions were created with MyISAM files. If this is normal, when will Galaxy create InnoDB tables? I don't believe you can create a database with InnoDB. Dwayne

2 1

Visualization in anonymous mode ?
by Loraine Guéguen 09 Apr '15

09 Apr '15

Hi, I'd like to use the visualization in anonymous mode, but the visualize icon on the dataset is not available in anonymous mode (although it works fine when connected). Is it possible to activate the dataset visualization in anonymous mode ? Thanks, Loraine -- Loraine Guéguen Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/

2 5

Tests not being run on toolsheds?
by Peter Briggs 09 Apr '15

09 Apr '15

Hello Apologies as this seems to be quite a frequent question (and I may have missed something) however: are the tool tests being executed on the main or test toolsheds at the moment? The most recent tests results that I can find for any of the tools I've uploaded is the 29th January. I've also looked at a couple of tools from other users and the situation appears to be the same (at least in those cases). Thanks for your help! Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

3 9

Problems after upgrade
by C. Ch. 09 Apr '15

09 Apr '15

Hi all, We had a problem on the NAS where we couldn't write anymore. This problem was solved and I decided that I would profit to upgrade galaxy. Now we have a problem where jobs do not get launched and the handler log shows a lot of error messages. The first error message is copied below. Afterwards it alternates between galaxy.jobs.handler ERROR 2015-04-07 15:58:24,959 Exception in monitor_step galaxy.jobs.runners ERROR 2015-04-07 15:58:23,542 Unhandled exception checking active jobs with, mainly, the same error stack (second error message shown below). Any idea what is going wrong? My first impression is that it is trying to recuperate some job that galaxy believes is still running but the machine has been stopped and rebooted in the mean time. Any help is welcome! Cristian galaxy.jobs.runners ERROR 2015-04-07 15:44:04,809 (27764) Unhandled exception calling queue_jobTraceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 97, in run_next method(arg) File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 115, in queue_job if not self.prepare_job( job_wrapper, include_metadata=True ): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 140, in prepare_job job_id = job_wrapper.get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 794, in get_id_tag return self.get_job().get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 790, in get_job return self.sa_session.query( model.Job ).get( self.job_id ) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 840, in get return loading.load_on_ident(self, key) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/loading.py", line 231, in load_on_ident return q.one() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2395, in one ret = list(self) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2434, in __iter__ context = self._compile_context() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2810, in _compile_context entity.setup_context(self, context) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 3194, in setup_context column_collection=context.primary_columns File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/interfaces.py", line 466, in setup strat.setup_query(context, entity, path, loader, adapter, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py", line 1111, in setup_query column_collection, parentmapper, chained_from_outerjoin File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py", line 1234, in _generate_row_adapter use_mapper_path=True) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/util.py", line 364, in __init__ else mapper.with_polymorphic_mappers, File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 1877, in _with_polymorphic_mappers configure_mappers() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 2589, in configure_mappers mapper._post_configure_properties() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 1694, in _post_configure_properties prop.init() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/interfaces.py", line 144, in init self.do_init() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py", line 1549, in do_init self._process_dependent_arguments() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py", line 1605, in _process_dependent_arguments self.target = self.mapper.mapped_table File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py", line 1522, in mapper argument = self.argument() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/ext/declarative/clsregistry.py", line 283, in __call__ (self.prop.parent, self.arg, n.args[0], self.cls)InvalidRequestError: When initializing mapper Mapper|Queue|kombu_queue, expression 'Message' failed to locate a name ("name 'Message' is not defined"). If this is a class name, consider adding this relationship() to the <class 'kombu.transport.sqlalchemy.Queue'> class after both dependent classes have been defined. Repeated error messages:galaxy.jobs.runners ERROR 2015-04-07 15:58:23,542 Unhandled exception checking active jobsTraceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 501, in monitor self.check_watched_items() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 238, in check_watched_items galaxy_id_tag = ajs.job_wrapper.get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 794, in get_id_tag return self.get_job().get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 790, in get_job return self.sa_session.query( model.Job ).get( self.job_id ) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 150, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1165, in query return self._query_cls(entities, self, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 108,in __init__ self._set_entities(entities) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 118,in _set_entities self._set_entity_selectables(self._entities) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 151,in _set_entity_selectables ent.setup_entity(*d[entity]) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 3036, in setup_entity self._with_polymorphic = ext_info.with_polymorphic_mappers File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 1877, in _with_polymorphic_mappers configure_mappers() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 2586, in configure_mappers raise eInvalidRequestError: One or more mappers failed to initialize - can't proceed with initialization of other mappers. Original exception was: When initializing mapper Mapper|Queue|kombu_queue, expression 'Message' failed to locate a name ("name 'Message' is not defined"). If this is a class name, consider adding this relationship() to the <class 'kombu.transport.sqlalchemy.Queue'> class after both dependent classes have been defined.

2 2

Tool Shed installs: coercing to Unicode: need string or buffer, NoneType found
by Peter Cock 09 Apr '15

09 Apr '15

I just hit a problem installing package_bx_python_0_7 via the Admin GUI. Name: bx-python Version: 0.7.1 Type: package Status: Error Error: Error installing tool dependency bx-python version 0.7.1: coercing to Unicode: need string or buffer, NoneType found Name: galaxy-ops Version: 1.0.0 Type: package Status: Installed Clicked on "Uninstall" for bx-python, told: Uninstall tool dependencies Name: bx-python Version: 0.7.1 Type: package Install directory: This dependency's installation directory does not exist, click <b>Uninstall</b> to reset for installation. Clicked on "Uninstall" again, seemed to work. I went through a series of "repair repositiory" rounds on package_bx_python_0_7 and intersect tool, and it all seems fine now. I'm not sure what the original glitch was, but there are two clear issues here. First, the Unicode error - perhaps previously reported e.g. http://dev.list.galaxyproject.org/How-to-remove-a-broken-toolshed-install-t… http://dev.list.galaxyproject.org/tophat2-install-tc4663736.html Those previous discussions suggest the database and file system were out of sync - but clearly this unexpected None value ought to be better handled. Second, the over-escaped bold in the message, presumably part of: https://trello.com/c/oX6i51Mn/2298-back-off-over-escaping Peter

1 0

Escaped HTML show in import-workflow red message
by Peter Cock 08 Apr '15

08 Apr '15

Hi all, I just imported a workflow and noticed the following glitch with the red message shown to the user about missing tools - the escaped HTML is show to the user (ampersand lt semi-colon etc). Specifically, I imported http://www.myexperiment.org/workflows/4601.html from Bjoern. Using the current v15.03 release, specifically: $ hg log | head changeset: 17050:6395e7035143 tag: tip parent: 17049:708cab19708d parent: 17048:28113eeee212 user: Nate Coraor <nate(a)bx.psu.edu> date: Mon Mar 16 22:50:21 2015 -0400 summary: Merge v15.03.1 to default Peter Red message: Workflow Finding 3 genes close to each other (imported from uploaded file) imported successfully. The workflow requires the following tools that are not available in this Galaxy instance.You can likely install the required tools from one of the Galaxy tool sheds listed below. Tool name Compute, id toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/1.1.0, version 1.1.0 Tool name Intersect, id toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/0.0.1, version 0.0.1 Tool name Intersect, id toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0, version 1.0.0 <a href="http://toolshed.g2.bx.psu.edu/repository/find_tools?galaxy_url=ht… main tool shed</a> <a href="http://testtoolshed.g2.bx.psu.edu/repository/find_tools?galaxy_ur… test tool shed</a>

1 1

Re: [galaxy-dev] a question on data manager and data table
by Daniel Blankenberg 08 Apr '15

08 Apr '15

Hi Rui, (cc’d back to list) One of the nice things about tool Data Tables is that it allows us to have multiple ‘.loc’ files as the source for the content for a single data table — so we can have many ‘all_fasta.loc’ files, but one ‘all_fasta’ tool data table, with merged contents, and a tool working on ‘all_fasta’ data table doesn’t know or care about the actual .loc files on disk. Then any tool that uses the ‘all_fasta’ table in an input parameter can simply call input.fields.path, to have the actual path to the fasta file passed to the tool. > 1.which file should the downloaded fasta data modify? tool_data_table_conf.xml, or shed_tool_data_table_conf.xml? Tools installed from the toolshed that include data Tables will programmatically modify the shed_ file, the other file can is for manual admin changes. > 2. which loc file is the primary one for tools to reference? the one in $GALAXYHOME/tool-data, or the one in $GALAXYHOME/tool-data//toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/? If first one, then should the data manager update that one too(right now it didn't)? Tools shouldn’t be referencing .loc files directly any more, and instead should use the tool data table abstraction. > 3. For now, that tool still uses alignseq.loc, I need to manually update that file, correct? Any tool that makes direct use of a .loc file will need to have that specific file modified. > 4. How to determine if a tool is using data table is to look at the tool xml file to see if it contains reference to a data table, right? And in tool_data_table_conf.xml, it should contain data tables definition for all the possible data, right? a.) Yes, you’ll have to look at the tool.xml, although we could add another section to the ToolShed preview to include this information if it is useful. b.) tool_data_table_conf.xml should contain data tables distributed with the Galaxy distribution and any manually configured data tables. tool_data_table_conf.xml and shed_tool_data_table_conf.xml will be merged into one set of tool data tables — a tool data table does not need to be in tool_data_table_conf.xml if it is in shed_tool_data_table_conf.xml and vis-versa (but can be in both). Thanks for using Galaxy, Dan On Apr 7, 2015, at 12:47 PM, Beginner TI <ruiwang.sz(a)gmail.com> wrote: > Hi Dan, > > Thanks for the reply! > > So, what would be the expected behavior for the tool that uses data table? For example, in my case, the tool_data_table_conf.xml would be modified to reflect the update to all_fasta.loc? > > I'm a little puzzled on how this works. As you could see, after I installed that data manager and downloaded the mm9 fasta data, it modified the file shed_tool_data_table_conf.xml. This should have nothing to do with the tool Extract Genome DNA, am I right? So, my question is, > > 1.which file should the downloaded fasta data modify? tool_data_table_conf.xml, or shed_tool_data_table_conf.xml? > > 2. which loc file is the primary one for tools to reference? the one in $GALAXYHOME/tool-data, or the one in $GALAXYHOME/tool-data//toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/? If first one, then should the data manager update that one too(right now it didn't)? > > 3. For now, that tool still uses alignseq.loc, I need to manually update that file, correct? > > 4. How to determine if a tool is using data table is to look at the tool xml file to see if it contains reference to a data table, right? And in tool_data_table_conf.xml, it should contain data tables definition for all the possible data, right? > > If you could please elaborate a bit, I'll really appreciate! > > Thanks, > Rui > > > > > On Tue, Apr 7, 2015 at 7:15 AM, Daniel Blankenberg <dan(a)bx.psu.edu> wrote: > Hi Rui, > > The Extract Genomic DNA Tool has not yet been updated to work with tool Data Tables. > > Additionally, this tool requires TwoBit formatted files when selecting from built-in data — when selecting a FASTA from your history, the fasta file is converted to TwoBit internally in the tool before extracting sequence chunks. > > There is a TwoBit builder Data Manager under development (https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/da…) that hasn’t made it out to the ToolShed just yet, but it won’t be helpful until the the Extract Genomic DNA Tool is updated: https://trello.com/c/8unnSp7H > > > Thanks for using Galaxy, > > Dan > > > On Apr 7, 2015, at 12:16 AM, Beginner TI <ruiwang.sz(a)gmail.com> wrote: > >> Hi All, >> >> I just made a new local instance, installed the fetch_all_fasta data manager, downloaded mm9 fasta. Then I noticed the following: >> >> in tool_data_table_conf.xml, it has: >> >> <table name="all_fasta" comment_char="#"> >> <columns>value, dbkey, name, path</columns> >> <file path="tool-data/all_fasta.loc" /> >> </table> >> >> however in shed_tool_data_table_conf.xml, it has: >> >> <table comment_char="#" name="all_fasta"> >> <columns>value, dbkey, name, path</columns> >> <file path="/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/all_fasta.loc" /> >> <tool_shed_repository><tool_shed>toolshed.g2.bx.psu.edu</tool_shed><repository_name>data_manager_fetch_genome_all_fasta</repository_name><repository_owner>devteam</repository_owner><installed_changeset_revision>cca219f2b212</installed_changeset_revision></tool_shed_repository></table> >> >> in tool-data, all_fasta.loc is empty, but in the all_fasta.loc of the shed_tool entry, it shows: >> >> mm9 mm9 Mouse July 2007 (NCBI37/mm9) (mm9) /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/mm9/seq/mm9.fa >> >> So if I try "Extract Genomic DNA function", I could see the parameter passed to the command line is >> >> -g "/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data" >> >> which does not host the data. It should be at least >> >> -g "/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/mm9/seq" >> >> I thought that data manager will automatically populate these loc files? Am I missing something obvious? I could manually modify tool_data_table_conf.xml to let it point to >> >> /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/all_fasta.loc >> >> This would work, but it is ugly. Could someone please give me a hand to fix this? >> >> Thanks, >> Rui >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > >

1 0

DRMAA chmod to world-writable if to user chown fails should not be default?
by David Trudgian 08 Apr '15

08 Apr '15

Hi, Looking through the DRMAA job runner code I noticed that in lib/galaxy/jobs/__init__.py:1582 that if chown to a user in change_ownership_for_run fails then there is chmod 0777 instead. I'm thinking that this probably shouldn't happen by default, or there needs to be a big warning on the wiki? I understand the world-readable security concerns posted there - but world-writable, for whatever reason, on a shared cluster seems like a disaster waiting to happen. Maybe either warn on the wiki, make the chmod 777 a non-default option, or an option to disable it? DT ________________________________ UT Southwestern Medical Center The future of medicine, today.

2 2

Multiple outputs of unknown types
by David Kelly 08 Apr '15

08 Apr '15

Hello, I've looked through the documentation about tools that produce multiple output files, but it seems like you're required to know the data types ahead of time (or make assumptions based on input data). But what about a case where you don't know what the output datatypes will be? Is it just possible to have N output files imported and sniffed? Thanks, Davids

3 2

Re: [galaxy-dev] Galaxy ToolShed Update Problem
by Martin Čech 08 Apr '15

08 Apr '15

Hi Zhaozhen, the --config_file argument needs to point to galaxy.ini config so the script can load the database. Galaxy continues to support Python 2.6 and 2.7, I am sorry that this script required 2.7 (or you could `pip install argparse`) - we will look into improving it. Martin, Galaxy Team On Wed, Apr 8, 2015 at 4:09 AM 孙兆朕 <sunzzn502(a)163.com> wrote: > Hi Martin, > > Thanks for the advise. It helped a lot for me to figure out where the > problem is. However, there were still error messages there. > I tried to run the update_shed_config_path.py script under the /script > folder. At first it says there was no module named argparse, then I found > out it was the python version problem, which required python 2.7+. Then I > updated my python. Then there was another error said there was no module > named galaxy.model.tool_shed_install. Then I tried harder, and found out > that it was the path problem. I figured out the sys path was in the above > folder. Then I copied and pasted the update_shed_config_path.py to the > parent folder and seemed solved the problem. Then I run > the update_shed_config_path.py and the command somehow run for a while, > with the error message No module named _sqlite3. It must be the DB > versions and plug-ins, then I fixed the DB. Then finally there came the > following message: > > usage: update_shed_config_path.py [-h] --config_file CONFIG_FILE --from > BAD_FILENAME --to GOOD_FILENAME [--force] > update_shed_config_path.py: error: argument --config_file is required > > I run the command line in: python2.7 update_shed_config_path.py > I wonder if I need some more input for the command line or something? > By the way, I hope there would be some advise for the new Galaxy > administrtators to know their Python should be 2.7+ and file path issues. > When I started to deploy Galaxy last year, the requirement in the web page > was Python 2.5 or something lower than 2.6 I used. I think that message > need to be updated as well. > Thanks in advance, > > Yours, > > Zhaozhen > > > > > argparse > > > At 2015-04-07 22:43:50, "Martin Čech" <marten(a)bx.psu.edu> wrote: > > Hello Zhaozhen, > > this is a known problem. The shed_tool_conf.xml got moved along with other > configs into /config folder but unfortunately some tools stored the path > absolutely in the DB. You can use the following script (which you should > find in the Galaxy /scripts folder): > > https://github.com/galaxyproject/galaxy/blob/dev/scripts/update_shed_config… > to update your DB to have the correct path to the config. > > Sorry for the inconvenience. > > Martin, Galaxy Team > > > On Tue, Apr 7, 2015 at 10:16 AM 孙兆朕 <sunzzn502(a)163.com> wrote: > >> Hi all, >> >> I want to update some of my tool plug-in and some problem shows, even if >> I updated my Galaxy instance to the latest. I suppose it is not related to >> the version of Galaxy. >> >> The problem occured when I used the administrator's panel and update my >> plug-in using the "Get updates" option. The next page reports that an >> internal Server Error has occured and the update was not successful. All >> the plug-in updates seems had the same problem. >> >> I checked the logs where I get the following message: >> >> CGI Variables >> ------------- >> CONTENT_LENGTH: '0' >> HTTP_ACCEPT: >> 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' >> HTTP_ACCEPT_ENCODING: 'gzip,deflate,sdch' >> HTTP_ACCEPT_LANGUAGE: 'zh-CN,zh;q=0.8' >> HTTP_CONNECTION: 'close' >> HTTP_COOKIE: >> 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424' >> HTTP_HOST: '222.28.163.110' >> HTTP_REFERER: ' >> http://222.28.163.110/admin_toolshed/browse_repositories?status=done&messag… >> .' >> HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 >> (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36' >> HTTP_X_FORWARDED_FOR: '222.28.170.89' >> HTTP_X_FORWARDED_HOST: '222.28.163.110' >> ORGINAL_HTTP_HOST: '127.0.0.1:8080' >> ORGINAL_REMOTE_ADDR: '127.0.0.1' >> PATH_INFO: '/admin_toolshed/update_to_changeset_revision' >> QUERY_STRING: 'tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> ' >> REMOTE_ADDR: '222.28.170.89' >> REQUEST_METHOD: 'GET' >> SERVER_NAME: '127.0.0.1' >> SERVER_PORT: '8080' >> SERVER_PROTOCOL: 'HTTP/1.0' >> >> >> WSGI Variables >> -------------- >> application: <paste.recursive.RecursiveMiddleware object at 0xd888510> >> is_api_request: False >> paste.cookies: (<SimpleCookie: >> galaxysession='c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424'>, >> 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424') >> paste.expected_exceptions: [<class >> 'paste.httpexceptions.HTTPException'>] >> paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object >> at 0xd888fd0> >> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at >> 0xdcf8650> >> paste.parsed_querystring: ([('tool_shed_url', ' >> https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), >> ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), >> ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')], >> 'tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> ') >> paste.recursive.forward: <paste.recursive.Forwarder from /> >> paste.recursive.include: <paste.recursive.Includer from /> >> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from >> /> >> paste.recursive.script_name: '' >> paste.throw_errors: True >> request_id: 'a12a5604dce911e4a05000215e543aa8' >> webob._parsed_query_vars: (MultiDict([('tool_shed_url', ' >> https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), >> ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), >> ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')]), >> 'tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> ') >> wsgi process: 'Multithreaded' >> ------------------------------------------------------------ >> ---------------------------------------- >> Exception happened during processing of request from ('127.0.0.1', 39438) >> Traceback (most recent call last): >> File >> "/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", >> line 1068, in process_request_in_thread >> self.finish_request(request, client_address) >> File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request >> self.RequestHandlerClass(request, client_address, self) >> File "/usr/lib64/python2.6/SocketServer.py", line 618, in __init__ >> self.finish() >> File "/usr/lib64/python2.6/SocketServer.py", line 661, in finish >> self.wfile.flush() >> File "/usr/lib64/python2.6/socket.py", line 303, in flush >> self._sock.sendall(buffer(data, write_offset, buffer_size)) >> error: [Errno 32] Broken pipe >> ---------------------------------------- >> 222.28.170.89 - - [07/Apr/2015:13:47:59 +0800] "GET >> /admin_toolshed/update_to_changeset_revision?tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> HTTP/1.0" 500 - " >> http://222.28.163.110/admin_toolshed/browse_repositories?status=done&messag…." >> "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) >> Chrome/34.0.1847.116 Safari/537.36" >> Error - <type 'exceptions.AttributeError'>: 'NoneType' object has no >> attribute 'get' >> URL: >> http://222.28.163.110/admin_toolshed/update_to_changeset_revision?tool_shed… >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', >> line 149 in __call__ >> app_iter = self.application(environ, sr_checker) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', >> line 84 in __call__ >> return self.application(environ, start_response) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', >> line 633 in __call__ >> return self.application(environ, start_response) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/base.py', >> line 133 in __call__ >> return self.handle_request( environ, start_response ) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/base.py', >> line 191 in handle_request >> body = method( trans, **kwargs ) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/decorators.py', >> line 87 in decorator >> return func( self, trans, *args, **kwargs ) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', >> line 1954 in update_to_changeset_revision >> persist=True ) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py', >> line 31 in __init__ >> metadata_dict=metadata_dict, user=None ) >> File >> '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/tool_shed/metadata/metadata_generator.py', >> line 60 in __init__ >> self.metadata_dict = { 'shed_config_filename' : >> self.shed_config_dict.get( 'config_filename', None ) } >> AttributeError: 'NoneType' object has no attribute 'get' >> >> >> CGI Variables >> ------------- >> CONTENT_LENGTH: '0' >> HTTP_ACCEPT: >> 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' >> HTTP_ACCEPT_ENCODING: 'gzip,deflate,sdch' >> HTTP_ACCEPT_LANGUAGE: 'zh-CN,zh;q=0.8' >> HTTP_CONNECTION: 'close' >> HTTP_COOKIE: >> 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424' >> HTTP_HOST: '222.28.163.110' >> HTTP_REFERER: ' >> http://222.28.163.110/admin_toolshed/browse_repositories?status=done&messag… >> .' >> HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 >> (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36' >> HTTP_X_FORWARDED_FOR: '222.28.170.89' >> HTTP_X_FORWARDED_HOST: '222.28.163.110' >> ORGINAL_HTTP_HOST: '127.0.0.1:8080' >> ORGINAL_REMOTE_ADDR: '127.0.0.1' >> PATH_INFO: '/admin_toolshed/update_to_changeset_revision' >> QUERY_STRING: 'tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> ' >> REMOTE_ADDR: '222.28.170.89' >> REQUEST_METHOD: 'GET' >> SERVER_NAME: '127.0.0.1' >> SERVER_PORT: '8080' >> SERVER_PROTOCOL: 'HTTP/1.0' >> >> >> WSGI Variables >> -------------- >> application: <paste.recursive.RecursiveMiddleware object at 0xd888510> >> is_api_request: False >> paste.cookies: (<SimpleCookie: >> galaxysession='c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424'>, >> 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424') >> paste.expected_exceptions: [<class >> 'paste.httpexceptions.HTTPException'>] >> paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object >> at 0xd888fd0> >> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at >> 0xdcf8650> >> paste.parsed_querystring: ([('tool_shed_url', ' >> https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), >> ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), >> ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')], >> 'tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> ') >> paste.recursive.forward: <paste.recursive.Forwarder from /> >> paste.recursive.include: <paste.recursive.Includer from /> >> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from >> /> >> paste.recursive.script_name: '' >> paste.throw_errors: True >> request_id: 'a58bb404dce911e481c700215e543aa8' >> webob._parsed_query_vars: (MultiDict([('tool_shed_url', ' >> https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), >> ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), >> ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')]), >> 'tool_shed_url= >> https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changese… >> ') >> wsgi process: 'Multithreaded' >> ------------------------------------------------------------ >> >> Could you please give me some advise to figure out where the problem is >> and how to fix it? >> >> Thanks, >> >> >> Zhaozhen >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > >

1 0