April 2015 - galaxy-dev - lists.galaxyproject.org

question about error output
by Roberto Alonso CIPF 16 Apr '15

16 Apr '15

Hello, I am defining some tools for my own galaxy installation and I would like to know if there is a more elegant solution for redirecting error output than the one in the next config file: <tool id="gatk_mio" name="calling with gatk"> <description>calling with gatk</description> <command> ln -s "${input}" input.bam ; samtools index input.bam; java -jar /home/ralonso/software/GenomeAnalysisTK-3.3-0/GenomeAnalysisTK.jar -T UnifiedGenotyper -R /home/ralonso/BiB/Galaxy/data/Cclementina_v1.0_scaffolds.fa -I input.bam -o $output *2>/dev/null;* </command> <inputs> <param format="bam" name="input" type="data" label="bam"/> </inputs> <outputs> <data format="vcf" name="output" /> </outputs> <help> bwa </help> </tool> Best regards many thanks -- Roberto Alonso Functional Genomics Unit Bioinformatics and Genomics Department Prince Felipe Research Center (CIPF) C./Eduardo Primo Yúfera (Científic), nº 3 (junto Oceanografico) 46012 Valencia, Spain Tel: +34 963289680 Ext. 1021 Fax: +34 963289574 E-Mail: ralonso(a)cipf.es

2 1

$REMOTE_USER
by Wang, Yu 16 Apr '15

16 Apr '15

Hi, Is there a way to query the actual name of the user, who is logged into galaxy and submitted the workflow ($REMOTE_USER), from inside a galaxy tool when galaxy is configured for external authentication using an apache server? It seems like galaxy passes this variable to scripts like 'drmaa_external_runner.py' but I could not find it in the environment of a tool which has been started via e.g. python ( using <command interpreter="python"> in the xml-file of the tool)." Cheers, Yu di29her wang(a)lrz.de<mailto:wang@lrz.de> On 13 Apr 2015, at 18:00, galaxy-dev-request(a)lists.galaxyproject.org<mailto:galaxy-dev-request@lists.galaxyproject.org> wrote: Send galaxy-dev mailing list submissions to galaxy-dev(a)lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> To subscribe or unsubscribe via the World Wide Web, visit https://lists.galaxyproject.org/listinfo/galaxy-dev or, via email, send a message with subject or body 'help' to galaxy-dev-request(a)lists.galaxyproject.org You can reach the person managing the list at galaxy-dev-owner(a)lists.galaxyproject.org When replying, please edit your Subject line so it is more specific than "Re: Contents of galaxy-dev digest..." HEY! This is important! If you reply to a thread in a digest, please 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. 2. Strip out everything else in the digest that is not part of the thread you are responding to. Why? 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative. Today's Topics: 1. Re: Strange repository dependency behavior with toolshed-installed naive_variant_caller (Martin Čech) ---------------------------------------------------------------------- Message: 1 Date: Mon, 13 Apr 2015 14:27:10 +0000 From: Martin Čech <marten(a)bx.psu.edu> To: Brian Claywell <bclaywel(a)fredhutch.org> Cc: galaxy-dev(a)lists.galaxyproject.org Subject: Re: [galaxy-dev] Strange repository dependency behavior with toolshed-installed naive_variant_caller Message-ID: <CAHKBquFw66cXyss7KjExM4-mpdK0P_TLT_xtCOBEK4pLs3OYOg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello Brian, we have made some changes in the past month that might address the issue you are having. Are you still encountering problems? If so could you please try re-installing the tools again? thank you Martin On Tue, Feb 24, 2015 at 5:15 PM Brian Claywell <bclaywel(a)fredhutch.org> wrote: Hi Martin, Thanks for the prompt reply! My responses are inline below. On Tue, Feb 24, 2015 at 8:26 AM, Martin Čech <marten(a)bx.psu.edu> wrote: Are you using Admin UI for installation or API scripts? The admin UI exclusively. Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these? Yes -- the iuc package_numpy_1_7 revision a4bdc17eed4a is listed twice, with both showing the green "Installed" status. Interestingly the installed repo page shows that they're installed in different locations: (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/ef12a3a11d5b/package_num… (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/a4bdc17eed4a/package_num… At one point during the troubleshooting steps I described in my original email, package_pybamparser_0_0_1 and package_pybamtools_0_0_1 showed up in the same way, with two records each having the same revision and a green "Installed" status. If you check on the filesystem in the {{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication? I didn't see anything that struck me as out of the ordinary. Here's a gist of the directory structures for the affected packages under both shed_tools and tool_deps: https://gist.github.com/bcclaywell/23c0b274f2d4bd0d659f How big is your Galaxy in terms of installed tools, complexity of setup and traffic? I don't think it's terribly big -- we have a couple of large tool suites like BEDTools, the gops interval operations suites, GATK 1 and 2, etc, and then maybe 15-20 other tools and their dependencies? Our setup isn't very complex, either; Galaxy is running with one web process and one handler process behind an nginx reverse proxy, all running on a single machine. Jobs are submitted to our slurm cluster via the SSH CLI runner. Traffic is pretty low -- we're just now rolling out of beta. Cheers, Brian -- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150413/6…> ------------------------------ Subject: Digest Footer _______________________________________________ galaxy-dev mailing list galaxy-dev(a)lists.galaxyproject.org https://lists.galaxyproject.org/listinfo/galaxy-dev To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ------------------------------ End of galaxy-dev Digest, Vol 106, Issue 12 *******************************************

2 2

Automated Tool Tests w/ Data Tables
by Jeremy Liu 15 Apr '15

15 Apr '15

Galaxy Devs, I've been following this documentation to write auto tests for my tools: https://wiki.galaxyproject.org/Admin/Tools/WritingTests?action=show&redirec… Straightforward enough, but I am not sure how to incorporate input parameters from data tables into this. My tools currently use as input parameters entries/files from a data table. These files were installed using a data manager. Ideally, the tests should also reference the same data table files as the tools do. How can I do this? Thanks, Jeremy Liu jeremy.liu(a)yale.edu

3 2

problems wit upgrading to "latest_15.03" - 'UniverseApplication' object has no attribute 'control_worker'
by Hans-Rudolf Hotz 14 Apr '15

14 Apr '15

Hi I am in the process of updating our development server (from "latest_2014.10.06", including the security fix from January to "latest_15.03" ). and when I restart the server, I get the following error: ... galaxy.jobs.handler INFO 2015-04-14 15:46:05,180 sending stop signal to worker thread galaxy.jobs.handler INFO 2015-04-14 15:46:05,181 job handler stop queue stopped Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/path_to_galaxy/galaxy_dist/lib/galaxy/app.py", line 167, in shutdown if self.control_worker: AttributeError: 'UniverseApplication' object has no attribute 'control_worker' Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/path_to_galaxy/galaxy_dist/lib/galaxy/app.py", line 167, in shutdown if self.control_worker: AttributeError: 'UniverseApplication' object has no attribute 'control_worker' Any help appreciated Thank you very much Hans-Rudolf -- Hans-Rudolf Hotz, PhD Bioinformatics Support Friedrich Miescher Institute for Biomedical Research Maulbeerstrasse 66 4058 Basel/Switzerland

2 2

problem with revision updates for repositories.
by Anna Terry 14 Apr '15

14 Apr '15

Hi, I have a few installed tool shed repositories which are showing that updates are available. When I click get repository updates, instead of the usual upate page, I get a page that simply says Found The resource was found at http://toolshed.g2.bx.psu.edu/repository/check_for_updates?galaxy_url=https…; you should be redirected automatically. This is for bam_to_sam, but same problem for blast_datatypes, pileup_interval and sam_to_bam too. Anyone have any ideas why I'm seeing this? Cheers Anna

2 9

MySQL InnoDB - Execution Order
by McCully, Dwayne (NIH/NIAMS) [C] 10 Apr '15

10 Apr '15

Hello Everyone, The running Galaxy in a production environment documentation states: "If you are using MySQL<http://dev.mysql.com/> with MyISAM<http://dev.mysql.com/doc/refman/5.1/en/myisam-storage-engine.html> table engine when Galaxy is in multiprocess configuration, workflow steps will get executed out of order<http://dev.list.galaxyproject.org/Job-execution-order-mixed-up-tt4662488.ht…> and fail. Use InnoDB<http://dev.mysql.com/doc/refman/5.1/en/innodb-storage-engine.html> engine instead or switch to PostgreSQL<http://www.postgresql.org/>." I've enabled InnoDB for my MySQL database via the my.cnf file but the default database and database to log all database transactions were created with MyISAM files. If this is normal, when will Galaxy create InnoDB tables? I don't believe you can create a database with InnoDB. Dwayne

2 1

Visualization in anonymous mode ?
by Loraine Guéguen 09 Apr '15

09 Apr '15

Hi, I'd like to use the visualization in anonymous mode, but the visualize icon on the dataset is not available in anonymous mode (although it works fine when connected). Is it possible to activate the dataset visualization in anonymous mode ? Thanks, Loraine -- Loraine Guéguen Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/

2 5

Tests not being run on toolsheds?
by Peter Briggs 09 Apr '15

09 Apr '15

Hello Apologies as this seems to be quite a frequent question (and I may have missed something) however: are the tool tests being executed on the main or test toolsheds at the moment? The most recent tests results that I can find for any of the tools I've uploaded is the 29th January. I've also looked at a couple of tools from other users and the situation appears to be the same (at least in those cases). Thanks for your help! Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

3 9

Problems after upgrade
by C. Ch. 09 Apr '15

09 Apr '15

Hi all, We had a problem on the NAS where we couldn't write anymore. This problem was solved and I decided that I would profit to upgrade galaxy. Now we have a problem where jobs do not get launched and the handler log shows a lot of error messages. The first error message is copied below. Afterwards it alternates between galaxy.jobs.handler ERROR 2015-04-07 15:58:24,959 Exception in monitor_step galaxy.jobs.runners ERROR 2015-04-07 15:58:23,542 Unhandled exception checking active jobs with, mainly, the same error stack (second error message shown below). Any idea what is going wrong? My first impression is that it is trying to recuperate some job that galaxy believes is still running but the machine has been stopped and rebooted in the mean time. Any help is welcome! Cristian galaxy.jobs.runners ERROR 2015-04-07 15:44:04,809 (27764) Unhandled exception calling queue_jobTraceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 97, in run_next method(arg) File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 115, in queue_job if not self.prepare_job( job_wrapper, include_metadata=True ): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 140, in prepare_job job_id = job_wrapper.get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 794, in get_id_tag return self.get_job().get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 790, in get_job return self.sa_session.query( model.Job ).get( self.job_id ) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 840, in get return loading.load_on_ident(self, key) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/loading.py", line 231, in load_on_ident return q.one() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2395, in one ret = list(self) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2434, in __iter__ context = self._compile_context() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2810, in _compile_context entity.setup_context(self, context) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 3194, in setup_context column_collection=context.primary_columns File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/interfaces.py", line 466, in setup strat.setup_query(context, entity, path, loader, adapter, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py", line 1111, in setup_query column_collection, parentmapper, chained_from_outerjoin File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py", line 1234, in _generate_row_adapter use_mapper_path=True) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/util.py", line 364, in __init__ else mapper.with_polymorphic_mappers, File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 1877, in _with_polymorphic_mappers configure_mappers() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 2589, in configure_mappers mapper._post_configure_properties() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 1694, in _post_configure_properties prop.init() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/interfaces.py", line 144, in init self.do_init() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py", line 1549, in do_init self._process_dependent_arguments() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py", line 1605, in _process_dependent_arguments self.target = self.mapper.mapped_table File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py", line 1522, in mapper argument = self.argument() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/ext/declarative/clsregistry.py", line 283, in __call__ (self.prop.parent, self.arg, n.args[0], self.cls)InvalidRequestError: When initializing mapper Mapper|Queue|kombu_queue, expression 'Message' failed to locate a name ("name 'Message' is not defined"). If this is a class name, consider adding this relationship() to the <class 'kombu.transport.sqlalchemy.Queue'> class after both dependent classes have been defined. Repeated error messages:galaxy.jobs.runners ERROR 2015-04-07 15:58:23,542 Unhandled exception checking active jobsTraceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 501, in monitor self.check_watched_items() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 238, in check_watched_items galaxy_id_tag = ajs.job_wrapper.get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 794, in get_id_tag return self.get_job().get_id_tag() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 790, in get_job return self.sa_session.query( model.Job ).get( self.job_id ) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 150, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1165, in query return self._query_cls(entities, self, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 108,in __init__ self._set_entities(entities) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 118,in _set_entities self._set_entity_selectables(self._entities) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 151,in _set_entity_selectables ent.setup_entity(*d[entity]) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 3036, in setup_entity self._with_polymorphic = ext_info.with_polymorphic_mappers File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 1877, in _with_polymorphic_mappers configure_mappers() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py", line 2586, in configure_mappers raise eInvalidRequestError: One or more mappers failed to initialize - can't proceed with initialization of other mappers. Original exception was: When initializing mapper Mapper|Queue|kombu_queue, expression 'Message' failed to locate a name ("name 'Message' is not defined"). If this is a class name, consider adding this relationship() to the <class 'kombu.transport.sqlalchemy.Queue'> class after both dependent classes have been defined.

2 2

Tool Shed installs: coercing to Unicode: need string or buffer, NoneType found
by Peter Cock 09 Apr '15

09 Apr '15

I just hit a problem installing package_bx_python_0_7 via the Admin GUI. Name: bx-python Version: 0.7.1 Type: package Status: Error Error: Error installing tool dependency bx-python version 0.7.1: coercing to Unicode: need string or buffer, NoneType found Name: galaxy-ops Version: 1.0.0 Type: package Status: Installed Clicked on "Uninstall" for bx-python, told: Uninstall tool dependencies Name: bx-python Version: 0.7.1 Type: package Install directory: This dependency's installation directory does not exist, click <b>Uninstall</b> to reset for installation. Clicked on "Uninstall" again, seemed to work. I went through a series of "repair repositiory" rounds on package_bx_python_0_7 and intersect tool, and it all seems fine now. I'm not sure what the original glitch was, but there are two clear issues here. First, the Unicode error - perhaps previously reported e.g. http://dev.list.galaxyproject.org/How-to-remove-a-broken-toolshed-install-t… http://dev.list.galaxyproject.org/tophat2-install-tc4663736.html Those previous discussions suggest the database and file system were out of sync - but clearly this unexpected None value ought to be better handled. Second, the over-escaped bold in the message, presumably part of: https://trello.com/c/oX6i51Mn/2298-back-off-over-escaping Peter

1 0