galaxy-dev May 2013

galaxy-dev@lists.galaxyproject.org

101 participants
197 discussions

Load balancing and job configuration
by Olivier Inizan 03 May '13

03 May '13

Dear galaxy-dev-list, I am trying to use new-style job configuration whith a load-balancing on 2 web servers and 2 job hanlders. I have configured load balancing as follows (universe_wsgi.ini): # Configuration of the internal HTTP server. [server:web0] use = egg:Paste#http port = 8083 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:web1] use = egg:Paste#http port = 8082 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 This configuration works fine (manager.log): galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner 'galaxy.jobs.runners.local:LocalJobRunner' as 'local' galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner 'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr' galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id 'local:///', url 'local:///' converted, got params: galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners plugins: lwr:local galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue started galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started To use new-style job configuration I have create the following job_conf.xml: <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/> </plugins> <handlers default="handlers">  <handler id="server:web0" tags="handlers"/> <handler id="server:web1" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> </destinations> </job_conf> When I restart the instance the manager.log outputs: galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from ./job_conf.xml galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler 'server:web0' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler 'server:web1' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <handlers> default set to child with id or tag 'handlers' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <destinations> default set to child with id or tag 'local' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration So everything seems fine but when I try to run a tool, the corresponding job is in permanent waiting status (gray color in history). I think I have missed something in the configuration process, Thanks for your help, Olivier

2 2

Quick question about new job configuration
by Ganote, Carrie L 03 May '13

03 May '13

Hi folks, First of all, I really love the new job configuration style. It makes it very easy to change swaths of jobs and to streamline the way jobs are submitted to the cluster. I had no issues with the dynamic runner and I like the way that tags interact. My question is for the dynamic job runner; is it possible to send the job to a different job *handler* dynamically? Or is it already given to the default handler at the time that the dynamic function is called? I want to do something like this: def ncbi_blastn_wrapper(job): # Send to a different handler? inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) inp_data.update( [ ( da.name, da.dataset ) for da in job.input_library_datasets ] ) query_file = inp_data[ "query" ].file_name query_size = os.path.getsize( query_file ) if query_size > 1024 * 1024: handlername = "bighandler" else: handlername = "smallhandler" return JobDestination(runner="pbs", handler=handlername, params={"Resource_List": "24:00:00"}) The way I have it set up now, my handlers are pointing at different clusters with different capabilities, and it would be cool to make the decision which to run at runtime. Thanks a ton! Carrie Ganote

1 0

workflow multiple inputs
by Olivia Doppelt 03 May '13

03 May '13

Dear developpers, Last week we made an update on the default branch of galaxy-dist. Some problems concerning the api were detected and resolved thanks to Dannon Baker by pulling the stable branch. However, a very interesting fonctionality was working well on this default branch, concerning the multiple inputs for workflows. There were two icons next to the input dataset window, a multiple file icon and a link icon to link paired input. Why is this functionnality not included in the stable branch ? isn't stable yet? It is a very useful though. Please keep me informed about that. Cheers, Olivia Doppelt-Azeroual Pasteur Institute, CIB, PARIS

2 1

Re: [galaxy-dev] [Biopython] GFF parsing with biopython
by Brad Chapman 03 May '13

03 May '13

Mic; (moving to galaxy-dev list so folks there can follow, but future questions are more appropriate for the Biopython list only since this isn't a Galaxy question) > I have the following GFF file from a SNAP > > X1 SNAP Einit 2579 2712 -3.221 + . X1-snap.1 [...] > With the code below I have tried to parse the above GFF file The attributes you're missing are parts of the feature, not the SeqRecord itself, which is why you're seeing attribute error. Here's a full example that pulls all of the information from an example line: from BCBio import GFF in_file = "snap.gff" with open(in_file) as in_handle: for rec in GFF.parse(in_handle): feature = rec.features[0] print rec.id print feature.qualifiers["source"][0] print feature.type print feature.location.start print feature.location.end print feature.qualifiers["score"][0] print feature.location.strand print feature.qualifiers.get("X1-snap.1", [None])[0] which outputs: X1 SNAP Einit 2578 2712 -3.221 1 true Hope this helps, Brad

2 1

Problem accessing test tool shed repositories
by Sytchev, Ilya 02 May '13

02 May '13

Hi, I can't install any tools from the test tool shed into the local Galaxy instances. I can see the list of the repositories but nothing happens when I click on any of them. I can still access repositories in the main tool shed. The same thing happens in fresh installations of both galaxy-central and galaxy-dist. Attached screenshot shows a list of Sequence Analysis repositories in the test tool shed (note the lack of buttons). I'd appreciate any help. Thanks, Ilya

2 2

MPileup for cloudman instance
by Iry Witham 02 May '13

02 May '13

Hi All, I have noticed that there is no Mpileup available in either of the tool sheds. Is there a simple way to install it in my instance? Thanks, Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

1 0

Tests for Workflows on the tool shed?
by Peter Cock 02 May '13

02 May '13

Hello all, I'm looking at how best to incorporate Galaxy workflows into a publication - a standalone *.ga file in supplementary materials would probably work but is static and I fear likely to bit rot. I think posting each workflow (or set of workflows) as a repository on the main Galaxy Tool Shed would be better, as I can continue to update it as needed, and it can be linking to it via the citable URL, e.g. Greg's example on the Test Tool Shed which declares some dependencies: http://testtoolshed.g2.bx.psu.edu/view/greg/heteroplasmy_workflow (This assumes the Tool Shed URLs will be stable, but I'm having to assume that anyway for tool wrappers.) However, my natural next question is how do we define unit tests for these workflows? In the case of a methods paper, some sample data for testing would be reasonable. In the case of a more applied paper, the workflow could perhaps include the actual data used in the paper and reproduce the published results (ambitious but quite possible for moderate sized datasets). I'm guessing this would be a major enhancement - something for Greg and Dave to tackle once the current round of work for testing tools on the Tool Shed is done (grin). Shall I open a Trello issue for this? Regards, Peter

1 0

toolshed down?
by Geert Vandeweyer 02 May '13

02 May '13

hi, I'm getting the following error when trying to reach the main/test toolsheds. Are they down? **** This webpage is not available Chromium's connection attempt to toolshed.g2.bx.psu.edu was rejected. The website may be down, or your network may not be properly configured. **** also from within galaxy, I can't reach it, nor can the galaxy-process itself: galaxy.model.migrate.check INFO 2013-05-02 12:12:45,219 At database version 114 tool_shed.galaxy_install.migrate.check DEBUG 2013-05-02 12:12:45,390 pysqlite>=2 egg successfully loaded for sqlite dialect The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_… raised the exception: <urlopen error [Errno 111] Connection refused> The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_… raised the exception: <urlopen error [Errno 111] Connection refused> The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz&o… raised the exception: <urlopen error [Errno 111] Connection refused> The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz_p… raised the exception: <urlopen error [Errno 111] Connection refused> Traceback (most recent call last): File "/galaxy/galaxy-beta/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 37, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/galaxy/galaxy-beta/galaxy-dist/lib/galaxy/app.py", line 54, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/galaxy_install/migrate/check.py", line 56, in verify_tools tool_shed_accessible, missing_tool_configs_dict = common_util.check_for_missing_tools( app, tool_panel_configs, latest_tool_migration_script_number ) File "/galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/util/common_util.py", line 66, in check_for_missing_tools return tool_shed_accessible, missing_tool_configs_dict UnboundLocalError: local variable 'missing_tool_configs_dict' referenced before assignment -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

2 2

Inconsistent menus in Galaxy Tool Shed
by Peter Cock 02 May '13

02 May '13

Hi all, This is just a usability bug, but I think it should be a simple fix that would make working with the Tool Shed a bit easier as the current menus seem randomised. --------------- If I click on "Repositories I own", then each tool has a drop down with only one action: * Mark as deprecated [image: Inline image 1] (This puzzles me - why just that rarely used option? Why not say upload, or view tests?) --------------- If I click on a tool, I go to the management page. Here the top right menu offers: * View tool functional test results * Upload files to repository * View change log * Browse or delete repository tip files * Reset all repository metadata * Mark repository as deprecated * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 2] --------------- If I go to the functional test results, the menu changes: * Manage repository * Upload files to repository * View change log * Browse or delete repository tip files * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 3] i.e. Aside the obvious change given which page we are on, why are "Reset all repository metadata" and "Mark repository as deprecated" missing? --------------- If I goto browse/delete tip files: * Manage repository * Upload files to repository * Browse or delete repository tip files * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 5] Notice "View change log" is now missing (as well as functional tests etc). --------------- If I go to the change log page, the menu changes again: * Upload files to repository * Manage repository * Browse or delete repository tip files * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 4] Again, several things are missing - and in particular the functional tests entry. To me that is highly relevant when looking at the change log. Also note the ordering of "Upload files to repository" and "Manage repository" has flipped which is confusing. --------------- My suggested fix is quite simple - use the same drop down menu for the repository actions on every page, either with the current page still on the list but disabled (and visually distinct as marker) is possible, or simply removed. Thanks, Peter

2 3

ASN.1 (text and binary) formats in Galaxy & Tool Shed
by Peter Cock 02 May '13

02 May '13

Hello all, Although they are these days also offering XML for many tools, the NCBI still make heavy use of the older ASN.1 file format (both as plain text and binary). This crops up in BLAST (e.g. as the BLAST archive format, or as dustmasker output), in the Entrez Utilities (e.g. for sequence data as an alternative to GenBank for FASTA format etc, or pubmed, etc) and also for 3D structures. I think it could make sense to define generic 'asn1' and 'asn1-binary' formats in the Galaxy core (name suggestions welcome), and even perhaps 'ncbi-asn1' and 'ncbi-asn1-binary' too. Then ToolShed entries can define domain specific subclasses. For instance, the BLAST+ wrapper could include definitions for the dustmasker output, and perhaps the BLAST archive format too. Separately anyone working with 3D structures as ASN.1 could define another sub-format, etc. I see this as a clear analogy to the assorted XML file formats in existence - defined in Galaxy as subclasses of the core XML format included with the Galaxy core. Would a pull request implementing this be acceptable? Peter P.S. Does anyone know an authoritative source for the MIME types used by the NCBI? Using the BLAST website they offer plain text ASN.1 just as text/plain, likewise efetch also seems to use text/plain for ASN.1 downloads. However I've seen references to chemical/ncbi-asn1-ascii and chemical/ncbi-asn1-binary mime-types mentioned, e.g. http://www.ncbi.nlm.nih.gov/data_specs/asn/NCBI_all.asn i.e. It appears that 3D structure NCBI ASN.1 files use a well defined MIME type, while most NCBI ASN.1 text files default to text/plain - which we can handle nicely in Galaxy as subclasses.

4 10

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

galaxy-dev May 2013