September 2014 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Some plugin for digital normalization
by Michael R. Crusoe 24 Nov '14

24 Nov '14

> From: Cristian Alejandro Rojas <alejandro.0317(a)gmail.com> > Date: December 26, 2013 2:37 AM > To: galaxy-dev(a)lists.bx.psu.edu > Subject: [galaxy-dev] Some plugin for digital normalization > Hello all, > > I've been searching in tool sheds(main and test) for a plugin to do digital normalization of reads, for example Trinity has one script to do this (./util/normalize_by_kmer_coverage.pl), diginorm is another alternative (http://ged.msu.edu/papers/2012-diginorm/) > > Do you know if there is a galaxy tool ready for this work or should I develop my own? Hello Cristian, I am actively working on a wrapper for normalize-by-median.py and the other tools in the khmer suite. https://github.com/ged-lab/khmer/blob/galaxy-integration/scripts/normalize-… It requires some changes on our end which you can track here: https://github.com/ged-lab/khmer/pull/237 You can install this branch of khmer via pip pip install -e git+git@github.com:ged-lab/khmer.git@output_naming1#egg=khmer or if you don't have a github account: pip install -e git+ https://github.com/ged-lab/khmer.git@output_naming1#egg=khmer (Instructions on installing using a virtualenv or on OS X are at: https://khmer.readthedocs.org/en/latest/install.html just substitute in the pertinent pip line from above) I haven't tested it much; doing so is the top of my to-do list for the rest of this week. If you or anyone else is feeling adventurous I would be happy to have the feedback. Cheers! -- Michael R. Crusoe: Software Engineer and Bioinformatician mcrusoe(a)msu.edu @ the Genomics, Evolution, and Development lab; Michigan State University http://ged.msu.edu/ http://orcid.org/0000-0002-2961-9670 @biocrusoe

3 5

filtering a <param> of type 'data' so only one type is available
by Dan Tenenbaum 24 Oct '14

24 Oct '14

Hi, I have a tool wrapper with a <param> of type="data". Currently, this renders as a text box with a drop down list that has 52 items in it (the number of things I have in my history, I guess). I'd like to filter this list so that only some items in the history (in my case any item whose name ends with '.rda') are shown in this dropdown. It seems like maybe the 'format' parameter to the 'param' tag is what I want, but: 1) I tried format="rda" and that didn't seem to change anything. 2) "rda" is not in the datatypes.conf file which the documentation suggests is required? (.rda is an extension used for serialized R objects.) The tool I'm working on only accepts rda files as input and the reason I'm asking for this is that it is all too easy to accidentally feed it a file of some other type. If the dropdown could be filtered so that only items which will work with the tool are shown, that would be great. Is there a way to do that? Thanks, Dan

4 5

API : Max retries exceeded
by Gildas Le Corguille 13 Oct '14

13 Oct '14

Hi, I wrote a script which need to retrieve a list of all users on our production instance. It seems that we have too many users (325) :P bioblend.galaxy.client.ConnectionError: HTTPConnectionPool(host='galaxy.sb-roscoff.fr', port=8080): Max retries exceeded with url: /api/users?key=fd420zefff9a9df586210fd8c8452d98 (Caused by <class 'socket.error'>: [Errno 111] Connection refused): None My script is running on an test instance which have less users. Is there a way to set this threshold ? Thanks Gildas PS : don't try my api_key, little hacker, it's a fake :P

2 3

Simple standard for API use of a global user/key that all loaded tools can draw upon?
by Dooley, Damion 02 Oct '14

02 Oct '14

Do any of you see the need to establish a way for tools to use a global (current site) API url/key combination? From a security perspective I'm assuming this is basically ok since tools run under the galaxy user anyways (and can directly read and write files to their heart's content). I understand that a particular user's API key could help limit what they have access to. My tool however wants to do some prepatory form display and post-job work on behalf of the Galaxy system in addition to working with user-centric data; I presume others have similar use-case needs. I think it would amount to having a standard GALAXY_TOOL_API_URL and GALAXY_TOOL_API_KEY managed in universe_wsgi.ini (left blank if not desired by sys admin). The key would be generated by a galaxy admin using a real admin user id. These would be available as python or shell environment variables when it came time to execute a tool's <command interpreter="python"> etc. work. The wee other detail is that I'm using <code file="versioned_data_form.py" /> in my tool xml because there is API information I have to retrieve to set up the form with. So the url and key need to be accessible in that execution environment too?!! Am I missing the boat; alternately has this already been talked about?! Thanks for advice ... Damion Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

2 3

'KeyError' in workflow extraction when using newer tool version
by Peter Briggs 02 Oct '14

02 Oct '14

Dear all One of my users has encountered a problem with extracting a workflow from a history in our local Galaxy installation (tag latest_2014.08.11, changeset 14342:db520d154716). The history was created with a version of a local tool which has since been updated, and the extraction page warns that the newer version of the tool will be used for those datasets i.e.: Dataset was created with tool version "0.0.4", but workflow extraction will use version "0.0.7". However when the extraction is run it fails immediately: the user sees the "Internal Server Error" page, and in the log I see: ===== Error - <type 'exceptions.KeyError'>: 'platform' URL: https://galaxy.ls.manchester.ac.uk/workflow/build_from_current_history File '/home/galaxy/production/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/production/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/production/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/production/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/production/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/home/galaxy/production/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/workflow.py', line 1220 in build_from_current_history workflow_name=workflow_name File '/home/galaxy/production/galaxy-dist/lib/galaxy/workflow/extract.py', line 30 in extract_workflow steps = extract_steps( trans, history=history, job_ids=job_ids, dataset_ids=dataset_ids, dataset_collection_ids=dataset_collection_ids ) File '/home/galaxy/production/galaxy-dist/lib/galaxy/workflow/extract.py', line 104 in extract_steps tool_inputs, associations = step_inputs( trans, job ) File '/home/galaxy/production/galaxy-dist/lib/galaxy/workflow/extract.py', line 271 in step_inputs associations = __cleanup_param_values( tool.inputs, param_values ) File '/home/galaxy/production/galaxy-dist/lib/galaxy/workflow/extract.py', line 330 in __cleanup_param_values cleanup( "", inputs, values ) File '/home/galaxy/production/galaxy-dist/lib/galaxy/workflow/extract.py', line 327 in cleanup group_values = values[input.name] KeyError: 'platform' ===== 'platform' is a conditional parameter that was introduced into the tool between the two versions. It looks like when tying to use the newer verison of the new tool, that the new parameter is not getting picked up. Is this expected behaviour? (Btw I've seen a similar issue i.e. the "KeyError: 'platform' exception when I have the two versions of this tool installed from a local toolshed, and try switching between to the newer one from the older one.) Any help or suggestions are greatly appreciated. Many thanks Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 2

Login with a system user
by Daniel Ruiz Molina 01 Oct '14

01 Oct '14

Hello, I have configured Galaxy, but I would be very interested in configure it with PAM or /etc/passwd (/etc/shadow) instead of creating a user in the web interface. How can I configure Galaxy for authenticating with system files? Thanks.

3 2

Internal Server error when attemptng to use Admin Tab
by David Bernick 01 Oct '14

01 Oct '14

Good morning - I have acquired an Internal server error on our local galaxy install. It seems to be an unterminated string. How do I resolve this? thanks in advance - David Bernick UCSC Here is the relevant portion of the log (I hope). Error - <type 'exceptions.ValueError'>: Unterminated string starting at: line 1 column 65528 (char 65528) URL: http://campusrocks.cse.ucsc.edu:8080/admin/index File '/projects/galaxy/bin/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/projects/galaxy/bin/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/projects/galaxy/bin/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/projects/galaxy/bin/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/projects/galaxy/bin/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/projects/galaxy/bin/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 383 in decorator return func( self, trans, *args, **kwargs ) File '/projects/galaxy/bin/galaxy-dist/lib/galaxy/web/base/controllers/admin.py', line 34 in index installing_repository_ids = suc.get_ids_of_tool_shed_repositories_being_installed( trans.app, as_string=True ) File '/projects/galaxy/bin/galaxy-dist/lib/tool_shed/util/shed_util_common.py', line 277 in get_ids_of_tool_shed_repositories_being_installed app.install_model.ToolShedRepository.status == loading_datatypes_status ) ): File '/projects/galaxy/bin/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2348 in instances File '/projects/galaxy/bin/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py', line 2120 in _instance File '/projects/galaxy/bin/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py', line 1974 in populate_state File '/projects/galaxy/bin/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 150 in fetch_col File '/projects/galaxy/bin/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/types.py', line 658 in process File '/projects/galaxy/bin/galaxy-dist/lib/galaxy/model/custom_types.py', line 52 in process_result_value return json_decoder.decode( str( _sniffnfix_pg9_hex(value) ) ) File '/usr/lib64/python2.6/json/decoder.py', line 319 in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File '/usr/lib64/python2.6/json/decoder.py', line 336 in raw_decode obj, end = self._scanner.iterscan(s, **kw).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 183 in JSONObject value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 217 in JSONArray value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 183 in JSONObject value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 217 in JSONArray value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 183 in JSONObject value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 217 in JSONArray value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 217 in JSONArray value, end = iterscan(s, idx=end, context=context).next() File '/usr/lib64/python2.6/json/scanner.py', line 55 in iterscan rval, next_pos = action(m, context) File '/usr/lib64/python2.6/json/decoder.py', line 155 in JSONString return scanstring(match.string, match.end(), encoding, strict) ValueError: Unterminated string starting at: line 1 column 65528 (char 65528)

2 1

Disk space issue - again
by Patel, Viren 30 Sep '14

30 Sep '14

Hello. I have a Galaxy user whose disk usage remains at 91% even after deleting and purging all histories and datasets (The initial disk usage was 99%). I have searched the mailing list (e.g. http://dev.list.galaxyproject.org/disk-space-usage-td4660164.html) and wiki posts and followed all the steps, including purging hidden data. I have the cleanup scripts running once a day via cron: 01 01 * * * cd ~/galaxy-dist && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -r -d 60 -1 11 01 * * * cd ~/galaxy-dist && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -r -d 60 -2 21 01 * * * cd ~/galaxy-dist && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -r -d 60 -4 31 01 * * * cd ~/galaxy-dist && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -r -d 60 -5 41 01 * * * cd ~/galaxy-dist && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -r -d 60 -3 51 01 * * * cd ~/galaxy-dist && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -r -d 60 -6 01 02 * * * find ~/tmp -mtime +60 -delete I have manually run the above commands with -d 1 to delete anything older than 1 day and I have restarted the Galaxy service. However the user¹s disk usage remains at 91%. I am at a loss of what to try next. Any suggestions or ideas of what the issue could be, or what I need to do additionally? I am running the April 2014 distribution of Galaxy. Thanks. Viren ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).

2 1

2014 Galaxy Questionnaire
by Dave Clements 30 Sep '14

30 Sep '14

Hello all, The Galaxy Project is preparing for our next grant cycle and we are seeking your feedback and comments on on all things Galaxy. We have created two questionnaires, each focused on a different way to interact with Galaxy: 1. *Galaxy User Questionnaire <https://docs.google.com/forms/d/1NT3ma2g7rR-mWA_vgOg2Ia_6VTFiZ86um2JhX2SecC…>* 2. *Galaxy Admin, Tool Developer, and Galaxy Developer Questionnaire <https://docs.google.com/forms/d/1lszXJ2lDJ2dYCWKXBk1qT1jO7XdDDfxXmGRCUMZFX2…>* Please take a few minutes and fill out the surveys that apply to you. The questionnaires are structured so you can skip topics (whole pages!) that don't apply to you, and every question is optional. *And, to thank you for your time and effort, the Galaxy Project will increase your storage quota on usegalaxy.org <https://usegalaxy.org/> by 50GB. *That's a 20% increase. Let your voice be heard (and, get some storage)! The Galaxy Team <https://wiki.galaxyproject.org/GalaxyTeam> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

datatype directory
by David Hoover 30 Sep '14

30 Sep '14

Why isn't there a datatype for a directory of files? This seems like such a simple thing. If an executable generates or expects a directory as its input or output, why must a fancy complicated composite datatype be created to handle this? David Hoover -------------------- David Hoover, PhD Helix Systems Staff SCB/DCSS/CIT/NIH 301-435-2986 http://helix.nih.gov

2 1