- galaxy-dev - lists.galaxyproject.org

PATH and local jobs
by Ryan Davis 12 Sep '13

12 Sep '13

Hi, We have a local server running Galaxy and a majority of the tools work correctly (bowtie, BWA, samtools and etc.). I am having issues with some of the FASTX-Toolkit tools. I am starting Galaxy with an init script on Centos 6 and the tools are in my $PATH and work at the command line, just not in Galaxy. The wiki suggest to "set $PATH in the startup file or use." It leaves me hanging and don't know what the other option is. Can someone share with me their workaround? Here is the wiki page: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies Here is the section I am hoping will help: Local Jobs Changes to $PATH can be persisted by setting them in your shell's startup file(s). This typically means ~/.bash_profile for bash, but please see the <http://www.gnu.org/software/bash/manual/bashref.html#Bash-Startup-Files> bash documentation on startup files or the INVOCATION section of the bash(1) man page to understand the intricacies of how that file is read. Of particular importance, if you are starting Galaxy in a method other than manual invocation from a shell prompt (with sh run.sh) such as with an init script, it is likely that your startup file will not be read. In this instance, you should set $PATH in the startup file or use. Thanks! Ryan

4 3

Reserved variables in param tags
by Kohler Manuel 12 Sep '13

12 Sep '13

Hi, I have a question regarding the param tag. I would like to pass on the user email to a external python script. I tried to use it like this: <param name="email" type="hidden" value=$__user_email__ /> <param name="experiment" type="select" label="Experiment" help="select Experiment" refresh_on_change="true" dynamic_options="getExperiments(email)"/> This does not work. Ideally I would like to have something like this: dynamic_options="getExperiments($__user_email__)" Has someone done this before? Cheers Manuel -- Manuel Kohler Center for Information Sciences and Databases (C-ISD) Department of Biosystems Science & Engineering (D-BSSE) ETH Zurich, Maulbeerstrasse (1078, 1.02), CH-4058 Basel, +41 61 387 3132

2 1

Problem with installing MAF cached dataset
by Leon Mei 12 Sep '13

12 Sep '13

Dear colleagues, I am having some trouble installing MAF datasets at our server. When I run "Extract MAF blocks given a set of genomic intervals", I got this error message: "Fatal Error: The MAF source specified (10_WAY_MULTIZ_hg19) appears to be invalid." Below is what I did by following the wiki page: http://wiki.galaxyproject.org/Admin/Datatypes/Add%20MAFs 1, download all mafs for hg19 from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/multiz46way/maf/ 2, index them using "maf_build_index.py --species=hg19,panTro2, gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 $file". Our user requested primates only. 3, Here is my maf_index.loc file: == 10-way multiZ (hg19) 10_WAY_MULTIZ_hg19 hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 /mnt/galaxyIndices/alignments/maf/chr1.maf,/mnt/galaxyIndices/alignments/m af/chr10.maf,/mnt/galaxyIndices/alignments/maf/chr11.maf,/mnt/galaxyIndices/alignments/maf/chr11_gl000202_random.maf,/mnt/galaxyIndices/ali gnments/maf/chr12.maf,/mnt/galaxyIndices/alignments/maf/chr13.maf,/mnt/galaxyIndices/alignments/maf/chr14.maf,/mnt/galaxyIndices/alignments /maf/chr15.maf,/mnt/galaxyIndices/alignments/maf/chr16.maf,/mnt/galaxyIndices/alignments/maf/chr17.maf,/mnt/galaxyIndices/alignments/maf/ch r17_ctg5_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000203_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000204_random.maf,/ mnt/galaxyIndices/alignments/maf/chr17_gl000205_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000206_random.maf,/mnt/galaxyIndices/a lignments/maf/chr18.maf,/mnt/galaxyIndices/alignments/maf/chr18_gl000207_random.maf,/mnt/galaxyIndices/alignments/maf/chr19.maf,/mnt/galaxy Indices/alignments/maf/chr19_gl000208_random.maf,/mnt/galaxyIndices/alignments/maf/chr19_gl000209_random.maf,/mnt/galaxyIndices/alignments/ maf/chr1_gl000191_random.maf,/mnt/galaxyIndices/alignments/maf/chr1_gl000192_random.maf,/mnt/galaxyIndices/alignments/maf/chr2.maf,/mnt/gal axyIndices/alignments/maf/chr20.maf,/mnt/galaxyIndices/alignments/maf/chr21.maf,/mnt/galaxyIndices/alignments/maf/chr21_gl000210_random.maf ,/mnt/galaxyIndices/alignments/maf/chr22.maf,/mnt/galaxyIndices/alignments/maf/chr3.maf,/mnt/galaxyIndices/alignments/maf/chr4.maf,/mnt/gal axyIndices/alignments/maf/chr4_ctg9_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr4_gl000193_random.maf,/mnt/galaxyIndices/alignments/maf/c hr4_gl000194_random.maf,/mnt/galaxyIndices/alignments/maf/chr5.maf,/mnt/galaxyIndices/alignments/maf/chr6.maf,/mnt/galaxyIndices/alignments /maf/chr6_apd_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr6_cox_hap2.maf,/mnt/galaxyIndices/alignments/maf/chr6_dbb_hap3.maf,/mnt/galaxyI ndices/alignments/maf/chr6_mann_hap4.maf,/mnt/galaxyIndices/alignments/maf/chr6_mcf_hap5.maf,/mnt/galaxyIndices/alignments/maf/chr6_qbl_hap 6.maf,/mnt/galaxyIndices/alignments/maf/chr6_ssto_hap7.maf,/mnt/galaxyIndices/alignments/maf/chr7.maf,/mnt/galaxyIndices/alignments/maf/chr 7_gl000195_random.maf,/mnt/galaxyIndices/alignments/maf/chr8.maf,/mnt/galaxyIndices/alignments/maf/chr8_gl000196_random.maf,/mnt/galaxyIndi ces/alignments/maf/chr8_gl000197_random.maf == Any hint on which step is wrong or missing is highly appreciate! Thanks, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_mei Mobile: +31 6 41709231

2 4

Stable Galaxy for local install
by test galaxy 12 Sep '13

12 Sep '13

Hi everyone, I want to install my local Galaxy with less bugs,and I don't know which version is more stable and work well.Could anyone give same advise?Thank you very much.

6 7

Deseq2 wrapper question
by ruiwang.sz 11 Sep '13

11 Sep '13

Hi All, we are making a wrapper for deseq2, and there is a step where control/experimental conditions need to be determined. For example, it would be a column like the following in the input file Type CTC CTC LM LM PT PT and we want the select list to contain CTC LM PT In other words, we want to filter out the 'Type' and have only distinct ones in the select list. We could make it in two steps, generate a intermediate file for this but it would be nice that we could directly retrieve the distinct types to make the list. I took a look at the tool config wiki, but didn't see anything (maybe because it was a quick scan). Does anyone have any tip on this? we'll really appreciate. Thanks, Rui

8 12

tool shed can't install via admin user
by Adhemar 11 Sep '13

11 Sep '13

Hi, I've just updated my galaxy-central version and I can't use the tool shed installer via admin user any more. When I click 'Install to Galaxy' it throws and 500 error. Could you please verify that? I tried the URL from tool_shed_url in the browser and it also doesn't open. Thanks, Adhemar 10.129.9.38, 10.129.0.1 - - [10/Sep/2013:16:05:59 -0200] "GET /galaxy/admin_toolshed/prepare_for_install?tool_shed_url= http://toolshed.g2.bx.psu.edu/&repository_ids=c45d6c51a4fcfc6c&changeset_re…" 500 - " http://toolshed.g2.bx.psu.edu/repository/preview_tools_in_changeset?reposit…" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36" Error - <class 'urllib2.HTTPError'>: HTTP Error 500: Internal Server Error URL: https://www.lmb.cnptia.embrapa.br/galaxy/admin_toolshed/prepare_for_install… File '/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/usr/lib/python2.6/site-packages/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/usr/lib/python2.6/site-packages/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/opt/bioinformatics/share/galaxy20130410/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 908 in prepare_for_install includes_tool_dependencies ) File '/opt/bioinformatics/share/galaxy20130410/lib/tool_shed/util/common_install_util.py', line 130 in get_dependencies_for_repository all_repo_info_dict = get_required_repo_info_dicts( trans, tool_shed_url, util.listify( repo_info_dict ) ) File '/opt/bioinformatics/share/galaxy20130410/lib/tool_shed/util/common_install_util.py', line 360 in get_required_repo_info_dicts response = urllib2.urlopen( request ).read() File '/usr/lib64/python2.6/urllib2.py', line 126 in urlopen return _opener.open(url, data, timeout) File '/usr/lib64/python2.6/urllib2.py', line 397 in open response = meth(req, response) File '/usr/lib64/python2.6/urllib2.py', line 510 in http_response 'http', request, response, code, msg, hdrs) File '/usr/lib64/python2.6/urllib2.py', line 435 in error return self._call_chain(*args) File '/usr/lib64/python2.6/urllib2.py', line 369 in _call_chain result = func(*args) File '/usr/lib64/python2.6/urllib2.py', line 518 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error CGI Variables ------------- CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE: 'pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4' HTTP_AUTHORIZATION: 'Basic YWRoZW1hcjpDYWxsIG1lIHRoZSBicmVlemUh' HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603a7b1853fca2041c148b00044fe9cfda0f39abd5fcb4fe71a0bc3b556f2ab9d86b; GUEST_LANGUAGE_ID=pt_BR; COOKIE_SUPPORT=true; JSESSIONID=EC393E44F6CC64AB7CECE0B38408E133' HTTP_HOST: 'www.lmb.cnptia.embrapa.br' HTTP_REFERER: ' http://toolshed.g2.bx.psu.edu/repository/preview_tools_in_changeset?reposit… ' HTTP_REMOTE_USER: 'adhemar@' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36' HTTP_X_URL_SCHEME: 'https' PATH_INFO: '/admin_toolshed/prepare_for_install' QUERY_STRING: 'tool_shed_url= http://toolshed.g2.bx.psu.edu/&repository_ids=c45d6c51a4fcfc6c&changeset_re… ' REMOTE_ADDR: '10.129.9.38, 10.129.0.1' REQUEST_METHOD: 'GET' SCRIPT_NAME: '/galaxy' SERVER_NAME: 'www.lmb.cnptia.embrapa.br' SERVER_PORT: '8080' SERVER_PROTOCOL: 'HTTP/1.0' WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0x8c58d90> is_api_request: False original_wsgi.url_scheme: 'http' paste.cookies: (<SimpleCookie: COOKIE_SUPPORT='true' GUEST_LANGUAGE_ID='pt_BR' JSESSIONID='EC393E44F6CC64AB7CECE0B38408E133' galaxysession='c6ca0ddb55be603a7b1853fca2041c148b00044fe9cfda0f39abd5fcb4fe71a0bc3b556f2ab9d86b'>, 'galaxysession=c6ca0ddb55be603a7b1853fca2041c148b00044fe9cfda0f39abd5fcb4fe71a0bc3b556f2ab9d86b; GUEST_LANGUAGE_ID=pt_BR; COOKIE_SUPPORT=true; JSESSIONID=EC393E44F6CC64AB7CECE0B38408E133') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x8c58610> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x22bca10> paste.parsed_querystring: ([('tool_shed_url', ' http://toolshed.g2.bx.psu.edu/'), ('repository_ids', 'c45d6c51a4fcfc6c'), ('changeset_revisions', 'bf1c3f9f8282')], 'tool_shed_url= http://toolshed.g2.bx.psu.edu/&repository_ids=c45d6c51a4fcfc6c&changeset_re… ') paste.recursive.forward: <paste.recursive.Forwarder from /galaxy> paste.recursive.include: <paste.recursive.Includer from /galaxy> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy> paste.recursive.script_name: '/galaxy' paste.throw_errors: True request_id: '0743e52a1a4c11e39742e41f13e30314' webob._parsed_query_vars: (MultiDict([('tool_shed_url', ' http://toolshed.g2.bx.psu.edu/'), ('repository_ids', 'c45d6c51a4fcfc6c'), ('changeset_revisions', 'bf1c3f9f8282')]), 'tool_shed_url= http://toolshed.g2.bx.psu.edu/&repository_ids=c45d6c51a4fcfc6c&changeset_re… ') wsgi process: 'Multithreaded' ------------------------------------------------------------

2 3

workflow_execute.py limitation?
by Hakeem Almabrazi 11 Sep '13

11 Sep '13

Hi, Has anyone use the API workflow_execute.py to run workflows with a lot of steps? The reason why I am asking is that I keep getting some errors when I ran workflows that are long (i.e ~ 50 steps). The same workflow could be executed successfully from the GUI. The first Few steps (16 steps) get scheduled in the created history but the rest don't and then after while error out. Here are the error messages I keep getting. HTTP Error 500: Internal Server Error 500 Internal Server Error The server has either erred or is incapable of performing the requested operation. ********************* And here is some error I capture in the log file> File "/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py", line 2126, in execute return self.tool_action.execute( self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/__init__.py", line 362, in execute history.add_dataset( data, set_hid = set_output_hid ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py", line 1124, in __getattr__ if key not in self: raise AttributeError, key AttributeError: add_dataset I am not sure if this is related to some kind of waiting processing time and since it does not get a reply then it errors out. I would really appreciate any kind of help. I might have to break my workflow into smaller ones if there is no other way. Regards

1 0

Proper way of using bioblend.galaxy.workflows.run_workflow() {'<input>': {'id': <encoded dataset ID>, 'src': '[ldda, ld, hda]'}
by Carlos Borroto 11 Sep '13

11 Sep '13

Hi, I'm rewriting my galaxy-api-tools project to use bioblend. I can't believe I missed it the first time around. I'm a little confused reading about LibraryDatasetDatasetAssociation (ldda), LibraryDataset (ld) and HistoryDatasetAssociation (hda). I found something in the wiki[1] but is not helping enough. I guess my question is, let's say I use any of bioblend.galaxy.libraries.upload_*() fucntions[2] to upload a file to a data library, the json response for that is something that looks like this: [{'url': '/api/libraries/ca4f6513e2849358/contents/e70b6e37a29da1c1', 'id': 'e70b6e37a29da1c1', 'name': '1.fastq'}] Now let's say I use bioblend.galaxy.tools.upload_file()[2] and upload a file to a history, the json response for that looks like: {'outputs': [{'misc_blurb': None, 'peek': '<table cellspacing="0" cellpadding="3"></table>', 'uuid': None, 'hda_ldda': 'hda', 'data_type': 'auto', 'deleted': False, 'history_id': '6eafbc3efd020327', 'purged': False, 'state': 'queued', 'name': '1.fastq', 'visible': True, 'genome_build': '?', 'model_class': 'HistoryDatasetAssociation', 'hid': 5, 'file_size': 0, 'metadata_data_lines': None, 'file_ext': 'auto', 'id': 'c3e7863ed8b6cf9e', 'misc_info': None, 'metadata_dbkey': '?'}]} Which 'src' setting should I use for bioblend.galaxy.workflows.run_workflow()'s[2] {'<input>': {'id': <encoded dataset ID>, 'src': '[ldda, ld, hda]'}? If I have to guess {'<input>': {'id': 'e70b6e37a29da1c1', 'src': 'ldda'} for the first one and {'<input>': {'id': 'c3e7863ed8b6cf9e', 'src': 'hda'} for the second one. That leaves out 'ld', when do I have to use that one? Finally and as a comment. I think the inconsistency on what to expect after delivering a payload to the API makes it hard to use these responses. See this issue[3] I submitted to bioblend but it seems is actually Galaxy who is at fault. [1]http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%… [2]http://bioblend.readthedocs.org/en/latest/api_docs/galaxy/all.html [3]https://github.com/afgane/bioblend/issues/40 Thanks, Carlos

1 0

'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'
by hogart 11 Sep '13

11 Sep '13

Dear all, I have succeffully installed the local instance of Galaxy, that uses the postgeSQL database, and it worked fine. After several monthes of uses, it appeared the neccessety to reinstall the whole Galaxy. For this, I have completely removed the old instance, and hg clone the new one to the same location. The psql database remains the same. After upgrading the database schema with the corresponding script, I have faced with inability of upgrading the counter of history in database. For example, after creating the new history, the addition of new dataset (by uploading via ftp, from local comp, from UCSC) result in the error: NotSupportedError: (NotSupportedError) DECLARE CURSOR ... FOR UPDATE/SHARE is not supported DETAIL: Cursors must be READ ONLY. 'SELECT history.hid_counter \nFROM history \nWHERE history.id = %(id_1)s FOR UPDATE' {'id_1': 5} I will be very grateful with any help with this issue. galaxy=> \du List of roles Role name | Superuser | Create role | Create DB | Connections | Member of -----------+-----------+-------------+-----------+-------------+----------- galaxy | no | no | yes | no limit | hogart | yes | yes | yes | no limit | postgres | yes | yes | yes | no limit | (3 rows) galaxy=> \l List of databases Name | Owner | Encoding -----------+----------+---------- galaxy | galaxy | LATIN9 hogart | hogart | LATIN9 postgres | postgres | LATIN9 template0 | postgres | LATIN9 template1 | postgres | LATIN9 (5 rows) galaxy=> select * from history where history.id=5; id | create_time | update_time | user_id | n ame | hid_counter | deleted | purged | genome_build | importable | slug | published | importing ----+----------------------------+----------------------------+---------+------- ----------+-------------+---------+--------+--------------+------------+------+- ----------+----------- 5 | 2013-09-10 21:38:31.888226 | 2013-09-10 21:38:31.888237 | 1 | Unname d history | 1 | f | f | ? | f | | f | f (1 row) psql -v 8.1.22 - sincerely, Sergei

2 6

ZFS storage recommendations
by Joachim Jacob | VIB | 11 Sep '13

11 Sep '13

Hi all, I am performing some tests to move my galaxy database to ZFS. Does anybody have experience with ZFS on linux, and some recommendations/experiences to optimize performance? The purpose is to share the database over NFS to the Galaxy VM. Thanks, Joachim. -- Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team

5 6