March 2011 - galaxy-dev - lists.galaxyproject.org

importing into history from library
by Glen Beane 01 Jul '11

01 Jul '11

We're running the latest galaxy-dist and noticed a problem importing files from libraries into our history. If we click the down arrow to the right of a dataset and choose "Import this dataset into selected histories" it works fine, however if our datasets are in folders within the library and we select multiple datasets and then do "Import selected datasets into histories" nothing happens. If I go to a library with datasets that are not in a folder then selecting multiple works. Has anyone else seen this? -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153

2 1

Support for subdirs in dataset extra_files_path
by Jim Johnson 02 Jun '11

02 Jun '11

Request is issue#494 https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-e… I'm finding that some qiime metagenomics applications build HTML results with an inherent directory structure. For some other applications, e.g. FastQC, I've been able to flatten the hierarchy and edit the html, but that appears problematic for qiime. Galaxy hasn't supported a dataset extra_files_path hierarchy, though the developers don't seem opposed to the idea: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-October/003605.html I added a route in lib/galaxy/web/buildapp.py and modified the dataset download code to traverse a hierarchy in lib/galaxy/web/controllers/dataset.py I don't think these add any security vulnerabilities, (I tried the obvious ../../ ). $ hg diff lib/galaxy/web/buildapp.py diff -r 6ae06d89fec7 lib/galaxy/web/buildapp.py --- a/lib/galaxy/web/buildapp.py Wed Mar 16 09:01:57 2011 -0400 +++ b/lib/galaxy/web/buildapp.py Wed Mar 16 10:24:13 2011 -0500 @@ -94,6 +94,8 @@ webapp.add_route( '/async/:tool_id/:data_id/:data_secret', controller='async', action='index', tool_id=None, data_id=None, data_secret=None ) webapp.add_route( '/:controller/:action', action='index' ) webapp.add_route( '/:action', controller='root', action='index' ) + # allow for subdirectories in extra_files_path + webapp.add_route( '/datasets/:dataset_id/display/{filename:.+?}', controller='dataset', action='display', dataset_id=None, filename=None) webapp.add_route( '/datasets/:dataset_id/:action/:filename', controller='dataset', action='index', dataset_id=None, filename=None) webapp.add_route( '/display_application/:dataset_id/:app_name/:link_name/:user_id/:app_action/:action_param', controller='dataset', action='display_application', dataset_id=None, user_id=None, app_name = None, link_name = None, app_action = None, action_param = None ) webapp.add_route( '/u/:username/d/:slug', controller='dataset', action='display_by_username_and_slug' ) $ $ hg diff lib/galaxy/web/controllers/dataset.py diff -r 6ae06d89fec7 lib/galaxy/web/controllers/dataset.py --- a/lib/galaxy/web/controllers/dataset.py Wed Mar 16 09:01:57 2011 -0400 +++ b/lib/galaxy/web/controllers/dataset.py Wed Mar 16 10:24:29 2011 -0500 @@ -266,17 +266,18 @@ log.exception( "Unable to add composite parent %s to temporary library download archive" % data.file_name) msg = "Unable to create archive for download, please report this error" messagetype = 'error' - flist = glob.glob(os.path.join(efp,'*.*')) # glob returns full paths - for fpath in flist: - efp,fname = os.path.split(fpath) - try: - archive.add( fpath,fname ) - except IOError: - error = True - log.exception( "Unable to add %s to temporary library download archive" % fname) - msg = "Unable to create archive for download, please report this error" - messagetype = 'error' - continue + for root, dirs, files in os.walk(efp): + for fname in files: + fpath = os.path.join(root,fname) + rpath = os.path.relpath(fpath,efp) + try: + archive.add( fpath,rpath ) + except IOError: + error = True + log.exception( "Unable to add %s to temporary library download archive" % rpath) + msg = "Unable to create archive for download, please report this error" + messagetype = 'error' + continue if not error: if params.do_action == 'zip': archive.close()

2 1

Sharing tool definitions between XML files
by Peter 02 Jun '11

02 Jun '11

Hi all, I'm wondering if there is any established way to share parameter definitions between tool wrapper XML files? For example, I am currently working on NCBI BLAST+ wrappers, and these tools have a lot in common. For example, the output format option will be a select parameter, and it will be the same for blastn, blastp, tblastn, etc. I can just cut and paste the definition but this seems inelegant and will be a long term maintenance burden if it ever needs updating (lots of places would need to be updated the same way). I'd like to have a shared XML snippet defining this parameter which I could then reference/include from each tool wrapper than needs it. I'm thinking of something like XML's <!ENTITY ...> might work. Is this possible? Peter

2 2

Upload fails, webapp and job runner running on different machines
by Louise-Amélie Schmitt 12 May '11

12 May '11

Hi everyone, I'm currently trying a new galaxy install with the webapp and the job runner running on different machines, sharing a nfs volume where the galaxy files are, and another one where the data is supposed to be stored, as specified in the file_path and new_file_path values in the universe_wsgi files. (one for the webapp and one for the runner, as stated in the doc) Both instances run properly with no error message but when I want to upload a file from wherever in a library, here's what happens: The file's row appears in the library with the "uploading" note. The job is registered in the database. And that's all. Nothing changes afterwards. Nothing is saved anywhere. This can't be a file-too-big issue since the test file I'm working with weighs 3.5kB. I probably stupidly missed something but I really don't see where the problem might be. Any idea? Cheers L-A

3 3

Using wget/curl with Tool Shed?
by Peter Cock 11 May '11

11 May '11

Hi all, I'm trying to download and install something from the Galaxy Tool Shed http://community.g2.bx.psu.edu/ on to our sever. To do this, I'd like to copy/paste the URL of a tool's tar ball (or individual XML file etc) from my browser running on my desktop to a terminal window logged in to our server to give to wget or curl. e.g. This URL will give me bam_to_bigwig_0.0.1.tar.gz if run in a web browser: http://community.g2.bx.psu.edu/common/download_tool?cntrller=tool&id=3b1686… However, the URL actually goes to an HTML page then does some javascript magic to redirect to the real file. I can't easily use this URL with wget or curl. Is this something that you could fix in the website code? For now I can just download the file on my desktop, then scp it to the server (or ssh -X into the server and run a browser from there), but this is an unnecessary hurdle I feel. Regards, Peter

2 1

Email notification - is workflow is the only way?
by Paul, Rohit (NIH/NCI) [C] 11 May '11

11 May '11

>From what I understand about Galaxy so far, in order to have a user notified that their job has completed, a workflow has to be created in which email notification is chosen as a "Step Action". For the project that I'm currently working on, I need to have a link to one particular workflow appear in the Tools left-hand side panel/frame for every user that logs in to Galaxy. They shouldn't have to import the specific workflow — it should automatically be listed. I know that I can create links in the Tools panel, but it seems that in order to access a workflow (via direct URL such as: '/workflow/run?id=12345') the logged in user has to either be the creator of the workflow or the creator has to have explicitly "shared" the workflow with the user. Is there a way to grant "share" permission on a workflow to all logged-in users, so that I can list it as a Tool link and it shows up for all users, current and those who may register in the future? Or is there some other approach I can/should take to enable job completion notification for all logged-in users? -Rohit

2 1

Fresh galaxy installation tips/requests
by Assaf Gordon 05 May '11

05 May '11

Hi all, It's been a long while since I had to install a fresh production galaxy server, and I can offer some tips or requests for minor improvements (but to give credit were it's due - installation is much smoother now, and DRMAA works great with SGE out of the box). These are just minor annoyances, but they can help make new installations easier. --- 1. Debugging REMOTE_USER issues. The mix of apache + authentication + non-root URL + load balancing is a killer. googling "mod_rewrite + mod_auth" shows that I'm not the only one having problems with it... Adding those two lines in "./lib/galaxy/web/framework/middleware/remoteuser.py", line 79 is a wonderful debugging tool, which enables you to see what Apache is passing on to Galaxy: for k,v in environ.items(): sys.stderr.write ( "%s:\t%s\n" % ( k, v ) ) I needed to print all environment variables (not just HTTP_REMOTE_USER) so I could debug mod_rewrite's variables setup. If there was an easy way to enable this debug from the INI file it would be a great help. --- 2. The file name "universe_wsgi.ini" is hard-coded in several places. Most of them are python scripts (like cleanup, ampq, etc.), but one is "./galaxy/eggs/init.py". Not a real problem, except that in the "WebApplicationScaling" wiki page it's kind of implied that "universe_wsgi.ini" is not used any more after creating the two separated ini files (universe_wsgi.runner.ini and universe_wsgi.webapp.ini). If you rename "universe_wsgi.ini" instead of copying it, galaxy will simply not start, even if given a different INI file name. So the best recommendation (in the documents) would probably be to maintain three identical INI files, except for the "[server:main]" part of each (and the "enable_job_running = False" in the runner). Most importantly, make sure that the database connection string is identical in all of them. As a side node, the "cp" command in the "WebApplicationScaling" wiki page is incorrect (you can't copy one file into two files). It'll also be good to mention that the cleanup scripts will always use "universe_wsgi.ini" regardless of what the galaxy server is using. --- 3. Saving SGE/PBS/DRMAA shell scripts if "debug=TRUE" in the jobrunner's INI file - this one is a gem, too bad I discovered it too late. If only it was documented :) --- 4. INI keys that accept multiple comma separated values (e.g. "admin_users") are sensitive to white space. Example: admin_users = gordon@cshl.edu,foobar@cshl.edu works fine, but: admin_users = gordon(a)cshl.edu , foobar(a)cshl.edu will not work (spaces around the comma). Same thing with "ucsc_display_sites": ucsc_display_sites = cshl,main will work (and display both "cshl" and "main"), but: ucsc_display_sites = cshl, main will display only "cshl" (because "main" is " main" with a space). It's probably easier to document it than to fix it, but either will do. --- 5. There's no "ensembl_display_sites" in the INI, so disabling the "display at ensembl" link (for BAM files, etc.) requires commenting out the corresponding "<display>" tag in "datatypes_conf.xml", whereas for other servers (gbrowse,ucsc) it's enough to comment out the "ucsc_display_sites" in the INI. --- 6. For galaxy servers running inside an intranet without access to the main/test UCSC Genome browser server, and want to use a local mirror (this might be an obscure setup but that's what I have): disabling both "main" and "test" in "ucsc_display_sites" will disable the authenticated user bypass, because "remouteuser.py" have them hard-coded before setting "self.allow_ucsc_main = True". I needed to patch and add "cshl" as a third option - this is kind of ugly. --- Two development issues not related to production server installation: 1. Debugging complex cheetah templates is very hard. If you have good tricks (beyond #slient and #breakpoint) - examples would be appriciated. If there's a way to see the compiled cheetah code which produced the error - it would be great. One problem is that the error is always reported without a line number (I guess because it's first converted into a string) and without the offending tag. 2. I only half-understand the new "tool_data_table_conf.xml" mechanism - is it explained somewhere ? Looks quite complicated, especially adding in-lined lamba for reading the "__app__" global variable inside the cheetah code of each tool :( --- And user interface quirks: 1. when managing workflows, the "configure your workflow menu" page is a bit uninformative: there's no title, header, instruction or anything else, and the button says "Submit Query" which doesn't mean much ("Update Workflow Menu" would be better). If you click on "configure your workflow menu" and you don't have any workflows, the page is down right mysterious . 2. JQuery Tipsy balloons - please make them go away. Or, make the appearance delay changeable in the INI file (to something like 600 seconds :) ). --- As always, grumbling aside, galaxy is GREAT! so keep up the good work. thanks, -gordon

2 2

Version Error and Python 2.7
by John David Osborne 03 May '11

03 May '11

I have (or rather had) a functional galaxy installation on a CentOS box running python 2.4. I did an mecurial update for the first time and broke my installation. It looks like some eggs are getting out of date and when I run scramble.py I get an error (see below). I'm guessing this is because scramble expects python 2.5 or above for the import (import in 2.4 I don't believe takes arguments). My question is does galaxy really still support 2.4 and should I move to python 2.7 (the current stable version) because the wiki just mentions versions 2.4 to 2.6 as being supported. -John run_scramble_script(): Executing in /home/galaxy/galaxy_dist/scripts/scramble/build/py2.4/python_daemon: /usr/bin/python -ES scramble.py Traceback (most recent call last): File "scramble.py", line 29, in ? execfile( "setup.py", globals(), locals() ) File "setup.py", line 22, in ? main_module = __import__(main_module_name, fromlist=['version']) TypeError: __import__() takes no keyword arguments scramble(): Egg already exists, remove to force rebuild: /home/galaxy/galaxy_dist/eggs/nose-0.11.1-py2.4.egg scramble(): Egg already exists, remove to force rebuild: /home/galaxy/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.4.egg scramble(): Egg already exists, remove to force rebuild: /home/galaxy/galaxy_dist/eggs/WebOb-0.8.5-py2.4.egg scramble(): Egg already exists, remove to force rebuild: /home/galaxy/galaxy_dist/eggs/PasteScript-1.7.3-py2.4.egg Traceback (most recent call last): File "scripts/scramble.py", line 23, in ? eggs = c.scramble() File "/home/galaxy/galaxy_dist/lib/galaxy/eggs/scramble.py", line 243, in scramble raise last_exc # only 1 failure out of the crate, be more informative galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for python_daemon 1.5.5

4 5

galaxy related
by shashi shekhar 26 Apr '11

26 Apr '11

if i want some in information from tool xml file to code file ,how i will get the info. it it's possible ,send me detailed info. Regards galaxy user shashi shekhar

2 1

Problem with "Map with Bowtie for Illumina" on a local Galaxy
by Leon Mei 26 Apr '11

26 Apr '11

Dear colleagues, I have a local installed Galaxy. When I try to run bowtie to align a small FASTQ file, I got the following message and the result was empty. """ empty, format: sam, database: Project-Id-Version: PACKAGE VERSION Report-Msgid-Bugs-To: POT-Creation-Date: 2008-09-21 18:33+0900 PO-Revision-Date: 2009-03-17 03:55-0400 Last-Translator: FULL NAME Language-Team: en Plural-Forms: nplurals=2; plural=(n != 1) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Generated-By: Babel 0.9.4 Info: """ It seems bowtie_wrapper.py wasn't executed at all. Has someone seen this message before and what could be the cause? Thanks, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center (http://www.nbic.nl/) Skype: leon_mei Mobile: +31 6 41709231

2 2