April 2012 - galaxy-dev - lists.galaxyproject.org

Problem with automated install & removal with of tool with tool_data_table_conf.xml.sample
by Peter Briggs 24 Apr '12

24 Apr '12

Hello I've encountered some problems installing and then trying to remove a tool from the main toolshed in a local Galaxy production instance. The tool is the tmap_wrapper but I don't think that the tool itself is the problem. When I tried the automated install from the toolshed, the Galaxy window came back with "Server error" and in paster.log I found this: == paster.log fragment start == galaxy.util.shed_util DEBUG 2012-04-20 15:46:15,151 Installing repository 'tmap_wrapper' galaxy.util.shed_util DEBUG 2012-04-20 15:46:15,186 Cloning http://toolshed.g2.bx.psu.edu/repos/nils homer/tmap_wrapper galaxy.util.shed_util DEBUG 2012-04-20 15:46:17,114 Updating cloned repository to revision "40ade4f81a30" galaxy.tools.parameters.dynamic_options WARNING 2012-04-20 15:46:17,278 Data table named 'tmap_indexes' is required by tool but not configured galaxy.util.shed_util DEBUG 2012-04-20 15:46:17,322 Adding new row (or updating an existing row) for repository 'tmap_wrapper' in the tool_shed_repository table. galaxy.tools.parameters.dynamic_options WARNING 2012-04-20 15:46:17,400 Data table named 'tmap_indexes' is required by tool but not configured 130.88.213.137 - - [20/Apr/2012:15:46:15 +0100] "POST /galaxy/admin_toolshed/install_repository?tool_shed_url=http%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F&repo_info_dict=17a5b933699a3c40818076daffd7d763e25e8071%3A7b22746d61705f77726170706572223a205b226d617070696e6720666f7220496f6e20546f7272656e74222c2022687474703a2f2f746f6f6c736865642e67322e62782e7073752e6564752f7265706f732f6e696c73686f6d65722f746d61705f77726170706572222c2022343061646534663831613330225d7d&includes_tools=True HTTP/1.1" 500 - "http://XXXXXX.XX.XXXXXXXXXX.XX.XX/galaxy/admin_toolshed/install_repository?…" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20100101 Firefox/10.0.1" Error - <type 'exceptions.NameError'>: global name 'handle_sample_tool_data_table_conf_file' is not defined URL: http://XXXXXX.XX.XXXXXXXXXX.XX.XX/galaxy/admin_toolshed/install_repository?… File '/home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 173 in decorator return func( self, trans, *args, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/controllers/admin_toolshed.py', line 316 in install_repository shed_tool_conf=shed_tool_conf ) File '/home/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py', line 890 in load_repository_contents repository_tools_tups = handle_missing_data_table_entry( trans.app, tool_path, sample_files, repository_tools_tups ) File '/home/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py', line 753 in handle_missing_data_table_entry error, correction_msg = handle_sample_tool_data_table_conf_file( app, sample_file ) NameError: global name 'handle_sample_tool_data_table_conf_file' is not defined == paster.log fragment end == (I've replaced the machine name in the URLs with X's so our sysadmin won't maim me.) It seems that the error is something to do with Galaxy failing to handle the tool's tool_data_table_conf.xml.sample. As far as I can tell the hg clone of the tool worked okay as the files for the tool can be found in /home/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/nilshomer/tmap_wrapper/40ade4f81a30/tmap_wrapper/tmap_wrapper_0.0.19 but the tool doesn't appear in the tool panel and isn't referenced in either the shed_tool_conf.xml or tool_conf.xml files. However: the tool does appear under the Admin -> Manage installed tool shed repositories option in Galaxy. Unfortunately attempting to uninstall it via this interface also fails with the "Server error" in the Galaxy window. This section of paster.log looks relevant here: == paster.log fragment start == Error - <type 'exceptions.TypeError'>: coercing to Unicode: need string or buffer, NoneType found URL: http://XXXXXX.XX.XXXXXXXXXX.XX.XX/galaxy/admin_toolshed/deactivate_or_unins… File '/home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 173 in decorator return func( self, trans, *args, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/controllers/admin_toolshed.py', line 181 in deactivate_or_uninstall_repository remove_from_tool_panel( trans, repository, shed_tool_conf, uninstall=remove_from_disk_checked ) File '/home/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py', line 989 in remove_from_tool_panel tool_panel_dict = generate_tool_panel_dict_from_shed_tool_conf_entries( trans, repository ) File '/home/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py', line 490 in generate_tool_panel_dict_from_shed_tool_conf_entries tree = util.parse_xml( shed_tool_conf ) File '/home/galaxy/galaxy-dist/lib/galaxy/util/__init__.py', line 105 in parse_xml tree = ElementTree.parse(fname) File '/home/galaxy/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py', line 859 in parse File '/home/galaxy/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py', line 576 in parse TypeError: coercing to Unicode: need string or buffer, NoneType found == paster.log fragment end == After this the tool files are still there under the "shed_tools" directory, and the tool still appears under "Manage installed tool shed repositories". We're using the most recent galaxy-dist (hg reports our most recent changeset as 6799:40f1816d6857). I don't know if anyone else has encountered this problem but it would be useful to know how to either complete the automated install or else be able to remove the tool completely and add it manually for now. Apologies btw for the long message, thanks for your help - Galaxy is fantastic and all the work that has gone into it is very much appreciated! Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 2

Re: [galaxy-dev] putting up a SNP module on galaxy
by Jeremy Goecks 23 Apr '12

23 Apr '12

Hi Xiangyu, This question is best directed to the galaxy development list, which I've cc'd. In general, to integrate a tool into Galaxy, you'll want to write a tool wrapper: http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax and then upload it to the Galaxy toolshed: http://wiki.g2.bx.psu.edu/Tool%20Shed You can also upload workflows to the tool shed as well: http://wiki.g2.bx.psu.edu/Tool%20Shed#Enabling_workflow_sharing:_importing_… Feel free to email the galaxy-dev list with any questions you may have. Best, J. On Apr 23, 2012, at 3:51 PM, Deng, Xiangyu (CDC/OID/NCEZID) (CTR) wrote: > Hi Jeremy, > > My name is Xiangyu, working across street at the enteric disease laboratory branch, CDC. I built a SNP calling pipeline and want to put it somewhere with a friendly GUI so our/other wet lab persons can use it. I wonder if I can do it on Galaxy. > > The pipeline, written in python, uses bowtie 2, samtools, velvet, biopython and a perl program. > > Thanks, > Xiangyu > > --- > Xiang-yu Deng, Ph.D. > > ASM Fellow > Enteric Diseases Laboratory Branch > Division of Foodborne, Waterborne and Environmental Diseases > National Center for Emerging and Zoonotic Infectious Diseases > Centers for Disease Control and Prevention > MS-C03, 1600 Clifton Road, Atlanta, GA 30333 > Office: 404-639-1036 > dengx(a)cdc.gov >

1 0

Galaxy does not generate consistent results
by zhengqiu cai 23 Apr '12

23 Apr '12

I used bwa to align a short read file to a reference sequence. I ran it several times, and I got different results for the runs on the same input dataset. The fist run, it generated a ~280,000 lines "gzipped file". I checked the file on the server, and it was actually a sam file, not gzipped. The second run, it generated ~90,000 lines sam file. The third run, it generated ~130,000 lines sam file. The fourth run, it generated ~220,000 lines "gzipped file" Does anyone meet the similar problem? Thank you very much. Cai

1 0

Galaxy with postgresql
by zhengqiu cai 23 Apr '12

23 Apr '12

Hi All, I want to use postgresql instead of sqlite, and below is how I did the settings: I am running Galaxy under the user ubuntu, so first I create the ubuntu user sudo -u postgres createuser -D -A -P ubuntu Then I create the database named galaxy_db sudo -u postgres createdb -O ubuntu galaxy_db in Galaxy configuration file, use postgresql postgres:///galaxy_db?user=ubuntu&host=/var/run/postgresql When I start Galaxy, sqlite dialect, not postgresql is selected. I am pasting part of the log below: galaxy.model.migrate.check DEBUG 2012-04-23 17:59:52,676 pysqlite>=2 egg successfully loaded for sqlite dialect galaxy.model.migrate.check INFO 2012-04-23 17:59:52,707 At database version 93 galaxy.tool_shed.migrate.check DEBUG 2012-04-23 17:59:52,718 pysqlite>=2 egg successfully loaded for sqlite dialect galaxy.tool_shed.migrate.check INFO 2012-04-23 17:59:52,730 At migrate_tools version 1 galaxy.model.custom_types DEBUG 2012-04-23 17:59:52,735 pysqlite>=2 egg successfully loaded for sqlite dialect Is there anything wrong with my configuration? Thank you. Cai

2 2

Re: [galaxy-dev] Toolshed initial upload errors
by Greg Von Kuster 23 Apr '12

23 Apr '12

Hi Paul, The use of mercurial eggs for all tool shed mercurial related actions is now available in our central repository in change set 7102:c8f4f73e9199. This will be available in the dist repository with the next release, which I believe will be coming soon. If you're tracking the central repository, updating will hopefully solve the problems you were seeing. Thanks! Greg Von Kuster On Apr 20, 2012, at 7:39 AM, Paul-Michael Agapow wrote: > Thanks Greg, > > It's a puzzling one alright. Looking forward to testing the new version, if only because I've run out of ideas on where to look. > > Cheers > > p > > ---- > Paul Agapow (paul-michael.agapow(a)hpa.org.uk) > Bioinformatics, Health Protection Agency > > -----Original Message----- > From: Greg Von Kuster [mailto:greg@bx.psu.edu] > Sent: 19 April 2012 22:47 > To: Paul-Michael Agapow > Cc: 'galaxy-dev(a)lists.bx.psu.edu' > Subject: Re: [galaxy-dev] Toolshed initial upload errors > > Paul, > > Sorry to see you're still experiencing problems. Based on the issues you've encountered (as well as one or two others recently) I've spent some time re-working things to eliminate the need for installing mercurial to use a local tool shed. We will build eggs for the mercurial package for the various versions of Python supported by Galaxy and include them in the distribution. > > I'm pretty close to having this finished, so it is likely that this will be available early next week. > > I'm not sure if this will fix the problems you're seeing, but at least it will eliminate one of the variables. > > Greg Von Kuster > > On Apr 19, 2012, at 5:02 AM, Paul-Michael Agapow wrote: > >> [For those who came in late - I've installed a local toolshed, which allows me to create repositories, but every time I attempt to upload files, it errors out with "TypeError: array item must be char". For those who come after me, here's what I worked out thus far.] >> >> Greg asked: >> >>> Since you've tried uploading various files with no success, the problem is likely >>> to be caused by something specific to your environment - possibly the version of >>> the mercurial package you're using. What version of Python are you running, and >>> what version of the mercurial package do you have installed with it? Also, what >>> version of Galaxy do you have, and what database / version are you using? >> >> We're CentOS, an older flavour (4), but my Mercurial is up to data (2.1.2). Python 2.6.4, Galaxy is 6799:40f1816d6857 (grabbed it fresh last week for testing), running it with sqlite. However, the Mercurial is actually installed local to the account I'm using, so I wonder if the toolshed is getting confused with another version, although hg doesn't seem to be installed on the system. >> >> Further investigations reveal that the files appear to be in the repo (database/community_files). The error manifest in the middle of Mercurial, in manifest.py where it attempts to coerce a Unicode string into a character array. (As there are some reported issues of Windows file names with Unicode under Mercurial, and I'm uploading from a Windows machine, I used a Mac to create a repo and add a file. Nope, same behaviour.) The Cistrome galaxy fork (https://bitbucket.org/cistrome/cistrome-harvard/src/e7e2fdd74496/lib/galaxy…) mentions occasional similar errors. >> >> I check the Mercurial installation: >> >> % hg --version >> Mercurial Distributed SCM (version 2.1.2+10-4d875bb546dc) >> ... >> % hg debuginstall >> Checking encoding (UTF-8)... >> Checking installed modules (/home/f0/paul/Installed/lib/python2.6/site-packages/mercurial)... >> Checking templates (/home/f0/paul/Installed/lib/python2.6/site-packages/mercurial/templates)... >> Checking commit editor... >> Checking username... >> No problems detected >> >> (Actually, I was missing a username and a user ~/.hgrc file. But making that, it passes. Error still persists.) >> >> Work continues. >> >> ---- >> Paul Agapow (paul-michael.agapow(a)hpa.org.uk) >> Bioinformatics, Health Protection Agency >> >> ----------------------------------------- >> ************************************************************************** >> The information contained in the EMail and any attachments is >> confidential and intended solely and for the attention and use of >> the named addressee(s). It may not be disclosed to any other person >> without the express authority of the HPA, or the intended >> recipient, or both. If you are not the intended recipient, you must >> not disclose, copy, distribute or retain this message or any part >> of it. This footnote also confirms that this EMail has been swept >> for computer viruses, but please re-sweep any attachments before >> opening or saving. HTTP://www.HPA.org.uk >> ************************************************************************** >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > >

1 0

multiple tool outputs
by Russell Bonneville 20 Apr '12

20 Apr '12

Hello all, I am working on integrating a few tools into Galaxy with multiple outputs. There are examples on the wiki for handling variable outputs determined by parameters and an arbitrary number of outputs, but none of them seemed to apply to this case. This tool must accept two lists of parameters a and b. The number of output files is (len(a) * len(b)) + 2. Is there a way to implement this in Galaxy without having to arbitrarily impose a limit on the length of the parameter lists? Also, if I would be required to use the arbitrary outputs mechanism, could the user still incorporate the output of this tool into workflows? Thank you. Sincerely, Russell Bonneville Russell Bonneville Undergraduate Research Assistant Jin Lab, Department of Biomedical Informatics The Ohio State University Wexner Medical Center bonneville.4(a)buckeyemail.osu.edu<mailto:bonneville.4@buckeyemail.osu.edu>

1 0

some weird thing about [shed tools] I have found
by JIE CHEN 20 Apr '12

20 Apr '12

Dear all, I installed and uninstalled some shed tools for my Galaxy instance. And I have found some weird things which I don't think they are as expected. 1. After I installed a shed tool using the Galaxy web UI and the tool can run successfully, I didn't find any change of the "shed_tool_conf.xml" file. According to the instructions under the link http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_installation_of_Galaxy_tool…, a entry of the newly added tool should be added. Thus I don't know where I go to modify the tools information(such as the section name..) like I did in the "tool_conf.xml" file. 2. When I go to "Admin |Manage installed tool shed repositories<http://localhost/admin_toolshed/browse_repositories> ", I can't find all the installed tools. Only some of them are displayed there. 3. After I uninstalled a shed tool, all the related files are deleted, which is good. But the empty directories(under the Shed_tools) are left. 4. Last, I have one question: if I install a shed tool into a new section(which means it doesn't exist before) , how could we modify the section's name? Anyone have encounter the same problems? Hope someone can explain to me. Thanks in advance!!! Cheers, Tyler

1 0

BAM to BigWig (and tool ID clashes)
by Peter Cock 20 Apr '12

20 Apr '12

Hi Brad & Lance, I've been using Brad's bam_to_bigwig tool in Galaxy but realized today (with a new dataset using a splice-aware mapper) that it doesn't seem to be ignoring CIGAR N operators where a read is split over an intron. Looking over Brad's Python script which calculates the coverage to write an intermediate wiggle file, this is done with the samtools via pysam. It is not obvious to me if this can be easily modified to ignore introns. Is this possible Brad? I wasn't aware of Lance's rival bam_to_bigwig tool in the ToolShed till now, and that does talk about this issue. It has a boolean option to ignore gaps when computing coverage, recommended for RNA-Seq where reads are mapped across long splice junctions. Lance, from your tool's help it sounds like it needs a genome database build filled in. I don't understand this requirement - Brad's tool works just fine for standalone BAM files (for example reads mapped to an in house assembly). Is that not supported in your tool? Galaxy team - why does the ToolShed allow duplicate repository names (here bam_to_bigwig) AND duplicate tool IDs (again, here bam_to_bigwig)? Won't this cause chaos when sharing workflows? I would suggest checking this when a tool is uploaded and rejecting repository name or tool ID clashes. Regards, Peter P.S. Brad, your tool is missing an explicit <requirements> tag listing the UCSC binary wigToBigWig, and the Python library pysam. Lance, your tool doesn't seem to include any author information like your name or email address. I'm inferring it is yours from the Galaxy tool shed user id, lparsons.

4 11

run job as real user error: environment variables issue?
by Louise-Amélie Schmitt 20 Apr '12

20 Apr '12

Hi everyone, I'm currently trying to set up our local Galaxy so it can run jobs as the real user. I followed the documentation and set the galaxy user as a sudoer. However, I get an error message whenever I'm trying to run a job: galaxy.jobs.runners.drmaa ERROR 2012-04-19 14:57:48,376 Uncaught exception queueing job Traceback (most recent call last): File "/g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py", line 133, in run_next self.queue_job( obj ) File "/g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py", line 219, in queue_job job_id = self.external_runjob(filename, job_wrapper.user_system_pwent[2]).strip() File "/g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py", line 427, in external_runjob raise RuntimeError("External_runjob failed (exit code %s)\nCalled from %s:%d\nChild process reported error:\n%s" % (str(exitcode), __filename__(), __lineno__(), stderrdata)) RuntimeError: External_runjob failed (exit code 127) Called from /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py:427 Child process reported error: python: error while loading shared libraries: libpython2.6.so.1.0: cannot open shared object file: No such file or directory Looking closely, it's the non-root user it tries to switch to that doesn't have the LD_LIBRARY_PATH properly set, so there should be an environment inheritance issue. However, I tried to print stuff from the scripts/drmaa_external_runner.py script in EVERY WAY I could think of, to no avail. As if it doesn't even run. Which is surprising since root can run python properly, so it really looks like it's really changing users. I really fail to see where the problem could come from, so if you have leads to suggest, I'll be forever grateful. Best, L-A

1 1

Galaxy not updating usage history after data deletion
by Eiseart Joseph F Dunne 20 Apr '12

20 Apr '12

Hi there, I'm having an issue that I believe is commonplace amongst Galaxy users: that of the webserver failing to update deleted data promptly. I've tried using the Python script to force an update that was suggested somewhere on the Wiki, I believe, but to no avail. In terms of making sure everything is "deleted permanently" etc., I'm pretty sure I've done it properly. I'm using the "Main" server & it's been, perhaps, two weeks since I've permanently deleted the said data. Any help is appreciated! Many thanks, Eiseart

1 0