October 2013 - galaxy-dev - lists.galaxyproject.org

Help with cluster setup
by Jurgens de Bruin 01 May '14

01 May '14

Hi, This is my first Galaxy installation setup so apologies for stupid questions. I am setting up Galaxy on a Cluster running Torque as the resource manager. I am working through the documentation but I am unclear on some things: Firstly I am unable to find : *start_job_runners *within the universe_wsgi.ini and I dont want to just add this anywhere - any help on this would be create. Further more this is my job_conf.xml : <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="hpc" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/> </plugins> <handlers> <!-- Additional job handlers - the id should match the name of a [server:<id>] in universe_wsgi.ini. <handler id="cn01"/> <handler id="cn02"/> </handlers> <destinations> <destination id="hpc" runner="drmaa"/> </destinations> </job_conf> Does this look meaning full, further more where to I set the additional server:<id> in the universe_wsgi.ini. As background the cluster has 13 compute nodes and a shared storage array that can be accessed by all nodes in the cluster. Thanks again -- Regards/Groete/Mit freundlichen Grüßen/recuerdos/meilleures salutations/ distinti saluti/siong/duì yú/привет Jurgens de Bruin

4 8

Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
by Guest, Simon 25 Apr '14

25 Apr '14

Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions. Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool. We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load. So instead of the tool_dependencies.xml fragment: <tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency> We have something like this (NB: element and attribute names are for illustrative purposes only): <tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency> This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one. We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example). What do you think? cheers, Simon PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-) ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

4 7

registration problem
by Wesley Schaal 22 Apr '14

22 Apr '14

Galaxy-Dev, I have a new galaxy installation under Linux. It's running as an Apache proxy (subdirectory) with authentication through galaxy (not apache). When trying to register a new user (which will be the administrator), the registration page declare success ("Now logged in as foo") but I am still just an anonymous user. If I try to login, it appears to succeed but I'm still anonymous. If I purposely enter the wrong password, the system complains as it should but I never get proper credentials. If I take away the Apache proxy configuration and run on the internal server, the problem goes away: I can login as expected and get the correct environment. The error is clearly here but I feel like I've followed the procedure written on the wiki. Specifically, Apache ----------- RewriteEngine on RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy(.*) http://localhost:8082$1 [P] # 8080 and 8081 are busy Galaxy --------- [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy [app:main] filter-with = proxy-prefix cookie_path = /galaxy Any idea of what might be wrong? Don't know if it matters but I'm using a MySQL database. I can see new users and it seems to work well enough to complain when bad passwords are entered. -- Wes

3 2

Jobs deleted staying in 'dr' status
by Mathieu Bahin 10 Apr '14

10 Apr '14

Hi all, We have been developing our own Galaxy instance for a while now. We have a cluster on which the job are sent to be executed, it is managed through SGE. Usually, communication between SGE and DRMAA is ok and we don't have any problem with that. When a job is deleted by the user, most of the times, the job disappears but sometimes, we don't know why, the job stays and has the status 'dr' within SGE. If we don't kill it 'manually', it stays forever. It is not always the same tools which produces this error . Have you any idea why how manage it ? We have another problem, a display one. Since the last update, we experienced problem with the history. Launching a job, it was very long to appear in the history and we had to refresh. We got a more recent version of firefox and it seems that we don't have the problem anymore. But some user of ours can't update their firefox. Are you aware of that problem ? Cheers, Genouest Platform, Rennes - FRANCE

4 3

Re: [galaxy-dev] Plugins
by Michael E. Cotterell 12 Mar '14

12 Mar '14

Over next couple days, I'll take a look at what's in lib/galaxy/web/base/pluginframework.py and see if there is a way to refactor it with my stuff in order to get a general plugin framework (galaxy.pluginframework) as proposed by James. Sorry for the delay. Teaching a math class this semester is really eating up my time. Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/ On Tuesday, September 3, 2013 at 12:45 PM, Carl Eberhard wrote: > It is web-specific and only handles the mako/static serving cases. I think the only areas we were generalizing at this point were the directory structures and main configuration. > > Agreed on more generalization and the hook system. Michael's stuff is a good direction to move in. > > I know John was looking to use it for dynamic job destinations and dynamic toolbox filters. I'd like to see some lab/community discussion on (even broad stroke) functional requirements and use cases. > > > On Tue, Sep 3, 2013 at 12:27 PM, James Taylor <james(a)jamestaylor.org (mailto:james@jamestaylor.org)> wrote: > > Carl, is what you have now totally web specific? It would be great to > > have a general plugin framework (galaxy.pluginframework) which the web > > stuff could perhaps extend? > > > > > > On Tue, Sep 3, 2013 at 10:56 AM, Carl Eberhard <carlfeberhard(a)gmail.com (mailto:carlfeberhard@gmail.com)> wrote: > > > Heya, Michael > > > > > > Right now our only plugin code is located in > > > lib/galaxy/web/base/pluginframework.py. It's only purpose so far is to: > > > - allow locally modified/created code to serve mako templates and static > > > pages/resources from paste. > > > - serve as a super class for the visualization framework. > > > > > > I think your git repo has some great ideas. > > > > > > Carl > > > > > > > > > On Fri, Aug 30, 2013 at 2:25 PM, Michael Cotterell <mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com)> > > > wrote: > > > > > > > > Is this plugin code hosted anywhere (like in central)? > > > > > > > > > > > > On Monday, August 26, 2013, James Taylor wrote: > > > > > > > > > > Michael, > > > > > > > > > > Carl has been working on a plugin framework for Galaxy, and John > > > > > Chilton has made some improvements. It would be great if you could > > > > > look over what each other has done and come up with some suggestions. > > > > > I personally am in favor of a very generic hook system where a hook > > > > > can be defined anywhere in the code and then the user can provide any > > > > > callable. > > > > > > > > > > I'm fond of the way sup implemented hooks, see the use of HookManager > > > > > here: https://github.com/sup-heliotrope/sup/blob/develop/lib/sup/time.rb > > > > > > > > > > Your approach is quite similar and I like it. > > > > > > > > > > This makes it easy to have a script that can list all available hooks > > > > > with their documentation. > > > > > > > > > > > > > > > -- > > > > > James Taylor, Assistant Professor, Biology/CS, Emory University > > > > > > > > > > > > > > > On Fri, Aug 23, 2013 at 10:29 AM, Michael E. Cotterell > > > > > <mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com)> wrote: > > > > > > James, > > > > > > > > > > > > Has there been any more talk about this? I'd be willing to work in my > > > > > > reference implementation into central as a pull request, but I don't want to > > > > > > do that unless there is an agreed upon roadmap or something. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Sincerely, > > > > > > Michael E. Cotterell > > > > > > > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > > > > Instructor of Record, Graduate RA & TA, University of Georgia > > > > > > Department Liaison, CS Graduate Student Association, University of > > > > > > Georgia > > > > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) > > > > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) > > > > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) > > > > > > http://michaelcotterell.com/ > > > > > > > > > > > > > > > > > > On Tuesday, July 16, 2013 at 2:08 PM, Michael E. Cotterell wrote: > > > > > > > > > > > > > James, > > > > > > > > > > > > > > That's exactly what I was thinking. In my opinion, hooks are the way > > > > > > > to go (as seen in my example), but if there's a better way then I'm game for > > > > > > > that to. I just want to be able to extend Galaxy without modifying Galaxy > > > > > > > itself. > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Sincerely, > > > > > > > Michael E. Cotterell > > > > > > > > > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > > > > > Instructor of Record, Graduate RA & TA, University of Georgia > > > > > > > Faculty Liaison, CS Graduate Student Association, University of > > > > > > > Georgia > > > > > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) > > > > > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) > > > > > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) > > > > > > > http://michaelcotterell.com/ > > > > > > > > > > > > > > > > > > > > > On Tuesday, July 16, 2013 at 2:05 PM, James Taylor wrote: > > > > > > > > > > > > > > > Michael, > > > > > > > > > > > > > > > > Galaxy is all about plugins! Tools, datatypes, batch systems, > > > > > > > > visualizations, et cetera. > > > > > > > > > > > > > > > > Additional ways to extend Galaxy through plugins are very welcome. > > > > > > > > We > > > > > > > > just need to make sure that whatever solution is chosen is extremely > > > > > > > > general. > > > > > > > > > > > > > > > > -- > > > > > > > > James Taylor, Assistant Professor, Biology/CS, Emory University > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 16, 2013 at 1:57 PM, Michael E. Cotterell > > > > > > > > <mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com)> wrote: > > > > > > > > > Are there any plans in the works to support plugins within Galaxy? > > > > > > > > > One of the tools I'm working on is actually more of an extension of the > > > > > > > > > workflow canvas/editor's user interface. Right now, I've minimized the > > > > > > > > > modifications required to the editor.mako file to simply adding a mako > > > > > > > > > include near the bottom of the file. However, like I just said, this > > > > > > > > > requires editing a Galaxy file. It would be a lot better if Galaxy supported > > > > > > > > > plugins with hooks, thus allowing me to extend the user interface (or any > > > > > > > > > other part of the application) by registering hooks in a plugin. > > > > > > > > > > > > > > > > > > To demonstrate how easy it would be to integrate plugins into a > > > > > > > > > python application, I've written up a simple plugin manager script (~100 > > > > > > > > > loc) an placed it in a gist: https://gist.git > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Sincerely, > > > > Michael E. Cotterell > > > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > > Graduate RA & TA, University of Georgia > > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) > > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) > > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) > > > > http://michaelcotterell.com/ > > > > > >

2 4

Difficulty dispatching toolshed tool jobs via job_conf.xml
by Guest, Simon 27 Feb '14

27 Feb '14

I am having trouble getting a toolshed tool to be dispatched to the destination I list in the job_conf.xml file. My job_conf.xml file looks like this: <?xml version="1.0"?> <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="20"/> <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" /> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="condor"> <destination id="local" runner="local"/> <destination id="condor" runner="condor">  </destination> </destinations> <tools> <tool id="upload1" destination="local"/>  <tool id="ucsc_table_direct1" destination="local"/>  ... stuff omitted ... <tool id="toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS: infoseq46/5.0.0" destination="local"/> </tools> </job_conf> So, you can see I have a default destination of condor, but I'm trying to run my toolshed EMBOSS infoseq tool on local. However, it is stubbornly running on condor. In lib/galaxy/tools/__init__.py:1132, I see this comment which got me wondering: # In the toolshed context, there is no job config. Is it possible to define tool destinations for toolshed tools? Are there some gotchas that I should know about? Any other ideas why my job is ignoring the config in job_conf.xml? (By the way, I can change say the upload1 tool to run on Condor by setting its destination in that file, so it is doing something.) The other thing I saw in the source code is stuff about old_id and toolshed guids. Do I need to understand this stuff? The paster.log contains the following when I submit the infoseq job: 147.158.130.216 - - [11/Sep/2013:13:53:34 +1300] "GET /tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0 HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" 147.158.130.216 - - [11/Sep/2013:13:53:40 +1300] "POST /tool_runner/index HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/tool_runner?tool_id=toolshed-dev.agresea…" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" galaxy.jobs DEBUG 2013-09-11 13:53:40,886 (92) Working directory for job is: /home/galaxy-dev/galaxy/database/job_working_directory/000/92 galaxy.tools DEBUG 2013-09-11 13:53:40,886 Tool::get_job_destination: {'runner': 'condor', 'legacy': False, 'params': {}, 'tags': None, 'url': None, 'converted': False, 'id': 'condor'}. galaxy.jobs.handler DEBUG 2013-09-11 13:53:40,894 (92) Dispatching to condor runner (I added the debug output for Tool::get_job_destination to see what was going on.) Any ideas? cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

3 8

Blast+: new feature filter/mask... by taxid before job in one task
by Berner, Thomas 26 Feb '14

26 Feb '14

Hi guys, We recently installed NCBI BLAST + to our local Galaxy instance and now we need to provide the possibility to filter/mask ... by taxon id (taxid) using the command line option (-window_masker_taxid) of BLAST (cf. http://www.ncbi.nlm.nih.gov/books/NBK1763/) before the job will be performed in one task. However searching in the documentation and the mailing list, I did not find anything about this. So it would be great to provide the possibility to subselect by taxid (e.g. NCBI BLAST server http://blast.ncbi.nlm.nih.gov/Blast.cgi) by changing the BLAST integration. We believe this could be of common interest. It would be great if anybody could comment on this. Thanks a lot, Thomas

3 4

Galaxy Tool Shed packages for Biopython
by Peter Cock 18 Feb '14

18 Feb '14

Hi all, I've send this to both the Galaxy and Biopython developers lists, and hope this will make sense to both groups. If you've not heard of Galaxy, start here: http://galaxyproject.org - while the easy to guess Biopython website is at http://biopython.org Brad Chapman and I are both Biopython core developers, and are also both on the "IUC" Galaxy Tool Shed committee because we've been quite involved in wrapping and writing tools for use on Galaxy. Fellow committee member Björn Grüning has done a lot of the hands on work defining package definitions for dependencies within the Galaxy Tool Shed ecosystem - including defining them for Biopython, NumPy, SciPy, MatPlotLib, etc. We're very grateful for his hard work - most of which is now available under the IUC group account: http://toolshed.g2.bx.psu.edu/view/iuc/ http://testtoolshed.g2.bx.psu.edu/view/iuc/ The Biopython packages, however, are under a dedicated "biopython" account on the Galaxy Tool Shed to which currently Bjoern, Brad and I have access to: http://toolshed.g2.bx.psu.edu/view/biopython/ http://testtoolshed.g2.bx.psu.edu/view/biopython/ This packaging work was initially tracked in Bjoern's own GitHub repository, https://github.com/bgruening/galaxytools/ We (me, Brad and Bjoern) agreed that a Biopython owned repository would be more sensible in the long term, so I have created this and ported Bjoern's commits to it: https://github.com/biopython/galaxy_packages Currently the "Galaxy packagers" team on GitHub which has read and write access to this new repository is just me, Brad and Bjoern. Regards, Peter

2 3

shed tool dependencies
by Geert Vandeweyer 13 Feb '14

13 Feb '14

Hi, Is there a default notation to specify perl modules (threading modules) in a tool configuration or in a submission to the toolshed? I've got some tools to share that make extensive use of these modules, and they are not default in a perl distribution. Hence, I'd like to inform the users that they need to install these modules (or that they get installed by the the toolshed using cpan ?) Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

4 3

Several jobs stalled in the queue: Job runner assigned but no external ID recorded
by Derrick Lin 04 Feb '14

04 Feb '14

Hi guys, I have several jobs have been stuck in the queue for long time, I checked the handler log and found the following message: Job runner assigned but no external ID recorded, adding to the job handler queue Is there a way that I can clean these orphan jobs out of the queue? Cheers, Derrick

1 1