December 2013 - galaxy-dev - lists.galaxyproject.org

user specific access/options
by Jennifer Jackson 19 May '14

19 May '14

Hi Petr, I am going to forward your email to the galaxy-dev list so that the development community can offer comments/suggestions. Best, Jen Galaxy team On 8/30/11 2:27 AM, Petr Novak wrote: > Hi everybody, > I am developing the application on Galaxy server. One of the requirement > is to create user specific list of options. Is it possible to access > somehow $__user_email__ in <options> tag or in <conditional> ?. I did > not found documentation how to used cheetah in galaxy tool xml files but > from files provided with galaxy, cheetah is used only in <command> and > <config> tag. Is that rigth? If it can be used in any part of xml > definition file it would make much easier to generate xml dynamicaly > based on the $__user_email__ > Does anybody have an idea how to manage this problem? > Petr Novak > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/Support

3 2

Default value for data_column not working?
by Peter Cock 14 May '14

14 May '14

Hi all, I'm trying to write a new tool working with tabular data (specifically a Reciprocal Best Hits (RBH) tool using BLAST style tabular output). I want the user to be able to choose a column number (from one of the input files), but I have a default column in mind. However, Galaxy doesn't seem to obey the default column number: <inputs> <param name="a_vs_b" type="data" format="tabular" label="Hits from querying A against B" description="Tabular file, e.g. BLAST output" /> <param name="b_vs_a" type="data" format="tabular" label="Hits from querying B against A" description="Tabular file, e.g. BLAST output" /> <param name="id1" type="data_column" data_ref="a_vs_b" multiple="False" numerical="False" value="1" label="Column containing query ID" help="This is column 1 in standard BLAST tabular output" /> <param name="id2" type="data_column" data_ref="a_vs_b" multiple="False" numerical="False" value="2" label="Column containing match ID" help="This is column 2 in standard BLAST tabular output"/> <param name="score" type="data_column" data_ref="a_vs_b" multiple="False" numerical="False" value="12" label="Column containing containing score to rank on" help="The bitscore is column 12 in standard BLAST tabular output"/> </inputs> I've tried giving the default column value numerically (as shown), and also using value="c2" etc. Regardless, Galaxy just defaults to the first column. Is this a bug, or am I doing something wrong? Thanks, Peter

3 9

history item status not updating
by Robert Baertsch 12 May '14

12 May '14

I submitted a workflow that in turn submits a drm job to sun grid engine. The queue had an error (probably due to a problem with automount). Traceback (most recent call last): File "/inside/depot4/galaxy/lib/galaxy/jobs/runners/__init__.py", line 60, in run_next method(arg) File "/inside/depot4/galaxy/lib/galaxy/jobs/runners/drmaa.py", line 169, in queue_job external_job_id = self.ds.runJob(jt) File "/inside/depot4/galaxy/eggs/drmaa-0.4b3-py2.7.egg/drmaa/__init__.py", line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File "/inside/depot4/galaxy/eggs/drmaa-0.4b3-py2.7.egg/drmaa/helpers.py", line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "/inside/depot4/galaxy/eggs/drmaa-0.4b3-py2.7.egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) DeniedByDrmException: code 17: error: no suitable queues When the sysadmin cleared the error, the job started running normally after being in an error state for 10 minutes. The cool thing is that galaxy kept running without a problem. galaxy.jobs.runners.drmaa DEBUG 2013-06-27 14:51:22,968 (10481/4767487) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2013-06-27 15:00:23,628 (10481/4767487) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2013-06-27 15:00:27,751 (10481/4767487) state change: job is running However in the history panel, the job shows as queued but not running, even if I refresh the history panel. Is this normal or should the status change to running? I'm using this version of straight galaxy: changeset: 10162:f295092476c7 branch: stable parent: 10160:2efb1083676b user: Dannon Baker <dannonbaker(a)me.com> date: Sat Jun 15 09:08:09 2013 -0400 summary: Fix reports import issue reported by Lance, https://trello.com/card/bug-in-reports-webapp-imports/506338ce32ae458f6d15e… -Robert Baertsch UC Santa Cruz

3 3

Help with cluster setup
by Jurgens de Bruin 01 May '14

01 May '14

Hi, This is my first Galaxy installation setup so apologies for stupid questions. I am setting up Galaxy on a Cluster running Torque as the resource manager. I am working through the documentation but I am unclear on some things: Firstly I am unable to find : *start_job_runners *within the universe_wsgi.ini and I dont want to just add this anywhere - any help on this would be create. Further more this is my job_conf.xml : <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="hpc" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/> </plugins> <handlers> <!-- Additional job handlers - the id should match the name of a [server:<id>] in universe_wsgi.ini. <handler id="cn01"/> <handler id="cn02"/> </handlers> <destinations> <destination id="hpc" runner="drmaa"/> </destinations> </job_conf> Does this look meaning full, further more where to I set the additional server:<id> in the universe_wsgi.ini. As background the cluster has 13 compute nodes and a shared storage array that can be accessed by all nodes in the cluster. Thanks again -- Regards/Groete/Mit freundlichen Grüßen/recuerdos/meilleures salutations/ distinti saluti/siong/duì yú/привет Jurgens de Bruin

4 8

Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
by Guest, Simon 25 Apr '14

25 Apr '14

Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions. Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool. We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load. So instead of the tool_dependencies.xml fragment: <tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency> We have something like this (NB: element and attribute names are for illustrative purposes only): <tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency> This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one. We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example). What do you think? cheers, Simon PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-) ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

4 7

registration problem
by Wesley Schaal 22 Apr '14

22 Apr '14

Galaxy-Dev, I have a new galaxy installation under Linux. It's running as an Apache proxy (subdirectory) with authentication through galaxy (not apache). When trying to register a new user (which will be the administrator), the registration page declare success ("Now logged in as foo") but I am still just an anonymous user. If I try to login, it appears to succeed but I'm still anonymous. If I purposely enter the wrong password, the system complains as it should but I never get proper credentials. If I take away the Apache proxy configuration and run on the internal server, the problem goes away: I can login as expected and get the correct environment. The error is clearly here but I feel like I've followed the procedure written on the wiki. Specifically, Apache ----------- RewriteEngine on RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy(.*) http://localhost:8082$1 [P] # 8080 and 8081 are busy Galaxy --------- [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy [app:main] filter-with = proxy-prefix cookie_path = /galaxy Any idea of what might be wrong? Don't know if it matters but I'm using a MySQL database. I can see new users and it seems to work well enough to complain when bad passwords are entered. -- Wes

3 2

Jobs deleted staying in 'dr' status
by Mathieu Bahin 10 Apr '14

10 Apr '14

Hi all, We have been developing our own Galaxy instance for a while now. We have a cluster on which the job are sent to be executed, it is managed through SGE. Usually, communication between SGE and DRMAA is ok and we don't have any problem with that. When a job is deleted by the user, most of the times, the job disappears but sometimes, we don't know why, the job stays and has the status 'dr' within SGE. If we don't kill it 'manually', it stays forever. It is not always the same tools which produces this error . Have you any idea why how manage it ? We have another problem, a display one. Since the last update, we experienced problem with the history. Launching a job, it was very long to appear in the history and we had to refresh. We got a more recent version of firefox and it seems that we don't have the problem anymore. But some user of ours can't update their firefox. Are you aware of that problem ? Cheers, Genouest Platform, Rennes - FRANCE

4 3

Galaxy packages for Bio-Linux - an update + Apache proxying
by Tim Booth 04 Apr '14

04 Apr '14

Hi All, As previously mentioned, I'm back working on packaging the Galaxy server as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately pushing towards something that could be Debian compliant. There's a way to go in that regard, but I do now have an updated package for Bio-Linux in final testing and it also has a new trick: doing "apt-get install galaxy-server-apache-proxy" will set up just that with no further configuration needed. The galaxy server appears at http://localhost/galaxy and users log in with their regular system username and password. Uploads are enabled via regular SFTP so no special FTP server configuration is needed. It's a little hacky in parts but I'm generally pleased with the result. If anyone want to take a look I'd welcome comments. It's not in the main BL repo yet but can be found here: https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-arc… Cheers, TIM -- Tim Booth <tbooth(a)ceh.ac.uk> NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705

4 7

Re: [galaxy-dev] Plugins
by Michael E. Cotterell 12 Mar '14

12 Mar '14

Over next couple days, I'll take a look at what's in lib/galaxy/web/base/pluginframework.py and see if there is a way to refactor it with my stuff in order to get a general plugin framework (galaxy.pluginframework) as proposed by James. Sorry for the delay. Teaching a math class this semester is really eating up my time. Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/ On Tuesday, September 3, 2013 at 12:45 PM, Carl Eberhard wrote: > It is web-specific and only handles the mako/static serving cases. I think the only areas we were generalizing at this point were the directory structures and main configuration. > > Agreed on more generalization and the hook system. Michael's stuff is a good direction to move in. > > I know John was looking to use it for dynamic job destinations and dynamic toolbox filters. I'd like to see some lab/community discussion on (even broad stroke) functional requirements and use cases. > > > On Tue, Sep 3, 2013 at 12:27 PM, James Taylor <james(a)jamestaylor.org (mailto:james@jamestaylor.org)> wrote: > > Carl, is what you have now totally web specific? It would be great to > > have a general plugin framework (galaxy.pluginframework) which the web > > stuff could perhaps extend? > > > > > > On Tue, Sep 3, 2013 at 10:56 AM, Carl Eberhard <carlfeberhard(a)gmail.com (mailto:carlfeberhard@gmail.com)> wrote: > > > Heya, Michael > > > > > > Right now our only plugin code is located in > > > lib/galaxy/web/base/pluginframework.py. It's only purpose so far is to: > > > - allow locally modified/created code to serve mako templates and static > > > pages/resources from paste. > > > - serve as a super class for the visualization framework. > > > > > > I think your git repo has some great ideas. > > > > > > Carl > > > > > > > > > On Fri, Aug 30, 2013 at 2:25 PM, Michael Cotterell <mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com)> > > > wrote: > > > > > > > > Is this plugin code hosted anywhere (like in central)? > > > > > > > > > > > > On Monday, August 26, 2013, James Taylor wrote: > > > > > > > > > > Michael, > > > > > > > > > > Carl has been working on a plugin framework for Galaxy, and John > > > > > Chilton has made some improvements. It would be great if you could > > > > > look over what each other has done and come up with some suggestions. > > > > > I personally am in favor of a very generic hook system where a hook > > > > > can be defined anywhere in the code and then the user can provide any > > > > > callable. > > > > > > > > > > I'm fond of the way sup implemented hooks, see the use of HookManager > > > > > here: https://github.com/sup-heliotrope/sup/blob/develop/lib/sup/time.rb > > > > > > > > > > Your approach is quite similar and I like it. > > > > > > > > > > This makes it easy to have a script that can list all available hooks > > > > > with their documentation. > > > > > > > > > > > > > > > -- > > > > > James Taylor, Assistant Professor, Biology/CS, Emory University > > > > > > > > > > > > > > > On Fri, Aug 23, 2013 at 10:29 AM, Michael E. Cotterell > > > > > <mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com)> wrote: > > > > > > James, > > > > > > > > > > > > Has there been any more talk about this? I'd be willing to work in my > > > > > > reference implementation into central as a pull request, but I don't want to > > > > > > do that unless there is an agreed upon roadmap or something. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Sincerely, > > > > > > Michael E. Cotterell > > > > > > > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > > > > Instructor of Record, Graduate RA & TA, University of Georgia > > > > > > Department Liaison, CS Graduate Student Association, University of > > > > > > Georgia > > > > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) > > > > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) > > > > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) > > > > > > http://michaelcotterell.com/ > > > > > > > > > > > > > > > > > > On Tuesday, July 16, 2013 at 2:08 PM, Michael E. Cotterell wrote: > > > > > > > > > > > > > James, > > > > > > > > > > > > > > That's exactly what I was thinking. In my opinion, hooks are the way > > > > > > > to go (as seen in my example), but if there's a better way then I'm game for > > > > > > > that to. I just want to be able to extend Galaxy without modifying Galaxy > > > > > > > itself. > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Sincerely, > > > > > > > Michael E. Cotterell > > > > > > > > > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > > > > > Instructor of Record, Graduate RA & TA, University of Georgia > > > > > > > Faculty Liaison, CS Graduate Student Association, University of > > > > > > > Georgia > > > > > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) > > > > > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) > > > > > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) > > > > > > > http://michaelcotterell.com/ > > > > > > > > > > > > > > > > > > > > > On Tuesday, July 16, 2013 at 2:05 PM, James Taylor wrote: > > > > > > > > > > > > > > > Michael, > > > > > > > > > > > > > > > > Galaxy is all about plugins! Tools, datatypes, batch systems, > > > > > > > > visualizations, et cetera. > > > > > > > > > > > > > > > > Additional ways to extend Galaxy through plugins are very welcome. > > > > > > > > We > > > > > > > > just need to make sure that whatever solution is chosen is extremely > > > > > > > > general. > > > > > > > > > > > > > > > > -- > > > > > > > > James Taylor, Assistant Professor, Biology/CS, Emory University > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 16, 2013 at 1:57 PM, Michael E. Cotterell > > > > > > > > <mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com)> wrote: > > > > > > > > > Are there any plans in the works to support plugins within Galaxy? > > > > > > > > > One of the tools I'm working on is actually more of an extension of the > > > > > > > > > workflow canvas/editor's user interface. Right now, I've minimized the > > > > > > > > > modifications required to the editor.mako file to simply adding a mako > > > > > > > > > include near the bottom of the file. However, like I just said, this > > > > > > > > > requires editing a Galaxy file. It would be a lot better if Galaxy supported > > > > > > > > > plugins with hooks, thus allowing me to extend the user interface (or any > > > > > > > > > other part of the application) by registering hooks in a plugin. > > > > > > > > > > > > > > > > > > To demonstrate how easy it would be to integrate plugins into a > > > > > > > > > python application, I've written up a simple plugin manager script (~100 > > > > > > > > > loc) an placed it in a gist: https://gist.git > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Sincerely, > > > > Michael E. Cotterell > > > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > > Graduate RA & TA, University of Georgia > > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) > > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) > > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) > > > > http://michaelcotterell.com/ > > > > > >

2 4

Difficulty dispatching toolshed tool jobs via job_conf.xml
by Guest, Simon 27 Feb '14

27 Feb '14

I am having trouble getting a toolshed tool to be dispatched to the destination I list in the job_conf.xml file. My job_conf.xml file looks like this: <?xml version="1.0"?> <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="20"/> <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" /> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="condor"> <destination id="local" runner="local"/> <destination id="condor" runner="condor">  </destination> </destinations> <tools> <tool id="upload1" destination="local"/>  <tool id="ucsc_table_direct1" destination="local"/>  ... stuff omitted ... <tool id="toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS: infoseq46/5.0.0" destination="local"/> </tools> </job_conf> So, you can see I have a default destination of condor, but I'm trying to run my toolshed EMBOSS infoseq tool on local. However, it is stubbornly running on condor. In lib/galaxy/tools/__init__.py:1132, I see this comment which got me wondering: # In the toolshed context, there is no job config. Is it possible to define tool destinations for toolshed tools? Are there some gotchas that I should know about? Any other ideas why my job is ignoring the config in job_conf.xml? (By the way, I can change say the upload1 tool to run on Condor by setting its destination in that file, so it is doing something.) The other thing I saw in the source code is stuff about old_id and toolshed guids. Do I need to understand this stuff? The paster.log contains the following when I submit the infoseq job: 147.158.130.216 - - [11/Sep/2013:13:53:34 +1300] "GET /tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0 HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" 147.158.130.216 - - [11/Sep/2013:13:53:40 +1300] "POST /tool_runner/index HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/tool_runner?tool_id=toolshed-dev.agresea…" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" galaxy.jobs DEBUG 2013-09-11 13:53:40,886 (92) Working directory for job is: /home/galaxy-dev/galaxy/database/job_working_directory/000/92 galaxy.tools DEBUG 2013-09-11 13:53:40,886 Tool::get_job_destination: {'runner': 'condor', 'legacy': False, 'params': {}, 'tags': None, 'url': None, 'converted': False, 'id': 'condor'}. galaxy.jobs.handler DEBUG 2013-09-11 13:53:40,894 (92) Dispatching to condor runner (I added the debug output for Tool::get_job_destination to see what was going on.) Any ideas? cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

3 8