November 2013 - galaxy-dev - lists.galaxyproject.org

Running jobs as real user and extra_file_path
by Louise-Amélie Schmitt 28 Apr '17

28 Apr '17

Hi everyone, I just wanted to ask how the extra_file_path is handled in case of job running as the real user since the file_path is only writable by the galaxy user. Any clue? Thanks, L-A

3 3

error loading files into galaxy
by Hakeem Almabrazi 04 Mar '16

04 Mar '16

Hi, I started getting the following error whenever I try to load a file into Galaxy local. Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 8, in <module> from galaxy import eggs ImportError: cannot import name eggs I will appreciate if someone can tell me what could happen to cause such issue and how to resolve it. Regards,

2 1

Upload issue in local installation
by Batsal Devkota 05 Feb '16

05 Feb '16

I installed galaxy locally in a linux server. However, I cannot upload the files (no matter how small, I have tried few kb size fasta files). When I try to upload, the link to the file shows up in the History and gets a new number (purple box). When I click on the link I get 'Dataset is uploading' forever. In the terminal window where I start galaxy, I get the following error report: 92.17.41.13 - - [02/Aug/2012:15:33:32 -0400] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:32 -0400] "GET /root/tool_menu HTTP/1.1" 200 - "http://redhat:8080/" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:32 -0400] "GET /history HTTP/1.1" 200 - "http://redhat:8080/" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:33 -0400] "POST /root/user_get_usage HTTP/1.1" 200 - "http://redhat:8080/history" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:40 -0400] "GET /tool_runner?tool_id=upload1 HTTP/1.1" 200 - "http://redhat:8080/root/tool_menu" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:59 -0400] "POST /tool_runner/upload_async_create HTTP/1.1" 200 - "http://redhat:8080/" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:59 -0400] "GET /tool_runner/upload_async_message HTTP/1.1" 200 - "http://redhat:8080/" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:59 -0400] "GET /history HTTP/1.1" 200 - "http://redhat:8080/tool_runner/upload_async_message" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:33:59 -0400] "POST /root/user_get_usage HTTP/1.1" 200 - "http://redhat:8080/history" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:34:04 -0400] "POST /root/history_item_updates HTTP/1.1" 200 - "http://redhat:8080/history" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" 92.17.41.13 - - [02/Aug/2012:15:34:08 -0400] "POST /root/history_item_updates HTTP/1.1" 200 - "http://redhat:8080/history" "Mozilla/5.0 (Windows NT 5.2; rv:13.0) Gecko/20100101 Firefox/13.0.1" The last line keeps going for ever... writes new line every 4 secs. I am stuck and don't know where to look. Please help. Batsal.

6 7

filtering a <param> of type 'data' so only one type is available
by Dan Tenenbaum 24 Oct '14

24 Oct '14

Hi, I have a tool wrapper with a <param> of type="data". Currently, this renders as a text box with a drop down list that has 52 items in it (the number of things I have in my history, I guess). I'd like to filter this list so that only some items in the history (in my case any item whose name ends with '.rda') are shown in this dropdown. It seems like maybe the 'format' parameter to the 'param' tag is what I want, but: 1) I tried format="rda" and that didn't seem to change anything. 2) "rda" is not in the datatypes.conf file which the documentation suggests is required? (.rda is an extension used for serialized R objects.) The tool I'm working on only accepts rda files as input and the reason I'm asking for this is that it is all too easy to accidentally feed it a file of some other type. If the dropdown could be filtered so that only items which will work with the tool are shown, that would be great. Is there a way to do that? Thanks, Dan

4 5

Workflow param error leads to template error
by Paul-Michael Agapow 19 Sep '14

19 Sep '14

Looking for pointers on what might be causing this problem: A user had a moderately complicated workflow that when run under certain parameter values results in a traceback that ends: Module workflow_run_mako:232 in render_body >> for ic in oc.input_step.module.get_data_inputs(): AttributeError: 'WorkflowStep' object has no attribute 'module' After some puzzling, I realized one of the parameters was outside the allowed range (a max on an integer param) and that Galaxy was trying to render "run.mako" flagging up the error but erroring out. Why Galaxy tries to call this non-existent member is unclear to me. Any insight or places I should start exploring? Galaxy version: various (started with an 6-month old one, upgraded to latest production to see if it would fix error, it didn't) Hosting OS: CentOS 6 ------ Paul Agapow (paul-michael.agapow(a)hpa.org.uk) Bioinformatics, Health Protection Agency (UK) ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************

3 3

Re: [galaxy-dev] Job execution order mixed-up
by Jean-Francois Payotte 04 Jul '14

04 Jul '14

Hi John, Thank you for your answer and for trying to help. This is greatly appreciated! I didn't really made any progress in tracking down this error, and hopefully this weird behaviour will not happen anymore with the November 4th, distribution. But here are my answers to your questions, in case it would ring a bell: Has this behaviour been reported with any other workflow? It has been reported with 2 different workflows as of now. These 2 workflows doesn't have anything in common, except that they are huge (one of them has 37 steps, producing a total of about 110 datasets). Are you running Galaxy as a single process or multiple processes? If multiple processes, how many web, handler and manager processes do you have and are they all on the same machine? We are running Galaxy in multiple processes with 5 web servers, 3 job handlers and no manager (I believe the manager was rendered obsolete in one of the latest Galaxy distributions). All these processes are run on the same machine. Have you made any modifications to Galaxy that could result in this behaviour? No. What is the value of track_jobs_in_database in your universe_wsgi.ini configuration file? We never touched this part of the configuration file and the line still reads: "#track_jobs_in_database = None". After reading your answer, I've decided to modify this line to: "track_jobs_in_database = True" Unfortunately, running one of the faulty workflows several times (5x), I noticed that one of them was still showing this strange behaviour where some jobs were executed before their inputs were ready. Do you think this issue could be related to the fact that we are using Galaxy with the multiple processes configuration? We implemented this configuration some time ago because some of our users were complaining about the slow responsiveness of the web interface. Would you recommend using Galaxy without the multiple processes configuration? (Lets say if updating to November 4th distribution doesn't fix this issue) I guess you are probably using the multiple processes configuration as well on Galaxy main? Thanks again for your help! Jean-François ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Posted by John Chilton on Nov 09, 2013; 2:50pm Hello Jean-François, Have you made any progress tracking down this error? This appears very serious, but to tell you the truth I have no clue what could cause it. The distribution you are using is pretty old at this point I feel like if it was a bug the exhibited under relatively standard parameter combinations someone else would have reported it by now. Can you tell me some things: has this been reported with any other workflows? Is there anything special about this workflow? Can you rebuild the workflow and see if the error occurs again? Additional questions if the problem is not restricted to the workflow: are you running Galaxy as a single process or multiple processes? If multiple processes, how many web, handler, and manager processes do you have? Are they all on the same machine? Have you made any modifications to Galaxy that could result in this behavior? What is the value of track_jobs_in_database in your universe_wsgi.ini configuration file? -John On Thu, Nov 7, 2013 at 10:34 AM, Jean-Francois Payotte <[hidden email]> wrote: Dear Galaxy mailing-list, Once again I come seeking for your help. I hope someone already had this issue or will have an idea on where to look to solve it. :) One of our users reported having workflows failing because some steps were executed before all their inputs where ready. You can find a screenshot attached, where we can see that step (42) "Sort on data 39" has been executed while step (39) is still waiting to run (gray box). This behaviour has been reproduced with at least two different Galaxy tools (one custom, and the sort tool which comes standard with Galaxy). This behaviour seems to be a little bit random, as running two times a workflow where this issue occurs, only one time did some steps were executed in the wrong order. I could be wrong, but I don't think this issue is grid-related as, from my understanding, Galaxy is not using SGE job dependencies functionality. I believe all jobs stays in some internal queues (within Galaxy) until all input files are ready, and only then the job is submitted to the cluster. Any help or any hint on what to look at to solve this issue would be greatly appreciated. We have updated our Galaxy instance to August 12th distribution on October 1st, and I believe we never experienced this issue before the update. Many thanks for your help, Jean-François

5 5

Using Mesos to Enable distributed computing under Galaxy?
by Kyle Ellrott 17 Jun '14

17 Jun '14

I think one of the aspects where Galaxy is a bit soft is the ability to do distributed tasks. The current system of split/replicate/merge tasks based on file type is a bit limited and hard for tool developers to expand upon. Distributed computing is a non-trival thing to implement and I think it would be a better use of our time to use an already existing framework. And it would also mean one less API for tool writers to have to develop for. I was wondering if anybody has looked at Mesos ( http://mesos.apache.org/) You can see an overview of the Mesos architecture at https://github.com/apache/mesos/blob/master/docs/Mesos-Architecture.md The important thing about Mesos is that it provides an API for C/C++, Java/Scala and Python to write distributed frameworks. There are already implementations of frameworks for common parallel programming systems such as: - Hadoop (https://github.com/mesos/hadoop) - MPI ( https://github.com/apache/mesos/blob/master/docs/Running-torque-or-mpi-on-m… ) - Spark (http://spark-project.org) And you can find example Python framework at https://github.com/apache/mesos/tree/master/src/examples/python Integration with Galaxy would have three parts: 1) Add a system config variable to Galaxy called 'MESOS_URL' that is then passed to tool wrappers and allows them to contact the local mesos infrastructure (assuming the system has been configured) or pass a null if the system isn't available. 2) Write a tool runner that works as a mesos framework to executes single cpu jobs on the distributed system. 3) For instances where mesos is not available at a system wide level (say they only have access to an SGE based cluster), but the user wants to run distributed jobs, write a wrapper that can create a mesos cluster using the existing queueing system. For example, right now I run a Mesos system under the SGE queue system. I'm curious to see what other people think. Kyle

5 8

Old Tool Shed URL http://community.g2.bx.psu.edu/ dead
by Peter Cock 28 May '14

28 May '14

Hello all, Either I made a typo in some old README files, or the old URL for the Tool Shed http://community.g2.bx.psu.edu/ has stopped working. Could this redirect to the current Tool Shed address http://toolshed.g2.bx.psu.edu/ please? Thanks, Peter

3 6

user specific access/options
by Jennifer Jackson 19 May '14

19 May '14

Hi Petr, I am going to forward your email to the galaxy-dev list so that the development community can offer comments/suggestions. Best, Jen Galaxy team On 8/30/11 2:27 AM, Petr Novak wrote: > Hi everybody, > I am developing the application on Galaxy server. One of the requirement > is to create user specific list of options. Is it possible to access > somehow $__user_email__ in <options> tag or in <conditional> ?. I did > not found documentation how to used cheetah in galaxy tool xml files but > from files provided with galaxy, cheetah is used only in <command> and > <config> tag. Is that rigth? If it can be used in any part of xml > definition file it would make much easier to generate xml dynamicaly > based on the $__user_email__ > Does anybody have an idea how to manage this problem? > Petr Novak > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/Support

3 2

Default value for data_column not working?
by Peter Cock 14 May '14

14 May '14

Hi all, I'm trying to write a new tool working with tabular data (specifically a Reciprocal Best Hits (RBH) tool using BLAST style tabular output). I want the user to be able to choose a column number (from one of the input files), but I have a default column in mind. However, Galaxy doesn't seem to obey the default column number: <inputs> <param name="a_vs_b" type="data" format="tabular" label="Hits from querying A against B" description="Tabular file, e.g. BLAST output" /> <param name="b_vs_a" type="data" format="tabular" label="Hits from querying B against A" description="Tabular file, e.g. BLAST output" /> <param name="id1" type="data_column" data_ref="a_vs_b" multiple="False" numerical="False" value="1" label="Column containing query ID" help="This is column 1 in standard BLAST tabular output" /> <param name="id2" type="data_column" data_ref="a_vs_b" multiple="False" numerical="False" value="2" label="Column containing match ID" help="This is column 2 in standard BLAST tabular output"/> <param name="score" type="data_column" data_ref="a_vs_b" multiple="False" numerical="False" value="12" label="Column containing containing score to rank on" help="The bitscore is column 12 in standard BLAST tabular output"/> </inputs> I've tried giving the default column value numerically (as shown), and also using value="c2" etc. Regardless, Galaxy just defaults to the first column. Is this a bug, or am I doing something wrong? Thanks, Peter

3 9