- galaxy-dev - lists.galaxyproject.org

How to speed up submissions of workflows
by Thon Deboer 14 Jun '13

14 Jun '13

Hi, I am routinely running hundreds of pair-end sequencing runs and find that STARTING a workflow on all of these can easily take hours since it seems each workflow preparation takes about 3 minutes, so for doing 128 fastq pairs like this it will take over 3 hours just to get all workflows running.. Is there any way for me to speed up this process, maybe by adding more handlers (although I am suspecting the WEB server is the bottleneck here)? I already run 8 handlers but only one web server. I could loadbalance the webservers through my proxy server.Will that speed up the submissions or are we stuck with this linear process? Thanks, Thon

2 1

formatting output from a java tool
by vijayalakshmi 14 Jun '13

14 Jun '13

Hi all, I am trying to integrate a tool written in java into galaxy. The tool opens a GUI window to output a graph and also writes a text file. I want to know how to output that graph in the middle pane and how to output the text file so that the user will be able to download it. *Sample.java* public class Sample { public static void main(String[] args) { try { int length = args.length; FileWriter fw = new FileWriter(args[length-1]+".txt"); fw.write("length = " + args.length); for(int i=0;i<args.length;i++) fw.write(args[i]); fw.close(); } catch (Exception e) { e.printStackTrace(); } } } *sample.xml* <tool id="sample" name="Compute correlation"> <description> for the given BAM file </description> <command>java -jar sample.jar $sample_type $window_frame_size $iteration $resolution $correlation_window_size $no_of_CPUs $output</command> <inputs> <param name="sample_type" type="select" label="Choose type of sampling"> <option value="random">Random Sampling</option> <option value="whole">Whole Genome</option> </param> <param name="window_frame_size" type="text" area="true" size="1x10" label="Frame size for extraction" help="Default:50000"/> <param name="iteration" type="text" area="true" size="1x10" label="Number of iterations per chromosome" help="Default:10"/> <param name="resolution" type="text" area="true" size="1x10" label="Resolution" help="Default:1"/> <param name="correlation_window_size" type="text" area="true" size="1x10" label="Correlation window Size" help="Default:1001"/> <param name="no_of_CPUs" type="text" area="true" size="1x5" label="Number of CPUs required" help="Default:1"/> </inputs> <outputs> <data format="txt" name="output" size="1x30" area="true" label="output"/> </outputs> <help> This tool helps plot the correlation function for the given input file </help> </tool> Thanks, VJ.

1 0

Download histories
by Doan, Ryan 14 Jun '13

14 Jun '13

Hello, Several people in our lab would like to download all of their histories from their respective Galaxy accounts. We tried to download them using the following steps: Select "Export to File" Copy the provided link and paste into WGet.exe program as follows: wget.exe --no-check-certificate https://main.g2.bx.psu.edu/history/export_archive?id=e0b0f57abd62f2aa -O F:\Galaxy\Ryan\rnd003bucknell.edu\Cattle_seq.galaxy However, we seem to be getting an error on all files from all of their accounts and no files are being saved. See below for cmd prompt output. Resolving main.g2.bx.psu.edu... 128.118.250.4 Connecting to main.g2.bx.psu.edu|128.118.250.4|:443... connected. WARNING: cannot verify main.g2.bx.psu.edu's certificate, issued by `/C=US/O=Inte rnet2/OU=InCommon/CN=InCommon Server CA': Self-signed certificate encountered. WARNING: certificate common name `bigsky.bx.psu.edu' doesn't match requested hos t name `main.g2.bx.psu.edu'. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `C:/Users/RDoan/Documents/Lab_Files/Analysis_Programs/Galaxy/test.txt Is this the correct method to download histories or is there a better solution? Thank you for your help! Ryan

2 1

Tool menu customization :: Card #727
by Björn Grüning 13 Jun '13

13 Jun '13

Hi, thanks to the awesome work from John Chilton in pull request #160 [1] I hacked up a first version of a tool customization that can be controlled by the user [2]. Requested as trello card #727. The user will have a new preference panel (see the attached screenshot) and can toggle several customization options. The options can be specified by each administrator as filter modules under lib/galaxy/tools/filters/. More details in John's original pull request. For example, one use case would be to offer different module sets for different studies. You can specify system-customizations (John's work) with tool_filters = module:function, module:function2 tool_label_filters = ... tool_section_filters = ... and offer user-customizations with: user_tool_filters = examples:restrict_upload_to_admins user_tool_section_filters = examples:restrict_text user_tool_label_filters = ... at the same time. Only user-customizations will be shown in the preference panel. The description of each filter is parsed from the docstring and shown to the user. The patch requires no modification to the database, all user preferences will be stored in user_preference with three special names. Is there any plan from the core development team how such a feature should be addressed. Is that approach flexible enough? It would be great to get some feedback in which direction such a feature should evolve and if its worth to put more time on it. Thanks John, hope you code will be merged! Thanks for your comments, Bjoern [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/160/implement-dyna… [2] https://bitbucket.org/BjoernGruening/galaxy-central-bgruening/commits/68e7b…

1 1

moving galaxy published pages
by Vipin TS 13 Jun '13

13 Jun '13

Hello dev-team, Usually when I want to move a "Galaxy Published Page" between instances I follows these steps: (a) moving workflows and dataset from instance_1 to instance_2 (b) run the workflow at instance_2 (c) create the page at instance_2 So my question is - Can I move/share published pages between instances other than the above way just curious to know, --/Vipin

2 1

Re: [galaxy-dev] [galaxy-user] How do I add Ray to Galaxy Central in the tool shed ?
by Sébastien Boisvert 13 Jun '13

13 Jun '13

Hi, On 12/06/13 05:50 PM, Jennifer Jackson wrote: > Hi Sébastien, > > This all looks fine, except for the part about including this in the > distribution. As James stated, we are moving to a model where eventually > all tools will be sourced from the Tool Shed. That is where you should > create your repository. So if someone installs galaxy-dist, the person will have to pick up each tool in the tool shed. Is that correct ? Will the schema of XML files remain the same ? > Whether or not to have the repo binaries > self-install & config or not will be up to you. You mean that some apps will self-install and other will require the intervention of a system administrator. > > Going forward, you probably will want to start up a new thread for new > questions and have discussions on the galaxy-dev(a)bx.psu.edu mailing > list. This will give your questions the most exposure to the development > community. > http://wiki.galaxyproject.org/MailingLists#The_lists > You are right, I should have posted to -dev in the first place. Sorry. So my deliverable to add Ray in galaxy is 2 XML files. Thanks. > Take care, > > Jen > Galaxy team > > On 6/12/13 8:05 AM, Sébastien Boisvert wrote: >> On 06/06/13 02:32 PM, James Taylor wrote: >>>> I want to add Ray (a scalable de novo assembler for genomes and >>>> metagenomes) >>>> to Galaxy. >>> >>> And I really want you to do this! >>> >>>> I will also have to write a wrapper for Ray to prepare the command >>>> line from >>>> the options provided by the >>>> Galaxy API. >>>> >>>> >>>> But where is stored the executable (in my case, where is sdtored Ray) ? >>>> >>>> Does Galaxy include the specs to build all the tools available in >>>> Galaxy-Central ? >>> >>> No, we are gradually moving all the tools out of galaxy-central into >>> the Tool Shed. You probably want to look at this page: >>> >>> http://wiki.galaxyproject.org/ToolShedToolFeatures#Automatic_third-party_to… >> >> >> From what I understand, I need to write 2 XML files in order to allow >> people to add Ray to their Galaxy instances: >> >> * ray.xml >> * tool_dependencies.xml >> >> These two files are self-contained -- that is Galaxy will install >> everything using tool_dependencies.xml (in my case, >> Open-MPI and Ray will be installed from source using the GNU compiler). >> >> >> Can I put these files directly in >> >> galaxy-dist/tools/sr_assembly/ray/ >> >> ? >> >> I suppose these 2 XML files will be distributed in galaxy-dist, but >> like Velvet, Ray won't be enabled by default in tool_conf.xml >> as assembly is more costly than mapping. >> >> >> Then, I need to add this to galaxy-dist/tool_conf.xml: >> >> <section name="NGS: Assembly" id="ngs_assembly"> >> <label text="Ray" id="ray"/> >> <tool file="sr_assembly/ray/ray.xml" /> >> </section> >> >> >> And finally, I just need to restart Galaxy. >> >> >> Is that correct ? >> >>> >>> Which describes how you can add a recipe to the toolshed that will >>> install the Ray binaries. >>> >> >> ___________________________________________________________ >> The Galaxy User list should be used for the discussion of >> Galaxy analysis and other features on the public server >> at usegalaxy.org. Please keep all replies on the list by >> using "reply all" in your mail client. For discussion of >> local Galaxy instances and the Galaxy source code, please >> use the Galaxy Development list: >> >> http://lists.bx.psu.edu/listinfo/galaxy-dev >> >> To manage your subscriptions to this and other Galaxy lists, >> please use the interface at: >> >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> >> http://galaxyproject.org/search/mailinglists/ >

2 1

Galaxy wrappers for NCBI BLAST+ BoF
by Peter Cock 13 Jun '13

13 Jun '13

(Apologies is anyone who go this twice, the Galaxy mailing list has been offline.) Hello all, As many of you know I wrote the original BLAST+ wrappers for Galaxy, which are now on the Tool Shed here: * http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus * http://toolshed.g2.bx.psu.edu/view/devteam/blast_datatypes Over the years other people have started to make contributions (e.g. Edward Kirton worked on BLAST databases as a datatype), and this is starting to become a team effort. I'd like that to continue. I've already been in contact with by email with those interested people I knew about to setup a developers meeting during GCC2013, but realise it might be nice to have this formally listed as a BoF - and to make sure we have an allocated time slot as there seems to be a lot going on at the Oslo meeting: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Not everyone will be at the conference, but don't worry - like most open source projects most of this will be done online. People using BLAST+ within Galaxy could come along too and tell us about what they think we should prioritise, but I am mainly hoping to get together with people who have or will be actively involved in the wrapper development and testing. Thanks all, Peter P.S. I will only have intermittent email access from now until the GCC2013 meeting.

1 0

Re: [galaxy-dev] Portable tool identifiers in workflows
by Peter Cock 13 Jun '13

13 Jun '13

Forwarding back to the list which had been offline and seemed to be missing these two emails... ---------- Forwarded message ---------- From: Greg Von Kuster <greg(a)bx.psu.edu> Date: Thu, Jun 13, 2013 at 6:34 PM Subject: Re: Portable tool identifiers in workflows To: Peter Cock <p.j.a.cock(a)googlemail.com> Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu>, Björn Grüning <bjoern.gruening(a)pharmazie.uni-freiburg.de> Hello Peter, Unfortunately, workflows are currently not very compatible with the tool shed - there is a Trello card that details the issues here: https://trello.com/card/workflow-issues/506338ce32ae458f6d15e4b3/692 See my inline comments below. On Jun 13, 2013, at 7:13 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hi Greg, > > I've recent uploaded a couple of workflows to the Tool Shed, > http://toolshed.g2.bx.psu.edu/view/peterjc/venn_list/ > http://toolshed.g2.bx.psu.edu/view/peterjc/rxlr_venn_workflow > > The *.ga files where created by saving the workflow from our > local Galaxy instance. I then wrote a repository_dependencies.xml > file, a README file, made a tar ball, and uploaded this to the > Tool Shed. > > Perhaps unusually the tools were installed manually rather than > via the Tool Shed. This is basically because our Galaxy instance > pre-dates the new hg based Tool Shed, and I was already installing > locally developed tools this way. > > This has a potential implication in the form of the tool identifiers, > drawn to my attention by Björn: > > On Thu, Jun 13, 2013 at 10:54 AM, Björn Grüning wrote: >> >> ... I'm just asking because your >> tool-ids in the ga file are not toolshed-tool-ids. So I'm curious if >> that can work. >> >> I will bug Greg. > > So, does Galaxy handle old fashioned tool identifiers and the > newer toolshed-tool-identifiers gracefully in a situation like this? Unfortunately this does not currently work - this is related to the 3rd item in the above Trello card: Workflows do not consider tool version lineage when inspecting the Galaxy tools_by_id in-memory dictionary, so tools that should be discovered for a workflow are not. > > Do I need to update the *.ga files to use toolshed-tool-ids? If > I did that, would it break the workflow on any Galaxy instance > (like ours) where the tools were installed manually? Yes, you would currently need to update the workflows to use toolshed-tool-ids, and your statement regarding the workflow not working with older tools is correct. This again is related to the 3rd item in the above Trello card: Workflows do not consider tool version lineage when inspecting the Galaxy tools_by_id in-memory dictionary, so tools that should be discovered for a workflow are not. > > Unfortunately from the point of view of testing, these first two > workflows both require a Tool Shed repository dependency > which is non-auto-installable (propriety licensing). > > Thanks, > > Peter >

1 0

Re: [galaxy-dev] blastn: Error: NCBI C++ Exception
by Peter Cock 13 Jun '13

13 Jun '13

Resending as the mailing list was down: On Thu, Jun 13, 2013 at 6:02 PM, Nicola Soranzo <soranzo(a)crs4.it> wrote: >> >> There are two issues here, one that we would like to be able to >> handle multi-threading better via Galaxy configuration on a >> per-tool level, but for now it must often be hard coded or >> done via tool-specific environment variables. > > Regarding the configuration of multi-threading for BLAST+ tools, instead > of hardcoding "-num_threads 8" in the XML files, I'd like to do > implement something like this: > > http://toolshed.g2.bx.psu.edu/repos/jjohnson/cdhit/rev/cca0838c1597 > > What do you think? Should I prepare a pull request to discuss at > GCC2013? That's a sensible idea, something like $BLASTTHREADS as an environment variable? I'd still prefer something built into Galaxy like $THREADS or $GALAXYTHREADS or whatever which can be sets with a default value and adjusted in the per-tool job runner setup (e.g. send BLAST jobs to this cluster queue with 16 threads). This is definitely a good general topic for the tool authors and/or BLAST wrapping BoF sessions at the conference: http://wiki.galaxyproject.org/Events/GCC2013/BoF/ToolDevelopers http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Peter

1 0

Question about the UCSC external viewer
by Malcolm Tobias 13 Jun '13

13 Jun '13

All, Let me try this again. My previous attempts have been bouncing. I've succeeded in getting in getting the "display at UCSC main" link to display, but I'm getting a strange error when I'm redirected to the page. I can't cut-and-paste the error, so hopefully the attached screen shot wont get eaten by the mailing list. From the galaxy log, I'm seeing the UCSC viewer request data: 128.114.119.133 - - [13/Jun/2013:08:47:22 -0500] "GET /display_application/6505e875ddb66fd2/ucsc_bam/main/None/data/galaxy_6505e875ddb66fd2.bam HTTP/1.0" 200 1130428 "-" "genome.ucsc.edu/net.c" but I don't see anything like "6505e875ddb66fd2" under display applications: [galaxy@login002 ~]$ ls -l galaxy-dist/display_applications/ total 48 drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 ensembl drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 gbrowse drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 igb drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 igv drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 rviewer drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 ucsc While the UCSC viewer is requesing data from the "display_application" subdirectory, I only have a "display_applications" subdirectory. I'm not seeing anything like "6505e875ddb66fd2" under galaxy-dist, so maybe I'm not doing some pre-processing needed before being redirected to UCSC? [galaxy@login002 ~]$ find galaxy-dist -name '*6505*' -print [galaxy@login002 ~]$ Any suggestions on what might be going wrong? Cheers, Malcolm -- Malcolm Tobias 314.362.1594

1 0