- galaxy-dev - lists.galaxyproject.org

January 2014 Galaxy Update Newsletter
by Dave Clements 31 Dec '13

31 Dec '13

1 0

ODoSE - error when using exclusively 'external' genomes
by Jen 31 Dec '13

31 Dec '13

hello, I've run across a possible bug in Odose that is preventing me from analyzing my data. I've scoured the various help documentation without luck, so I'm writing you. I'm using the program via this site: http://www.odose.nl/ The history showing the steps below is here: http://www.odose.nl/u/publ/h/unnamed-history The problem is this: I'm unable to work with a set of genomes that is exclusively "user-generated" (i.e., external, and uploaded from my local drive). To illustrate this problem with an example, I downloaded the .ffn files for four published genomes NCBI's ftp site. (This way, I'm sure my 'external' files are formatted correctly.) I've tried to follow these instructions from http://www.odose.nl/ (Analyze Data tab) - >"User-generated whole-genome data not yet archived in GenBank can be uploaded to be analysed separately or in conjunction with published genomes. To do this, first create a new history: in the right ‘History’ pane go to Options > Create New. In the left ‘Tools’ pane go to Get Data > Upload File and select one or multiple .ffn files from your computer and click ‘Execute’..." This worked fine. >"Go to Divergence > Upload genomes in the ‘Tools’ pane, press the ‘Add new genome’ button, specify a genome label and press ‘Execute’ for each genome to be included…" I was unable to find a "Divergence" option in the Tools pane. So, I assumed this should be O'DoSE > Upload Genomes, in the Tools Pane. Using this function, I uploaded each genome and selected "Execute." >"Proceed to the Workflow menu to add listed genomes and/or to specify input settings…" Perhaps this is my downfall, but I was unable to see how to add genomes in the Workflow menu. At this point, I had uploaded four genomes, and attempted to select the first option in the analysis: "Extract & Translate Genes." I was presented with the option of selecting my external genomes, but the analysis failed, giving the error "History does not include a dataset…" error shown in the attached (1_Extract+TranslateFailed.tiff) Interestingly: The problem is solved by selecting at least one genome from the Complete Microbial Genomes table. Selecting one genome generates a Project ID list (shown in the attached, 2_SelectedGenome.tiff). I am then able to perform "Extract & Translate Genes" and all further functions, acting on the set of external genomes and the one selected (internal) genome. I've tried this several times, with different external genome files. I can successfully analyze dozens of external genomes, but only if at least one 'internal' genome is selected as well. Unfortunately, I need to run an analysis on a set of genomes that is exclusively unpublished, so the data will be entirely "external" genomes. Including a published genome will throw off the comparisons we need to make. Based on the ODoSE instructions above, I sense I should be able to use exclusively 'external' genomes in an analysis. Do you have any suggestions on how I can do this? Any advice would be much appreciated. thank you, Jen

1 0

galaxy prod installation with apache proxy
by Fernandez Edgar 30 Dec '13

30 Dec '13

Hello everyone, So I've successfully installed on rhel6 a production environment of galaxy with an apache proxy. Unavoidably, I have some question when it comes to the permissions of the galaxy directory : Currently, all the galaxy directory is a. chown -R galaxy: users /home/galaxy/galaxy-dist b. chmod -R 700 /home/galaxy/galaxy-dist Will this cause me some issues with the apache proxy? Furthermore, I've try to run run_functional_tests.sh and I get a log file that you can find in attachment. Can you please explain the last error and why don't I get any successful tests? Lastly, I was wondering if it's possible to move the directory /home/galaxy/galaxy-dist/database somewhere else, and create a logical link instead (/home/galaxy/galaxy-dist/database --> /another/path/database)? Cordialement / Regards, Edgar Fernandez

2 1

how to create users through Galaxy API?
by xlwang 29 Dec '13

29 Dec '13

hello, I want to create users through Galaxy API.For example, I can get users information through url like this(use postman): http://localhost:8080/api/users?key=my_key But how to create a user?

2 1

Installing Repositories from Galaxy Tool Shed To Galaxy
by Crystal Goh Wei Pin 28 Dec '13

28 Dec '13

Hi Galaxy, The Cufflinks and Cuffdiff available on Galaxy are version 0.0.6. I would like to use Cufflinks and Cuffdiff version 0.0.5. Cufflinks and Cuffdif version 0.0.5 are available on Galaxy Tool Shed. I read through the ways to install Repositories from Galaxy Tool Shed To Galaxy, an admin user is required. Could Galaxy help me with the installation? My account on Galaxy is crystal86(a)live.com.my Thank you so much. Best regards, Crystal

1 0

Is there a way yet to more easily determine name of inputs for renaming in workflows
by Thon deBoer 27 Dec '13

27 Dec '13

Hi, In order to make Galaxy workflows even more useful, it is going to be important for data files to be renamed in a workflow. I know we can use the "#{input1 | base name}-BWA.sam" trick which is useful IF you know what the actual parameter name is in the XML file. There was some discussions some years ago (http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-September/011288.html) but it is is now very time consuming to figure out from the XML file what the name of the input parameter is you would like to use (and it is often very tricky, due to the fact that the name of the actual input parameter may actually change based on some selection in the UI…. I often had to resort to actually chaining the XML file to make the output something like: <data format="sam" name="output" label="#echo os.path.splitext ( str ( $paired.input1.name ).replace ( '_R1' , '' )) [ 0 ]#-bwa_#echo str ( $paired.sPaired ) #.sam”> But that is time consuming and does no longer work with the toolshed since an update of a tool is merely a brute force replacement of the previous version, no longer allowing me to have to change it once and let the updates work with on the changed version as was working brilliantly before all tools were moved out of the main branch and into the toolshed)… So, are there any plans to at least expose the Inputs (or the XML) during the workflow editing or something similar? Now I have to dig up the XML file every time I use a new tool, making the creation of workflow a tedious task... Regards, Thon [cid:enz_logo_2_new0600d5] Thon deBoer Director of Software 3415 Colorado Ave. Room A255 Boulder, CO 80303 Cell: (650) 799-6839 tdeboer(a)enzymatics.com www.enzymatics.com Enzymatics Inc. Confidentiality Notice: This e-mail and any attachments are confidential, proprietary and or privileged information. It is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone other than the intended recipient(s). If you are not an intended recipient, please delete this e-mail immediately and notify the sender by e-mail.

1 0

Some plugin for digital normalization
by Cristian Alejandro Rojas 26 Dec '13

26 Dec '13

Hello all, I've been searching in tool sheds(main and test) for a plugin to do digital normalization of reads, for example Trinity has one script to do this (./util/normalize_by_kmer_coverage.pl), diginorm is another alternative ( http://ged.msu.edu/papers/2012-diginorm/) Do you know if there is a galaxy tool ready for this work or should I develop my own? Happy Holidays! -- *Cristian Alejandro Rojas Quintero* *Estudiante Ingeniería de Sistemas * *Universidad Distrital Francisco José de Caldas* Bogotá - Colombia

1 0

Including descriptions etc in extended BLAST+ tabular output
by Peter Cock 25 Dec '13

25 Dec '13

Hello all, FAO: Administrators of local Galaxy instances using the NCBI BLAST+ wrappers. Over on the galaxy_blast repository I have been updating the NCBI BLAST+ wrappers (including unit tests) to work with the current release, NCBI BLAST+ 2.2.28 (aka BLAST 2.2.28+): https://github.com/peterjc/galaxy_blast The initial set of changes is now on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus This includes a workaround for a known regression in the makeblastdb tool dealing with duplicated identifiers: https://github.com/peterjc/galaxy_blast/commit/349e31c6cec4429c5523fde5975e… In terms of end-user features, the big improvement in the BLAST+ 2.2.28 release was the ability to get the BLAST match descriptions in the tabular output, and other fields: http://blastedbio.blogspot.co.uk/2012/05/blast-tabular-missing-descriptions… staxids means Subject Taxonomy ID(s), separated by a ';' sscinames means Subject Scientific Name(s), separated by a ';' scomnames means Subject Common Name(s), separated by a ';' sblastnames means Subject Blast Name(s), separated by a ';' (in alphabetical order) sskingdoms means Subject Super Kingdom(s), separated by a ';' (in alphabetical order) stitle means Subject Title salltitles means All Subject Title(s), separated by a '<>' sstrand means Subject Strand qcovs means Query Coverage Per Subject qcovhsp means Query Coverage Per HSP On this branch I am including the new salltitles field as the 25th column in the extended BLAST tabular output offered within the Galaxy interface: https://github.com/peterjc/galaxy_blast/tree/c25 However, I'm not so sure about the taxonomy fields. Since (thus far) they are not available via the XML, I am leaning to introducing a third tabular mode, e.g. * Standard 12 columns (can convert from XML) * Extended 25 columns (can convert from XML) * Extended also with taxonomy (cannot currently convert from XML) Instead, we could offer a pick-you-own columns route (in all the primary BLAST tools, handled via macros)?: * Standard 12 columns (can convert from XML) * Extended 25 columns (can convert from XML) * Pick your own columns from the full NCBI list (depending on columns, can convert from XML) This is inspired by JJ's changes to the BLAST XML to tabular conversion tool for Galaxy-P, https://github.com/jmchilton/galaxy_blast/commit/d79afc03522768323494818a40… I would be much keener on the pick-you-own columns option if it was possible for the tool to record arbitrary column names for a tabular file in Galaxy's metadata (I can't find a trello card, but I'm sure I've asked about this before). Any thoughts or comments? eg Hurry up and just release this branch adding the hit descriptions as column 25 - we want that now ;) [*] Regards, Peter [*] For our local instance, the taxonomy stuff will be useful, but right now I would prioritise the description, which we currently get via the BLAST XML using this tool: https://github.com/peterjc/galaxy_blast/tree/master/tools/blastxml_to_top_d…

4 13

Request: Option to reduce server data transfer for big workflow in cluster
by Ben Gift 20 Dec '13

20 Dec '13

We've run into a scenario lately where we need to run a very large workflow (huge data in intermediate steps) many times. We can't do this because Galaxy copies all intermediate steps to all notes, which would bog down the servers too much. I asked about something similar before and John mentioned the feature to automatically delete intermediate step data in a workflow once it completed, was coming soon. Is that a feature now? That would help. Ultimately though we can't be copying all this data around to all nodes. The network just isn't good enough, so I have an idea. What if we have an option on the 'run workflow' screen to only run on one node (eliminating the neat Galaxy concurrency ability for that workflow unfortunately)? Then it just propagates the final step data. Or maybe only copy to a couple other nodes, to keep concurrency. If the job errored then in this case I think it should just throw out all the data, or propagate where it stopped. I've been trying to work on implementing this myself but it's taking me a long time. I only just started understanding the pyramid stack, and am putting in the checkbox in the run.mako template. I still need to learn the database schema, message passing, and how jobs are stored, and how to tell condor to only use 1 node, (and more I'm sure) in Galaxy. (I'm drowning) This seems like a really important feature though as Galaxy gains more traction as a research tool for bigger projects that demand working with huge data, and running huge workflows many many times.

3 8

Tool Access Control
by Eric Rasche 20 Dec '13

20 Dec '13

Howdy devs, I've implemented some rather basic tool access control and am looking for feedback on my implementation. # Why Our organisation wanted the ability to restrict tools to different users/roles. As such I've implemented as an "execute" tag which can be applied to either <section> or <tools> in the tool configuration file. # Example galaxy-admin changes For example: <section execute="a@b.co,b@b.co" id="EncodeTools" name="ENCODE Tools"> <tool file="encode/gencode_partition.xml" /> <tool execute="b(a)b.co" file="encode/random_intervals.xml" /> </section> which would allow A and B to access gencode_parition, but only B would be able to access random_intervals. To put it explicity - by default, everyone can access all tools - if section level permissions are set, then those are set as defaults for all tools in that section - if tool permissions are set, they will override the defaults. # Pros and Cons There are some good features - non-accessible tools won't show up in the left hand panel, based on user - non-accessible tools cannot be run or accessed. There are some caveats however. - existence of tools is not completely hidden. - Labels are not hidden at all. - workflows break completely if a tool is unavailable to a shared user and the user copies+edits. They can be copied, and viewed (says tool not found), but cannot be edited. Tool names/id/version info can be found in the javascript object due to the call to app.toolbox.tool_panel.items() in templates/webapps/galaxy/workflow/editor.mako, as that returns the raw tool list, rather than one that's filtered on whether or not the user has access. I'm yet to figure out a clean fix for this. Additionally, empty sections are still shown even if there aren't tools listed in them. For a brief overview of my changes, please see the attached diff. (It's missing one change because I wasn't being careful and started work on multiple different features) # Changeset overview In brief, most of the changes consist of - new method in model.User to check if an array of roles overlaps at all with a user's roles - modifications to appropriate files for reading in the new tool_config.xml's options - modification to get_tool to pass user information, as whether or not a tool exists is now dependent on who is asking. Please let me know if you have input on this before I create a pull request on this feature. # Fixes I believe this will fix a number of previously brought up issues (at least to my understanding of the issues listed) + https://trello.com/c/Zo7FAXlM/286-24-add-ability-to-password-secure-tools + (I saw some solution where they were adding "_beta" to tool names which gave permissions to developers somewhere, but cannot find that now) Cheers, Eric Rasche -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru

3 6