April 2015 - galaxy-dev - lists.galaxyproject.org

Fwd: Failure to set attributes_type metadata for interval files / error in gff_filter_by_attribute tool
by Marius van den Beek 01 May '15

01 May '15

Hello all, Not sure if the mailing list or trello would be appropriate for this. I had a problem using the gff_filter_by_attribute tool, both locally and on usegalaxy.org. Neither filtering for GTF or GFF were working. The error message was: Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 158, in prepare_job job_wrapper.prepare() File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 832, in prepare self.command_line, self.extra_filenames = tool_evaluator.build() File "/home/galaxy/galaxy-dist/lib/galaxy/tools/evaluation.py", line 408, in build self.__build_command_line( ) File "/home/galaxy/galaxy-dist/lib/galaxy/tools/evaluation.py", line 424, in __build_command_line command_line = fill_template( command, context=param_dict ) File "/home/galaxy/galaxy-dist/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/home/galaxy/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "DynamicallyCompiledCheetahTemplate.py", line 92, in respond NotFound: cannot find 'attribute_types' while searching for 'input.metadata.attribute_types' I could fix it by importing safe_dumps from galaxy.util.json, which apparently was not imported. The next problem occurred because gff_filter_by_attribute expects GTF attribute name-value separators (" ") but allowed GFF input, which uses ("=") as separator. I added a logic to switch between seperators based on the file extension. Everything works smoothly now, the code changes are here: https://github.com/mvdbeek/galaxy Should I create a pull request? -- Marius van den Beek, PhD Drosophila Genetics and Epigenetics Institut de Biologie Paris Seine <http://www.ibps.upmc.fr/fr/Recherche/umr-biologie-developpement/genetique-e…> 9, Quai St Bernard, Boîte courrier 24 75252 Paris Cedex 05 Tel +33 01 44 27 34 01

2 2

Configuring galaxy to run on LSF
by Asma Riyaz 30 Apr '15

30 Apr '15

Hi, Configuration of Galaxy for an LSF server led me to this link: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#DRMAA The issue being the /lsf folder under galaxy does not exist. I am sort of stuck at this point and would highly appreciate anybody's help in this regard. Thank you, Asma

2 1

Migrating user account from galaxy specific to LDAP
by Teshome Dagne Mulugeta 30 Apr '15

30 Apr '15

?Hi, We are currently using galaxy specific login using email address. We changed the authentication to LDAP. Most users email address doesn't match with their registered email address in LDAP. Now, users are not able to see their previous works which is expected of course but I have no clue how to fix it. Please help. Cheers, Teshome

4 5

Galaxy for Natural Language Processing
by Keith Suderman 30 Apr '15

30 Apr '15

Hello, Our group is investigating using Galaxy as a workflow engine for NLP (Natural Language Processing) tasks. I have installed a local Galaxy instance and created wrappers for the services we use and so far everything is working great. I do have a few questions and they all fall under the “Advanced Topics” section as defined at the end of the tutorial for creating a Histogram [1] 1. parameter validation: Many of our tools rely on additions made by previous tools in the workflow; for example, a tool that identifies noun phrases may require that the input has been run through a part of speech (POS) tagger, the POS tagger may require that the input has been run through a tokenizer, etc. Our tools can do this validation, I am just looking for a way to wire this into Galaxy so a user can only connect tools in the workflow editor if this validation passes. I have been looking looking at the code for lib/galaxy/tools/parameters/validation.py and I don’t see anything that I can (easily) bend to our use case. What I was hoping for was something like: <input type=“data” format=“our_custom_format” name=“input”> <validator type=“dataset_custom”> <command interpreter=“bash”>validate.sh $input</command> <!— OR —> <tool file=“custom_validator.xml”/> </validator> </input> I also see the tantalizing sentence, "Custom code execution at various time points of the workflow that allows a fine grained control over the execution process", but I can't find any examples of how this is done. 2. data repositories / data collections I need to be able to process collections of data pulled from remote servers. I have been looking at DataManagers and data collections in Galaxy, but everything seems to assume the data is local to the server, or can be copied/uploaded to the server. For practical and legal reasons beyond my pay grade this is not a solution in our case. For example, an organization may be willing to allow our users to query their service for documents, run the documents through our workflow, and store the intermediate results; but they will not allow us to copy their data to another server verbatim. There are possibilities for me to cache data, but the general use case is that I have to call an external service to fetch documents one at a time and then run the same workflow on each document. Any suggestions on how to accomplish this in Galaxy? I can do single documents, I just need to expand this to include collections of documents. A typical workflow might look something like: a) Query Tool -> Server, find all documents that contain the word “cheese” b) Server -> Here is the list of document IDs [ id1, id2, …, idn ] c) WorkFlow -> for each id in the list do c1) Download document c2 ) Work work work work… c3) Persist output I can do all of the above except the most important bit; iterating… 3. format conversion: Is it possible for Galaxy to automatically convert between formats when designing a workflow? I see the <change_format/> tag, but that seems to change the output format of a tool based on the input (or some other condition) in the same tool; I need to be able to change the format based on the input requirements of the next tool in the workflow. For example, if Tool A produces format X, Tool B requires format Y, and a converter from X to Y has been defined in the datatypes_conf.xml; I would like for Galaxy to implicitly insert the converter from X to Y when I drag the output noodle from Tool A to Tool B in the designer. Is this possible? 4. OAuth 2.0 / OpenID Connect: I need to be able to fetch documents from data providers that require an OAuth 2.0 access token. Currently, I use a separate service to go through the OAuth authentication/authorization process and then have the user copy/paste their access token into a text field in Galaxy. Is there a way to perform the OAuth authentication dance required by the remote service inside Galaxy itself? I’ve looked at the Trello site for Galaxy and see that both OAuth 2.0 and OpenID Connect are on the radar, hopefully this use case is being considered as well. I’m sure to have more questions after working through some visualization examples, but this should keep me busy for now. Sincerely, Keith Suderman REFERENCES 1. https://wiki.galaxyproject.org/Admin/Tools/AddingTools ------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY

4 9

new tool works correctly but galaxy thinks it failed
by Ryan G 30 Apr '15

30 Apr '15

Hi - I implemented a tool I us into Galaxy but when I run the tool view Galaxy, Galaxy thinks the tool failed. I checked the output file the tools create, and it succeeded. How do I tell Galaxy that a tool it ran was successful? Is it something in my XML file? The tool is outputting some info to stderr. Could this be why?

6 10

Re: [galaxy-dev] <code> tagset in tools
by Simon Gladman 29 Apr '15

29 Apr '15

Hi Dan, Thanks for the reply and the suggestions. I think the data libraries would work much better than the get_microbial_data tool as then it would be more easily kept up to date. Cheers, Simon. On 30 April 2015 at 01:01, Daniel Blankenberg <dan(a)bx.psu.edu> wrote: > The <code> tag won’t be going away any time soon, but in many cases there > are other ways to do what this tag does, using the built-in dynamic options > (in particular, currently for tab-delimited files), that are much > preferred. So it is deprecated, and it should be used sparingly, but there > are times when there isn’t a better way for tools to dynamically build > their options without using a code file. > > In the case of the get microbial data tool, if it was working prior to > 15.03, it should still be working, afaik we haven’t changed the handling of > these recently (with the exception of not loading them inside of a ToolShed > application instance). Its been a while since I’ve played with that > specific tool (and we deprecated it ourselves and removed it from main), so > I can’t confirm off-the-cuff whether it should still be working or not. If > something really did break with respect to the code files, we should take a > look though. > > > I think using a Data Library is the perfect replacement for for the > microbial import tool, and would cause less file duplication when used. But > what exactly is no longer working? > > > Thanks for using Galaxy, > > Dan > > On Apr 29, 2015, at 10:38 AM, Peter Cock <p.j.a.cock(a)googlemail.com> > wrote: > > > On Wed, Apr 29, 2015 at 6:33 AM, Simon Gladman > > <simon.gladman(a)unimelb.edu.au> wrote: > >> Hi dev, > >> > >> Now that the <code> tagset is deprecated, how do we replace that > >> functionality. For example the "get microbial data" tool runs the script > >> listed in the <code> tagset to load the list of available microbe data > when > >> the tool is selected. This doesn't seem to work in the latest stable > 15.03.1 > >> or 15.03.2. > >> > >> How do we configure tools like this now? Or is there an updated version > of > >> the microbial_import tool? > >> > >> Cheers, > >> > >> Simon. > > > > Good question Simon. > > > > I am still using <code> for advanced parameter validation, e.g. > > > https://github.com/peterjc/pico_galaxy/blob/64aecfa6041b3b7fe2c70a30236a7aa… > > > > i.e. Validation before the job gets submitted to the cluster or run. > > If the <code> tag suddenly goes away, my tool will still work but > > would be a little less user friendly (some errors would only be > > spotted after the tool starts running). > > > > Peter > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > > > -- Simon Gladman | Bioinformatician Victorian Life Sciences Computation Initiative - LSCC 187 Grattan Street, The University of Melbourne | Victoria 3010 Australia T +61 3 9035 5822 | M 0418 421 077 Mail: simon.gladman(a)unimelb.edu.au Web: http://www.vlsci.org.au

1 0

Data Libraries
by Ryan G 29 Apr '15

29 Apr '15

Hi all - We are trying to use Galaxy as a mechanism for our sequencing lab to create data libraries for data they generate. I noticed in the docs, only Admins are able to create data libraries. Is there a way to change this? I'd like to give specific users in our group this ability without giving them admin rights. Ryan

3 4

Authenticating Against Galaxy from within a tool?
by Ben Bimber 29 Apr '15

29 Apr '15

Hello, I am new to galaxy. I"m trying to write a data input tool wrapper. It will call a script that does a query, produces a list of files, and then the plan is to use the BioBlend API to create datasets/data collections in galaxy. In other words I am making the dataset(s) based on the contents of this file, rather than making a dataset from that file. In my case this file has a pair of columns representing FASTQ files, and I want to create one paired dataset collection in galaxy for each genome. In theory using bioblend to create datasets is easy. However, when I call this as a galaxy tool, I have not found a clean way to pass the credentials to bioblend. BioBlend needs to know the serverURL and either the user's API key or username/password. I think if I poke around $__app__ or $__user__ there's a good change I will find a property with the information I need; however, this has the obvious problem of writing the API key to the log. Is there another way to approach this problem? Thanks in advance for any help, Ben

2 3

optional tool arguments
by Ryan G 29 Apr '15

29 Apr '15

Hi - I'm implementing a new tool in Galaxy. The tool either takes 1 or 2 fastq files as input. If 1 file is specified, then 1 output file is created. If 2 inputs are specified, then 2 output files will be created. How do I specify the optional output parameter? I'm using this as my tool description right now: <tool id="subsample_fastq" name="Random subsample"> <description>a fastq file</description> <command> ### Check for optional input file #if strlen($fq2) > 0: RandomSubFq -w $readsRequested -i "$fq1" -i "$fq2" -o $out1 -o $out2 #else: RandomSubFq -w $readsRequested -i "$fq1" -o $out1 #end if </command> <inputs> <param name="fq1" type="data" format="fastq" label="FASTQ file (1st of Pair)"/> <param name="fq2" type="data" format="fastq" label="FASTQ file (2nd of Pair)" optional="true"/> <param name="readsRequested" type="integer" min="1" value="100000" label="Total number of reads you want from the file"/> </inputs> <outputs> <data format="fastq" name="out1" /> <data format="fastq" name="out2" /> </outputs> <stdio> <exit_code range="-1" level="fatal" description="RandomSubFq failed" /> </stdio>

3 3

Question about automated cleanup scripts
by Van Der Pol William 29 Apr '15

29 Apr '15

Hello, I had a quick question about how one of the cleanup scripts works. I was wondering if admin_cleanup_datasets.py will delete old datasets even if they have been shared or published. Thanks for the help -Liam

1 0