October 2011 - galaxy-dev - lists.galaxyproject.org

FW: how do I pass the URL of the input and output files to my tool wrapper?
by Chorny, Ilya 24 Oct '11

24 Oct '11

I thought I would add this to the list in case anyone in the galaxy :) knew the answer. I get a mapping error every time I invoke url_for. What variable stores the base url for the galaxy instance? Thanks, Ilya -----Original Message----- From: Chorny, Ilya Sent: Wednesday, October 19, 2011 4:21 PM To: 'James Taylor' Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? Hi James, I got the code working but I can't figure out how to set the URL. I have a hack right now (i.e. url = 'http://' + socket.getfqdn() + config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) + '/display/?preview=True' ). I tried using url_for() but was unable to get that to work. Attached are the diff's for the files I made changes to. If you have any suggestions on how to properly set the url please let me know. Thanks, Ilya -----Original Message----- From: James Taylor [mailto:james@jamestaylor.org] Sent: Tuesday, October 18, 2011 11:38 AM To: Chorny, Ilya Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? Hi Ilya, I think any case where $__app__ is used should raise red flags, it is definitely breaking abstraction. In this case, you're passing a very specific implementation detail of the web layer all the way down to your tool. I think the right way to do what you want is to add a 'web_display_url' attribute to the class DatasetFilenameWrapper in tools/__init__.py. Every dataset gets wrapped by this when put into the namespace for building command lines. If you added an attribute it would them be available as ${input.web_display_url} in the command line template. Historically tool wrappers were designed so that they have no knowledge of the UI, allowing non-web UIs to be added on top. At this point it seems unlikely that this will ever happen, but it is still a good idea to avoid this kind of coupling. Otherwise we can't change how the web interface does security without changing individual tools. Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +0000: > So we came up with a work around and I wouldn't mind getting your opinion on it. Specifically with respect to passing cookie_path. I could not figure out how to get the proxy prefix from the universe_wsgi.ini file via the $__app__ call. Also you mention in the wiki the the $__app__ may get deprecated. > > Thanks, > > Ilya > > > We copied the encryption code (i.e) > > def encode_id(dataset_id,id_secret): > id_cipher = Blowfish.new(id_secret) > s = dataset_id > s = ( "!" * ( 8 - len(s) % 8 ) ) + s > encrypted_id = id_cipher.encrypt( s ).encode( 'hex' ) > return encrypted_id > > and we passed to the too wrapperl: > > -i input -i $input.dataset.id > > as well as > > --id_secret=$__app__.config.id_secret > --cookie_path=$__app__.config.cookie_path > > -----Original Message----- > From: Chorny, Ilya > Sent: Monday, October 17, 2011 11:10 AM > To: Chorny, Ilya; James Taylor > Cc: galaxy-dev > Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? > > Would it be difficult to add the url as an attribute of the input and output parameters ($param.url)? If you point me to where these values are set, I can try to make the changes to the code. > > Thanks, > > Ilya > > > -----Original Message----- > From: galaxy-dev-bounces(a)lists.bx.psu.edu > [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya > Sent: Sunday, October 16, 2011 7:36 AM > To: James Taylor > Cc: galaxy-dev > Subject: Re: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? > > PDF report. > > Sent from my iPhone > > On Oct 15, 2011, at 7:36 PM, "James Taylor" <james(a)jamestaylor.org> wrote: > > > Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +0000: > >> Anyone know if it possible to pass the url (shown in the eye ball) to a tool wrapper? > > > > Ilya, this isn't currently supported because it breaks the > > abstraction between the web interface and tools (tools shouldn't > > know they are being invoked by a web application). What are you trying to achieve with this? > > Is this for a tool generating an HTML report? > > -- > > James Taylor, Assistant Professor, Biology / Computer Science, Emory > > University > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this and other > > Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University

3 9

Few possible galaxy bugs ?
by michael burrell (TOC) 24 Oct '11

24 Oct '11

Good Morning, I am just in the process of updating our development galaxy, the following issues are been reported and I am unable to resolved them, would anyone have any suggestions. Just a little more background info, galaxy@jic55666:~/software/galaxy-central$ hg tip changeset: 6143:6c8695c2a949 tag: tip user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Oct 19 16:47:25 2011 -0400 summary: Enhanced Galaxy instance <=> Galaxy tool shed integration: galaxy@jic55666:~/software/galaxy-central$ python --version Python 2.6.6 the errors reported.... ==== galaxy.tools DEBUG 2011-10-20 13:18:47,691 Loaded tool: hmmscan 1.0.0 docutils WARNING 2011-10-20 13:18:47,697 <string>:2: (WARNING/2) Inline strong start-string without end-string. docutils WARNING 2011-10-20 13:18:47,700 <string>:17: (ERROR/3) Unexpected indentation. ==== galaxy.tools ERROR 2011-10-20 13:18:47,722 error reading tool from path: toolshed/nlstradamus/tools/protein_analysis/nlstradamus.txt Traceback (most recent call last): File "/home/galaxy/software/galaxy-central/lib/galaxy/tools/__init__.py", line 106, in load_tool_tag_set tool = self.load_tool( os.path.join( tool_path, path ), guid=guid ) File "/home/galaxy/software/galaxy-central/lib/galaxy/tools/__init__.py", line 186, in load_tool tree = util.parse_xml( config_file ) File "/home/galaxy/software/galaxy-central/lib/galaxy/util/__init__.py", line 105, in parse_xml tree = ElementTree.parse(fname) File "/home/galaxy/software/galaxy-central/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py", line 859, in parse tree.parse(source, parser) File "/home/galaxy/software/galaxy-central/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py", line 583, in parse parser.feed(data) File "/home/galaxy/software/galaxy-central/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py", line 1242, in feed self._parser.Parse(data, 0) ExpatError: syntax error: line 1, column 0 ==== galaxy.tools DEBUG 2011-10-20 13:18:50,172 Loaded tool: tophat 1.5.0 docutils WARNING 2011-10-20 13:18:50,208 <string>:50: (WARNING/2) Literal block expected; none found. galaxy.tools.test DEBUG 2011-10-20 13:18:50,221 Error in add_param for min_coverage_intron: Unable to determine parameter type of test input 'min_coverage_intron'. Ensure that the parameter exists and that any container groups are defined first. galaxy.tools.test DEBUG 2011-10-20 13:18:50,222 Error in add_param for max_coverage_intron: Unable to determine parameter type of test input 'max_coverage_intron'. Ensure that the parameter exists and that any container groups are defined first. galaxy.tools DEBUG 2011-10-20 13:18:50,222 Loaded tool: tophat_color 1.0.0 galaxy.tools DEBUG 2011-10-20 13:18:50,255 Loaded tool: cufflinks 0.0.5 galaxy.tools DEBUG 2011-10-20 13:18:50,428 Loaded tool: cuffcompare 0.0.5 galaxy.tools DEBUG 2011-10-20 13:18:50,462 Loaded tool: cuffdiff 0.0.5 ==== galaxy.tools DEBUG 2011-10-20 13:18:51,301 Loaded tool: rgManQQ1 1.0.3 galaxy.tools.test DEBUG 2011-10-20 13:18:51,323 Error in add_param for force: Unable to determine parameter type of test input 'force'. Ensure that the parameter exists and that any container groups are defined first. ========= ----------------------------------------------------------------------------------------------------- Please note, that my email address is no longer givenname.surname(a)bbsrc.ac.uk. Please update your records to reflect my new email address of givenname.surname(a)nbi.ac.uk as shown on this email.

2 1

Data Retention Policies
by John Chilton 24 Oct '11

24 Oct '11

Hello All, This is more of a general policy question. The head of our Research Informatics Support Systems group here at the University of Minnesota wants to know how other institutions are handling the large amount of data generated by next generation sequencing facilities and processed by local Galaxy instances. Specifically, I think she is interested in what sort of data retention policies institutions have in place - for instance how long do you store data, what sort of quotas do you put in place, what sort of backups do you have in place, etc.... Any comments would be appreciated. Thanks for your time, -John

3 2

Re: [galaxy-dev] [galaxy-user] tool.xml "&>" console redirection within command-tag
by Peter Cock 24 Oct '11

24 Oct '11

Hi Uwe, On Mon, Oct 24, 2011 at 10:47 AM, Appelt, Uwe <uwe.appelt(a)nct-heidelberg.de> wrote: > Hi again, > > yes, you're right. The problem source is something else. As soon as I specify one of the following > > <command interpreter="bash">$hisapWrapperScript &> $logFile</command> > <command interpreter="sh">$hisapWrapperScript &> $logFile</command> > <command>bash $hisapWrapperScript &> $logFile</command> > <command>sh $hisapWrapperScript &> $logFile</command> > > I get a according debug-line in the galaxy.log, reporting something like: > > galaxy.jobs.runners.local DEBUG 2011-10-24 11:26:55,360 executing: bash /home/extuser/www/galaxy/galaxy_working/database/job_working_directory/2222/tmpZmFMfg &> /home/extuser/www/galaxy/galaxy_working/database/files/005/dataset_5410.dat > > So the "&>" definitely makes it through XML, Cheetah, etc. -parsers and > finally to the cmd-line. However, neither BASH nor SH produce a log-file > as intended. You didn't try the precise combination I meant, <command>$hisapWrapperScript &> $logFile</command> where $hisapWrapperScript is marked as executable and has a Unix hashbang line saying which shell to use, e.g. #!/usr/bin/bash Apologies if I was too vague. > For some strange reason (Peter, you're right again) all log-lines appear in > the green box. To be expected, Galaxy captures any stdout (assuming nothing else captured it first) and puts it in the info field. > And even more, the green box turns of course red as soon as something is > written to stderr. Yes, it is a long standing Galaxy bug that any stderr out is treated as an error condition, rather than looking at the return code: https://bitbucket.org/galaxy/galaxy-central/issue/325/ > Aaaaaah, but the different redirection syntax works. So to conclude: > > <command interpreter="bash">$hisapWrapperScript > $logFile 2>&1</command> > > works like a charme - problem solved! Thanks Peter!!! > > Cheers, > Uwe Glad you got there in the end :) Peter By the way - this whole discussion would have been better suited to the galaxy-dev list (CC'd) since it is about developing tools for Galaxy rather than Galaxy end users.

1 0

Fill field with user email, if logged in
by Steven Platt 23 Oct '11

23 Oct '11

Hi, I a similar vein to http://gmod.827538.n3.nabble.com/Dynamic-default-values-tt3010740.html I am trying to populate a default value based on settings elsewhere, in this case the user email if they are logged in. So far I have: <conditional name="email_on_completion"> <param name="send_email" type="select" label="Send email when job is complete?"> <option value="yes">Yes</option> <option value="no">No</option> </param> <when value="no"/> <when value="yes"> <param name="address" type="text" value="$userEmail" label="Email address"/> </when> </conditional> Which views OK, except that the field contains $userEmail instead of e.g. bob(a)gmail.com Can anyone tell me how to dynamically populate a field with a users email address? (I'd like to allow non-registered users to manually enter their address if necessary, hence the text field rather than capturing the email address directly in the command string.) Thanks Steve ----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************

2 2

update of cufflinks and tophat to use dbkey attribute of input files
by Chorny, Ilya 21 Oct '11

21 Oct '11

Hi Jeremy, I made some more modifications to the wrappers to allow for the tools to pull the genome information from the input files. The only drawback is that there is no way to validate whether the input has the dbkey attribute as a parameter (can you suggest a way to validate?). Attached are the diff's to the current version in galaxy-central. Best, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 2

Re: [galaxy-dev] HOW TO RETRIEVE DATA FROM HISTORY??!!
by colin molter 20 Oct '11

20 Oct '11

> > > Is there a way to directly move/copy data from your galaxy history to a > given location in the filesystem of the same galaxy server? > Said differently, there is a nice way to import data from the server to > galaxy, is it possible to do the reverse? > > So far, I am obliged to download the file from galaxy to my client machine > and then back to the server!!!! with huge bam files of 3Gb it is not so > convenient!! > OK, I found a better way by (a) go to the admin panel, push the 'add a new dataset in the library' button and selecting the one needed from the current history (b) move the selected dataset from the library to a location mounted to galaxy. that is ok for me. However, if someone has a better solution, any advices are fine for me thnks colin

5 5

Multiple formats for tool options
by Alex R Bigelow 20 Oct '11

20 Oct '11

Hi, I've been tweaking the GATK wrappers to try to get my pipeline working in Galaxy; for drop-down menus, I've been able to add additional eligible formats to some parameters: <param name="input_rod" type="data" format="vcf,gatk_dbsnp,bed" label="ROD file" /> This works just fine, but it doesn't work for others: <param name="input_intervals" type="data" format="bed,gatk_interval,picard_interval_list" optional="True" label="A list of genomic intervals over which to operate" /> When I make the change, and reload the configuration (or even restart Galaxy), the menu still doesn't show any options, even though I have a bed file loaded in my history (the same bed file WILL show up in the first menu). Does anyone have an idea what might be wrong? Thanks, Alex Bigelow

2 4

Upload File Button
by Chorny, Ilya 20 Oct '11

20 Oct '11

I create an upload file button using upload.xml in one of my xml scripts. <param name="file_data" type="file" size="30" label="File" ajax-upload="true" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator)."> </param> But when I add it to a work flow and try to have it set the parameters at runtime, the set at runtime options fails. Any ideas why? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 4

Questions about reusing workflows
by Lukasse, Pieter 19 Oct '11

19 Oct '11

Hi, I have some questions about adding new custom tools to my Galaxy instance: 1- How can I reuse workflows? a. Should I wrap a workflow as a tool? b. Or can I just add a custom workflow as a workflow step in another workflow? 2- I noticed that option 1.b above seems to copy the complete workflow into my new workflow, so if I have an existing workflow A with N steps and want to use this in workflow B which already has M steps, I end up with a workflow that has M+N steps while I would expect M+1 steps... This behaviour gives me the impression that if I change workflow A, this will not automatically imply a change in workflow B. Is my interpretation correct? 3- If option 1.a above is the way to go for now, what are the steps to do this? E.g. what is the command to run a workflow from the command line? Thanks and regards, Pieter Lukasse.

3 3