March 2013 - galaxy-dev - lists.galaxyproject.org

Problem running Tophat for Illumina
by greg 06 Mar '13

06 Mar '13

Hi guys, On our local galaxy install I'm getting the error below when I run tophat for illumina like this: http://snag.gy/HsWyS.jpg Any ideas? Thanks, Greg Error: Dataset generation errors Dataset 27: Tophat for Illumina on data 15: deletions Tool execution generated the following error message: Error in tophat: tophat: option --no-closure-search not recognized for detailed help see http://tophat.cbcb.umd.edu/manual.html cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/junctions.bed': No such file or directory cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/insertions.bed': No such file or directory cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/accepted_hits.bam': No such file or directory cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/deletions.bed': No such file or directory The tool produced the following additional output: TopHat v2.0.6 tophat -p 4 -a 8 -m 0 -i 70 -I 500000 -g 20 --library-type fr-unstranded --max-insertion-length 3 --max-deletion-length 3 --coverage-search --min-coverage-intron 50 --max-coverage-intron 20000 --no-closure-search --initial-read-mismatches 2 --segment-mismatches 2 --segment-length 25 --min-segment-intron 50 --max-segment-intron 500000 /usr/local/galaxy/indexes/Mus_musculus/UCSC/mm10/Sequence/Bowtie2Index/genome /home/galaxy/000/dataset_306.dat

3 5

Built-in reference genome
by Charles Murphy 06 Mar '13

06 Mar '13

Hi, I' am trying to add built-in reference genomes for my local galaxy installation. I know this topic has been covered multiple times: I have looked through the discussion threads and the wiki (e.g. NGS Local Setup), but I' am still unsuccessful. Previously I have just loaded a reference genome through the history, but in the interest of saving time and space while I experiment with Galaxy I wish to use the built-in feature. Here is what I have tried: 1. Indexed ecoli.fa using bwa and put all files (fasta and index files) under /Users/charlesmurphy/Desktop/Research/data/genome/ecoli/bwa/ The directory looks like: -rwxrwxrwx@ 1 charlesmurphy staff 4706043 Mar 5 14:54 ecoli.fa -rwxrwxrwx 1 charlesmurphy staff 12 Mar 5 16:38 ecoli.fa.amb -rwxrwxrwx 1 charlesmurphy staff 111 Mar 5 16:38 ecoli.fa.ann -rwxrwxrwx 1 charlesmurphy staff 4639752 Mar 5 16:38 ecoli.fa.bwt -rwxrwxrwx 1 charlesmurphy staff 1159920 Mar 5 16:38 ecoli.fa.pac -rwxrwxrwx 1 charlesmurphy staff 2319888 Mar 5 16:38 ecoli.fa.sa 2. Downloaded and installed the bwa wrapper files from the Galaxy main toolshed, and put bwa in $PATH. 3. Added the following line to the bwa_index.loc (using tabs as the delimiter): ecoli ecoli ecoli /Users/charlesmurphy/Desktop/Research/data/genome/ecoli/bwa/ecoli.fa 4. Added the following line to builds.txt (using tabs as the delimiter): ecoli ecoli 5. Restarted Galaxy. The ecoli genome does not show up as a reference genome for BWA (Map with BWA for Illumina). Any help would be appreciated. Thanks, Charlie

2 4

indexes file of Megablast
by shenwiyn 06 Mar '13

06 Mar '13

Hi everyone, I have some questions about the tool of Megablast in NGS:Mapping when I download the needed indexes . There are four indexes files we need for Megablast:htgs 28-Jan-2013 ,nt 28-Jan-2013,wgs 28-Jan-2013,phiX174;but there are too many types of the indexes file in ftp://ftp.ncbi.nlm.nih.gov/blast/db/ ,for example : nt.00.tar.gz,nt.01.tar.gz,nt.02.tar.gz....I don't know the difference among these files ,and I don't which one I should choose to download.In addition ,I can't find the index file for phiX174. Could anyone help me and give me some advice please? Thank you very much. shenwiyn

1 0

Re: [galaxy-dev] Validating dynamic inputs stream of consciousness
by John Chilton 06 Mar '13

06 Mar '13

Dr. Taylor, Thanks for the e-mail. I think that is a solid instinct, it makes me nervous too, though most things do. I am responding to the list because I think others are going to have the same concern (hopefully!). This whole concept puts a lot of onus on the tool developer. A biologist who has taken a two week course on perl could probably write a Galaxy tool, they probably couldn't write a secure tool for a public LWR. I think some experience in thinking about how to secure web accessible applications and prevent injection style attacks is needed. I will update the documentation urging additional caution with respect to this. That said, there have been in the recent past multiple tools on public Galaxy servers (main included) that were developed by serious programmers that allowed arbitrary code execution. This is something the whole community (or at least that subset hosting public servers) needs to address and take more seriously. I guess the other big question is is there something inherently insecure about the framework - to me the most obvious attack vector there is the regular expression that is generated. I worry if that is that too porous somehow, if there some XML that a developer would write that would reasonably make the tool look secure but in fact there is some malicious command that could get through. I cannot guarantee that it is secure, but I would be eager for counter examples or specific issues I can address. The one thing I will say is over and over, things are whitelisted - not blacklisted - which is generally securer. After that level, I think the LWR does still have issues, but they are all the same issues Galaxy itself has. Can you do a denial of service type attack, yes pretty easily, though I think I could hit any public Galaxy the same way. Also are the tools and wrappers themselves secure, is there some fastq file you could give to bowtie to cause it to run an arbitrary command? My guess is probably, but there is not much the framework could do to prevent that. If we are honest and accept that there are going to security problems with the tools we wrap, one idea that might be worth pursuing for both the LWR and Galaxy itself is running tools in chrooted environments or at least as a different user then the webapp. Thanks again, -John On Wed, Mar 6, 2013 at 10:01 AM, James Taylor <james(a)jamestaylor.org> wrote: > John, something about this makes me really nervous security wise. Need > to think hard on it. > > (Not a complaint, or criticism, or copied to the list, just letting > you know it caught my attention). > > -- > James Taylor, Assistant Professor, Biology/CS, Emory University > > > On Tue, Mar 5, 2013 at 10:19 PM, John Chilton <chilton(a)msi.umn.edu> wrote: >> I have been extending the LWR so that one can now stand up public LWR >> servers (https://lwr.readthedocs.org/en/latest/#setting-up-a-public-lwr-server) >> the idea is that you can publicly share data and computation with any >> Galaxy instances in the world easily and seamlessly via vanilla Galaxy >> tools. This could serve the foundation of a distributed service >> architecture where tools are the contracts, Galaxy instances the >> clients, and LWRs the server containers. >> >> One thing that did need to change with the LWR is that inputs need to >> be validated. It would obviously not be a good idea to allow arbitrary >> command-line or script executions to unauthenticated clients. >> >> This was accomplished in part by allowing you to configure a Galaxy >> toolbox or tool_conf file for the LWR. The LWR reads validation logic >> for command-line and configfiles from the XML and verifies them before >> execution. >> >> The test cases I used to build up this validation extension to the >> tool XML can be found here: >> >> https://bitbucket.org/jmchilton/lwr/src/tip/test/validator_test.py >> >> Question 1: Would Galaxy benefit from implementing this validation >> stuff in the core framework as well? I think of myself as fairly >> security conscious and my sense is no, but it could I see the >> argument. There are potentially complex interactions between user >> inputs and cheetah templates that one might sleep better knowing >> everything is being validated before execution (the user didn't cause >> the cheetah template to somehow render "; rm -rf /"). >> >> Question 2: Looking back on the validation stuff, it seems I am just >> rewriting in XML what cheetah is doing. The duplication makes me think >> the validation stuff might serve the foundation for a better (or at >> least more secure) way to build up commands. >> >> <command interpreter="python"> >> <tool_wrapper>tophat_wrapper.py</tool_wrapper> >> <if> >> <cond> >> <equals left="$refGenomeSource.genomeSource" right="history"> >> <then> >> <parameter name="--own_file">$refGenomeSource.ownFile</parameter> >> .... >> >> The client could easily compile that into a cheetah template to build >> the command, the server could compile it into a regular expression to >> validate the input. >> >> These aren't things that pressingly need to be added to the core >> Galaxy framework, but things to think about longer term. If there is >> any interest from the core Galaxy team I would be happy to implement >> any of this with any desired changes. >> >> Thanks all, >> -John >> >> ------------------------------------------------ >> John Chilton >> Senior Software Developer >> University of Minnesota Supercomputing Institute >> Office: 612-625-0917 >> Cell: 612-226-9223 >> https://twitter.com/jmchilton >> https://bitbucket.org/jmchilton >> https://github.com/jmchilton >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/

2 2

Composite datatype output for Cuffdiff
by Jim Johnson 06 Mar '13

06 Mar '13

Checking to see if there is any interest in including a parameter option to select outputs for cuffdiff, potentially including a composite output and a cummeRbund sqlite database. Issues: cuffdiff produces 21 output files, which is a little unwieldy in a galaxy history. cummeRbund generates its database when given a cuffdiff output directory, but manually hooking up 21 outputs to the cummerbund_wrapper is a pain. I've put demo code in the testtoolshed under the name repository name cummerbund http://jjohnson@testtoolshed.g2.bx.psu.edu/repos/jjohnson/cummerbund This includes new datatypes defined in datatypes_conf.xml and implemented in cuffdata.py:  <datatype extension="cuffdata" type="galaxy.datatypes.cuffdata:CuffDiffData"/>  <datatype extension="cuffdatadb" type="galaxy.datatypes.cuffdata:CuffDataDB"/> The cuffdiff wrapper has a multiple select parameter to choose which output files to put in the history. In addition to the 21 cuffdiff outputs, the wrapper can also generate: cuffdata - which is a composite HTML output with links to the 21 cuffdiff outputs cuffdatadb - which is the cummeRbund SQLite database I also added utility tools: cuffdata_datasets - which will take files from the composite cuffdata and copy them as datasets into the history cuffdata_cummerbund - which generates the cummeRbund cuffdatadb from the composite cuffdata I updated the cummerbund_wrapper: with tryCatch so that a R error on a plot won't exit the Rscript to include a small image of each plot on the html page added plots for : dispersion, scatter matrix, MDS, and PCA Thanks, JJ

7 10

How to count the number of Workflow steps?
by Richard Park 06 Mar '13

06 Mar '13

Hi everyone, Does anybody know how Galaxy currently determines which outputs from a tool are shown when a workflow is run? I'm currently dynamically creating workflows based on the number of given inputs and iterating through all "HideDatasetAction*" for each tools "post_job_actions" to figure out the total number of steps that will be run. "post_job_actions": { "HideDatasetActionoutput": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output" }, "HideDatasetActionoutput_suppressed_reads_l": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_suppressed_reads_l" }, "HideDatasetActionoutput_suppressed_reads_r": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_suppressed_reads_r" }, "HideDatasetActionoutput_unmapped_reads_l": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_unmapped_reads_l" }, "HideDatasetActionoutput_unmapped_reads_r": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_unmapped_reads_r" } }, In the latest galaxy, there no longer seems to be any "HideDatasetAction*" in each tools "post_job_actions" options. How does Galaxy know which output files for a tool it should show or process? Does it have to do something with "RenameDatasetAction*"? "post_job_actions": { "RenameDatasetActionoutput_file": { "action_arguments": { "newname": "19_output_file" }, "action_type": "RenameDatasetAction", "output_name": "output_file" } }, Any guidance would be much appreciated. Thanks,

3 7

Proteomics category on test toolshed
by Ira Cooke 06 Mar '13

06 Mar '13

Hi, I've just uploaded a set of updates to my proteomics tools on the toolshed. These contain my best attempt to implement automatic dependency installation ... and for the most part it seems to work. I have encountered the following issues though (1) I don't see any way to ensure that one repository_dependency is installed before all others. In my case this is important because I have an installer package (galaxy_protk) which needs to be installed before other package installs will work. For example .. try installing http://toolshed.g2.bx.psu.edu/view/iracooke/omssa (2) Running multiple repo installs at once is not reliable (sorry this is so vague). When I have many dependencies being installed at once I get the following error (for example) "Could not cd to ./database/tmp/tmpbIqBW8/galaxy_protk/./database/tmp/tmp3vhKn6/galaxy_protk" It looks like a path is not being constructed correctly. If I wait until the currently installed repo is finished .. then try uninstalling and reinstalling it works. Sorry this is vague but if you try installing any of the following repositories you should see what I mean http://toolshed.g2.bx.psu.edu/view/iracooke/xtandem http://toolshed.g2.bx.psu.edu/view/iracooke/msgfplus http://toolshed.g2.bx.psu.edu/view/iracooke/tpp_prophets Anyway .. and just so it doesn't just sound like I'm complaining ... I think the dependency installation stuff is hugely useful. It took me a while to figure out all the details .. but I think it makes sense now. Cheers Ira

3 6

Redirect http://usegalaxy.org/community broken?
by Peter Cock 06 Mar '13

06 Mar '13

Hello all, What is the recommended URL for the main Tool Shed? Simply http://toolshed.g2.bx.psu.edu/ itself? I believe the URL http://usegalaxy.org/community used to go to the main Tool Shed, but right now I am instead taken to https://main.g2.bx.psu.edu/community/ and told: "This link may not be followed from within Galaxy." Could the redirect be fixed please? And while looking at this, it would perhaps be nice to be have http://usegalaxy.org/community/view/devteam go to http://toolshed.g2.bx.psu.edu/view/devteam/ to see a user's tools, likewise for linking to a specific repo? Thanks, Peter

2 1

Validating dynamic inputs stream of consciousness
by John Chilton 05 Mar '13

05 Mar '13

I have been extending the LWR so that one can now stand up public LWR servers (https://lwr.readthedocs.org/en/latest/#setting-up-a-public-lwr-server) the idea is that you can publicly share data and computation with any Galaxy instances in the world easily and seamlessly via vanilla Galaxy tools. This could serve the foundation of a distributed service architecture where tools are the contracts, Galaxy instances the clients, and LWRs the server containers. One thing that did need to change with the LWR is that inputs need to be validated. It would obviously not be a good idea to allow arbitrary command-line or script executions to unauthenticated clients. This was accomplished in part by allowing you to configure a Galaxy toolbox or tool_conf file for the LWR. The LWR reads validation logic for command-line and configfiles from the XML and verifies them before execution. The test cases I used to build up this validation extension to the tool XML can be found here: https://bitbucket.org/jmchilton/lwr/src/tip/test/validator_test.py Question 1: Would Galaxy benefit from implementing this validation stuff in the core framework as well? I think of myself as fairly security conscious and my sense is no, but it could I see the argument. There are potentially complex interactions between user inputs and cheetah templates that one might sleep better knowing everything is being validated before execution (the user didn't cause the cheetah template to somehow render "; rm -rf /"). Question 2: Looking back on the validation stuff, it seems I am just rewriting in XML what cheetah is doing. The duplication makes me think the validation stuff might serve the foundation for a better (or at least more secure) way to build up commands. <command interpreter="python"> <tool_wrapper>tophat_wrapper.py</tool_wrapper> <if> <cond> <equals left="$refGenomeSource.genomeSource" right="history"> <then> <parameter name="--own_file">$refGenomeSource.ownFile</parameter> .... The client could easily compile that into a cheetah template to build the command, the server could compile it into a regular expression to validate the input. These aren't things that pressingly need to be added to the core Galaxy framework, but things to think about longer term. If there is any interest from the core Galaxy team I would be happy to implement any of this with any desired changes. Thanks all, -John ------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 https://twitter.com/jmchilton https://bitbucket.org/jmchilton https://github.com/jmchilton

1 0

Workflow Actions
by Richard Park 05 Mar '13

05 Mar '13

Hi everyone, Does anybody know how Galaxy currently determines which outputs from a tool are shown when a workflow is run? I'm currently dynamically creating workflows based on the number of given inputs and iterating through all "HideDatasetAction*" for each tools "post_job_actions" to figure out the total number of steps that will be run. "post_job_actions": { "HideDatasetActionoutput": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output" }, "HideDatasetActionoutput_suppressed_reads_l": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_suppressed_reads_l" }, "HideDatasetActionoutput_suppressed_reads_r": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_suppressed_reads_r" }, "HideDatasetActionoutput_unmapped_reads_l": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_unmapped_reads_l" }, "HideDatasetActionoutput_unmapped_reads_r": { "action_arguments": {}, "action_type": "HideDatasetAction", "output_name": "output_unmapped_reads_r" } }, In the latest galaxy, there no longer seems to be any "HideDatasetAction*" in each tools "post_job_actions" options. How does Galaxy know which output files for a tool it should show or process? Does it have to do something with "RenameDatasetAction*"? "post_job_actions": { "RenameDatasetActionoutput_file": { "action_arguments": { "newname": "19_output_file" }, "action_type": "RenameDatasetAction", "output_name": "output_file" } }, Any guidance would be much appreciated. Thanks, Richard

1 0