- galaxy-dev - lists.galaxyproject.org

Providing Galaxy tool inputs as an (XML) file?
by Peter Cock 07 Sep '11

07 Sep '11

Hi all, Some of my medium/long term plans for Galaxy involve running one tool which then generates input parameters (and perhaps input files) for another tool. This would be particularly powerful in combination with workflows. One example is phylogenetics where you might have a model selection tool which selects assorted parameters for tree building (e.g. Gamma on or off). General model selection for classifiers might also benefit from such a flow - e.g. Model selection on a subset of the data assessed on a test data set, then running the best model on the full dataset. I can see how I might solve this on a case by case basis, where for each tool which I want to be able to call with a parameter file a wrapper script would be needed, plus the tool would have a top level parameter selection between values from a file, or values entered by the user in the normal way. At least in some niche fields such parameter files already exist - for example the Nexus file format (although many tools which support this as an input format do so in a simplistic way and only look at the alignment or trees, not the commands/settings). However, I'm envisioning a general mechanism for providing (some or all of) the inputs to a Galaxy tool via a file, which for extensibility and consistency with the rest of Galaxy would be in XML. Galaxy would need a nice viewer for this new datatype, which could appear in the history as just another dataset. However, the key new functionality would be on any tool dialogue (if a suitable tool parameter XML file was in the history) would be a new option/button to apply settings from this file to the current form. I'm imagining the new XML would include a list of tool IDs for which it can be used on (e.g. Maximum likelihood model selection might produce settings for both PhyML and RAxML, but these tools might have differently named parameters in their Galaxy tool XML files). Have the Galaxy team been thinking of anything like this already? Thanks, Peter

1 0

Re: [galaxy-dev] [galaxy-bugs] Error message when trying to dowload Manhattan plot using the web version
by Ross 06 Sep '11

06 Sep '11

Hi, Alison I've made some fixes to the Manhattan/QQ tool (galaxy-central revision #dacf97a6c663) so it now handles your data - thanks again for discovering these edge cases - the revised code is now available and working on http://test.g2.bx.psu.edu/ where your data produces plots. In particular X in your data, and for good measure Y and Mt are now translated into 23,24 and 25 respectively so the code can now cope with those inputs and they should appear in linear order in the Manhattan x axis The floppydisk link (download) problem remains to be fixed as it's more complex. We only update the main server every few weeks so the changes I just made will take a while to appear there. Meanwhile, please try the revised tool by making yourself an account and then uploading your data to http://test.g2.bx.psu.edu/ - please let me know how you get on? Of course, the (?FDR adjusted) p values in your data are a challenge - typically, the Manhattan and QQ plots are run with unadjusted p values which are expected to be distributed uniformly 0-1 under the null... FYI, when I use the grouping tool on your data, I see that chromosomes 20,21 and 22 are absent from the file - is this expected? Group with a count gives: 1 22910 10 9990 11 12764 12 13413 13 13068 14 11319 15 11175 16 8483 17 12017 18 10073 19 6927 2 18332 3 16173 4 15401 5 15259 6 15487 7 15217 8 12485 9 14560 X 3951 Thanks again for helping us improve Galaxy... On Mon, Jul 18, 2011 at 10:52 PM, Alison Harrill <AHarrill(a)thehamner.org> wrote: > The first time I uploaded the file, I loaded in the -logP values > instead. After that, I changed the file to raw p values and reuploaded. > Do you think it would help to upload the file again, but change the > filename? I just wonder if that original file didn't get cleared out. > > Alison > > -----Original Message----- > From: Ross [mailto:ross.lazarus@gmail.com] > Sent: Monday, July 18, 2011 8:51 AM > To: Alison Harrill > Subject: Re: [galaxy-bugs] Error message when trying to dowload > Manhattan plot using the web version > > ok - thanks. These are not raw pvalues and your data is definitely > causing the Manhattan plot to fail. > > This will take me a little time to figure out - I'll get back to you > in a day or two - thanks again for reporting this problem > > On Mon, Jul 18, 2011 at 10:16 PM, Alison Harrill > <AHarrill(a)thehamner.org> wrote: >> Good morning, >> >> Sorry for taking a while to get back to you (I was out of town for the >> weekend). I set permissions to share the dataset with you. Please let > me >> know if you see an obvious workaround. >> >> Thank you, >> Alison >> >> >> Alison Harrill, Ph.D. >> Research Investigator >> Institute for Drug Safety Sciences >> Hamner Institutes for Health Sciences >> phone: (919) 226-3142 >> fax: (919) 226-3150 >> >> >> >> -----Original Message----- >> From: Ross [mailto:ross.lazarus@gmail.com] >> Sent: Sunday, July 17, 2011 7:25 AM >> To: Alison Harrill >> Subject: Re: [galaxy-bugs] Error message when trying to dowload >> Manhattan plot using the web version >> >> Can you please share a history with me (use this email address) so I >> can take a look please? >> >> I'm talking about links on the results page where you (should!) see >> some graphs - when you click the history item display ('eye') icon - >> not the history item save ('floppy disk') icon >> >> On Sun, Jul 17, 2011 at 9:15 PM, Alison Harrill > <AHarrill(a)thehamner.org> >> wrote: >>> Hi, Ross, >>> >>> Thanks so much for kindly following up with me. I tried the > workaround >>> you suggested, but it appears to not be working for me. > Right-clicking >>> prompts me to save a file called display.htm and when I open the > file, >>> it just reads "Unable to remove temporary library download archive > and >>> directory". >>> >>> Best wishes, >>> Alison >>> >>> -----Original Message----- >>> From: Ross [mailto:ross.lazarus@gmail.com] >>> Sent: Sunday, July 17, 2011 4:09 AM >>> To: Alison Harrill >>> Subject: Re: [galaxy-bugs] Error message when trying to dowload >>> Manhattan plot using the web version >>> >>> Hi, Alison - I wanted to be sure you are able to get your images from >>> the tool - the proper fix is going to take a while so I hope those >>> links are working for you? >>> >>> cheers... >>> >>> >>> On Sat, Jul 16, 2011 at 9:39 AM, Ross <ross.lazarus(a)gmail.com> wrote: >>>> Hi, Alison, >>>> >>>> Thanks for reporting this error - I just confirmed that you have >> found >>>> a bug - the download function is currently not working correctly for >>>> composite datatypes in the history. >>>> >>>> Fortunately, the images themselves have download links under each > one >>>> and can be saved - eg on a PC or linux workstation, while hovering >>>> over the '(Click to download image Armitagep_manhattan.png)' press >> the >>>> right side mouse button and choose "save link as.." - you will be >> able >>>> to choose a location to save the image file on your local >> workstation. >>>> I hope this "work around" allows you to get on with your work - > sorry >>>> for the inconvenience and thanks again for finding and reporting > this >>>> error - we will fix it as soon as we can. >>>> >>>> >>>> >>>> On Sat, Jul 16, 2011 at 6:46 AM, Alison Harrill >>> <AHarrill(a)thehamner.org> wrote: >>>>> Greetings, >>>>> >>>>> I have a question about generating a Manhattan/QQ plot using the > web >>> tool. I >>>>> was able to successfully run the plot (the result box is green), > but >>> when I >>>>> try to download the results, I get a message that says "Unable to >>> remove >>>>> temporary library download archive and directory." Is there another >>> way to >>>>> retrieve the data? >>>>> >>>>> Thank you, >>>>> >

2 1

get data interface for public Galaxy instance
by Scott Emrich 06 Sep '11

06 Sep '11

Dear team, There are multiple 3rd party sources to import arthropod vector data (e.g. BioMart) from our NIH Bioinformatics Resource Center (BRC) VectorBase into Galaxy, but some are outdated and some of our users would like a direct source. Given this increase of Galaxy in our vector biology community, what is the policy for adding a new server to "Get Data" and is there information on formatting our results appropriately (i.e., using a REST interface)? Our NIH BRC server is www.vectorbase.org. I noticed our sister BRC EuPathDB has made their data available via http://main.g2.bx.psu.edu, and we have similar developer resources to make our server compatible; however, I couldn't find this information online besides addressing this list. Thanks in advance, and best regards, - S --- Scott Emrich, PhD Scientific Manager, VectorBase Assistant Professor, Computer Science and Engineering University of Notre Dame

3 4

Galaxy megablastdb indexer script proposal
by Jacob Israel Cervantes Luevano (LANGEBIO) 06 Sep '11

06 Sep '11

Hi Galaxy Dev, Nice to meet you. My name is Jacob and I am from México , Actually I'm working in LANGEBIO (Laboratorio Nacional de Genomica para la Biodiversidad) at Bioinformatics. I would like to contribute with galaxy dev team, I developed shell script to set up reference genomes by megablast. the linux shell script that I prepared uses your current prepdb.py script. the mini pipeline flow-work: 1. read input values: * fasta filename (nucleotide) * blast format: blast format: F->nucleotide, T->protein * database output name * fasta sequence IDs format (ncbi-gbk , other) where: ncbi-gbk means something like ">gi|gi-number|gb|accesión|locus" other means something like ">my own IDs" 2. if fasta sequence IDs are like ncbi format(ncbi-gk) then execute prepdb.py python script and generate new fasta file then go step 4 example: >gi|gi-number|gb|accesión|locus 3. if fasta sequence IDs has diferent (other) identifiers set to galaxy format first then execute prepdb.py python script then go step 4 >my own identifier 4. to format database using formatdb blast utility 5. setting up Megablast Galaxy Loc File 5. delete temp files 6. done Usage example: query help: ./set_galaxy_blastdb.sh -help format blast nucleotide dabatabase yeast, set galaxy loc file ./set_galaxy_blastdb.sh -fasta /media/xlinux/genomes/galaxy/palomero/megablast/yeast_nt.fasta -p F -n yeastdb -fastadesc ncbi-gk when script finished we have following results: $ls /media/xlinux/genomes/galaxy/palomero/megablast/ blastdb.loc new.yeast_nt.fasta yeastdb.nhr yeastdb.nin yeastdb.nsq yeast_nt.fasta How can I contribute with this shell script ? Thanks in advance -- Jacob http://www.langebio.cinvestav.mx/bioinformatica/jacob -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

2 1

TEMP variable in cluster install
by Shantanu Pavgi 06 Sep '11

06 Sep '11

We have configured galaxy to work with our SGE cluster using drmaa job runner interface. We are using 'unified method' for this install and both TEMP environment variable and new_file_path in universe_wsgi.ini file have been configured correctly. However, we are seeing some errors where local /tmp space on compute nodes is being referenced by the galaxy tools. Specifically we saw it mentioned in error messages from following tools: * bwa_wrapper and upload tools: 'No space left on device: /tmp...' * sam_to_bam tool: 'Error sorting alignments from /tmp/filename..' Shouldn't it be referencing TEMP environment variable or new_file_path configuration value? Is it getting overridden by TMP or TMPDIR variables in python wrapper code? Has anyone else experienced similar issue? -- Thanks, Shantanu.

5 6

Galaxy API question: workflow_execute
by Roxanne Tapia 06 Sep '11

06 Sep '11

I was trying to use the workflow_execute.py And I can't understand the last argument: usage: workflow_execute.py key url workflow_id history step=src=dataset_id So in step=src=dataset_id step: I think equals the next step #in the history? src: ??? dataset_id: This has to be the dataset id from a data library which I can get by using display.py to query libraries & contents of libraries. Right? Can anyone explain this to me? Roxanne Tapia Software Developer 3 B-6, Genome Science Bioscience Division rox(a)lanl.gov (505)665-0206

3 2

mycoplasma genomes - suggestion
by Lukasz Kielpinski 06 Sep '11

06 Sep '11

Hello everybody, I would like to suggest including mycoplasma genomes (preferably all genomes in one file) as reference genomes. It would enable fast checking if cells used for deep-seq experiments were not contaminated. Best, Lukasz

2 1

Galaxy API question - histories
by Roxanne Tapia 06 Sep '11

06 Sep '11

Is there a way to create/view histories via the api, and to associate data in libraries to histories? Then to share the histories or make them public, and build a URL to view the history? Roxanne Tapia Software Developer 3 B-6, Genome Science Bioscience Division rox(a)lanl.gov (505)665-0206

2 1

Galaxy API question:
by Roxanne Tapia 06 Sep '11

06 Sep '11

Hi, I was going through the examples in the README, and everything went along fine until I tried to upload files from the import directory: usage: library_upload_from_import_dir.py key url folder_id file_type server_dir dbkey I don't understand what the last two arguments 'server_dir' and 'dbkey' represent, or how to determine the correct values to pass along. I can, however, successfully upload via the import dir from the webapp. Any help would be appreciated, Roxanne Tapia Software Developer 3 B-6, Genome Science Bioscience Division LANL

2 1

Quotas
by SHAUN WEBB 06 Sep '11

06 Sep '11

Hi. I am interested in putting quotas in to effect on our local Galaxy server. From the admin page I can create a quota but this only seems to apply to disk space. I noticed on your test site you have quotas for concurrent jobs also. Have I missed how to set this or is this functionality not available in galaxy-dist yet? 2 other issues: 1) My account says I am using "0 bytes" in the top right corner although my history displays 456mb of data? 2) Are quotas set by default? I was able to create quotas before I had updated universe_wsgi.ini to include enable_quotas = True Shaun -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1