November 2011 - galaxy-dev - lists.galaxyproject.org

Bug merging sam files
by Chorny, Ilya 17 Nov '11

17 Nov '11

There seems to be a bug merging sam files when using splitting and BWA. The merge cats the files together and thus multiple headers are distributed through the file. Does anyone have plans to add a merge method to the SAM datatype class? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

1 0

Broken published history in a local installation
by Nicolas Delhomme 17 Nov '11

17 Nov '11

Hi all, Since the last pull from galaxy-central (6056), whenever I try to access published histories, I get a file not found error; see the attached screenshot. I have not changed any configuration. Thanks for any pointers, Nico --------------------------------------------------------------- Nicolas Delhomme Genome Biology Computational Support European Molecular Biology Laboratory Tel: +49 6221 387 8310 Email: nicolas.delhomme(a)embl.de Meyerhofstrasse 1 - Postfach 10.2209 69102 Heidelberg, Germany ---------------------------------------------------------------

5 5

Dataset Cleanup Question
by Lance Parsons 17 Nov '11

17 Nov '11

I am running a local instance of Galaxy and I've been trying to sort out some issue with dataset cleanup. For the most part, things are working OK running the shell scripts in the recommended order: delete_userless_histories.sh purge_histories.sh purge_libraries.sh purge_folders.sh delete_datasets.sh purge_datasets.sh I have the number of days set to 10. When I look at the reports webapp however, it reports that there are "62 datasets were deleted more than 15 days ago, but have not yet been purged, disk space: 12975717335." These have stuck around now for 45 days (and counting). I have even tried running the scripts with the -f option to force galaxy to re-evaluate the datasets to no avail. Any suggestions? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 2

Re: [galaxy-dev] Inquiring
by Yan Luo 17 Nov '11

17 Nov '11

Could anybody answer my previous following questions one more week ago? Thanks, Yan On Mon, Nov 7, 2011 at 8:52 PM, Yan Luo <luomails(a)gmail.com> wrote: > Dear Dannon, > > Thanks for your response. I will discuss with our manager to see if we can > do sth. We want to find some solutions for our PIs. AS you mentioned. > > Option 1: main.g2.bx.psu.edu, it is more stable, and recommended by you. > Option 2: test.bx.psu.edu, it provides more tools and data, is very much > for testing and breaking and shouldn't be used without acknowledging that > > Option 1 and 2 are free for us (NIH), are they? > > Option3: From other vendors, such as "amazon" : > http://aws.amazon.com/ec2/pricing/ , it is not free. But it provides more > tools. > > Option 4: local installation, where is the official download source(link > and installation)? > > Option5: Cloud installation, do you have any more detailed information > regarding supporting hardware and software? > > > > Best Wishes, > > Yan > > > On Mon, Nov 7, 2011 at 8:26 PM, Dannon Baker <dannonbaker(a)me.com> wrote: > >> Yan, >> >> 1. Right now, we host and you can use usegalaxy.org ( main.g2.bx.psu.edu) and >> test.bx.psu.edu. Main will be the most reliable server and what we >> recommend you use, but test has a few beta tools that aren't available on >> main yet. That said, test.g2.bx.psu.edu is very much for testing and >> breaking and shouldn't be used without acknowledging that. >> >> 2. A local galaxy instance is hosted locally, on your server(s). If you >> have a local compute cluster or even a single beefy server that you can >> access and use, this is free, but comes with the caveat that you actually >> run and administer your own servers, install required tools, etc. A cloud >> galaxy instance is hosted on Amazon EC2, and comes with most tools and many >> common indices installed and preconfigured. The downside to this is that >> it does cost money ( http://aws.amazon.com/ec2/pricing/ ) to run EC2 >> instances based on your needs. There's a lot more I can point out if >> you're interested in running your own galaxy in the cloud, but I would >> tentatively suggest that you experiment with our main galaxy server at >> usegalaxy.org, and potentially migrate to a local and/or cloud solution >> as your analysis needs demand. >> >> Thanks! >> >> Dannon >> >> On Nov 7, 2011, at 6:33 PM, Yan Luo wrote: >> >> > Hi, Nate, >> > >> > How are you? Last week our director attended a meeting at New York and >> got some information from you. I just visited your homepage " >> http://galaxy.psu.edu/" he mentioned. We have some question to "Use >> Galaxy" and "Get Galaxy". We want to figure out how our PIs at NIH can use >> them. Do you have any information regarding the 2 above issues? for >> example, slides, introduction, and links, etc. >> > >> > 1. Which servers are available for users to use right now? Are they >> free? Is there any restriction? for example, file size, speed, etc. >> > >> > 2. Is there any difference between "local version" and "cloud version"? >> What are they different and benefits? >> > >> > >> > Please let me know if you have any suggestion. >> > >> > >> > Best Wishes, >> > >> > Yan >> >> > <http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=I%20>

3 4

Missing requirements in xml wrappers in galaxy-dist?
by Clare Sloggett 16 Nov '11

16 Nov '11

Hi James & all, I have been getting some errors to do with the path environment variable. For instance, when uploading a sam file to our local galaxy instance, we got: "Traceback (most recent call last): File "/data/ugalaxy/galaxy-dist/tools/data_source/upload.py", line 394, in __main__() ..... line 63, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/local/lib/python2.7/subprocess.py", line 1228, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory" I can post the full error if you'd like, but basically the problem was that samtools wasn't in the PATH. This was because we have our tools installed in a non-standard place, so we are depending on the requirements being specified as James described below, and samtools isn't specified as a requirement in upload.xml, so when upload.py calls datatypes.py and tries to use samtools, it gives an error. I've found a couple of other examples like this - for instance samtools is also used by some picard scripts so should be specified as a requirement in the picard wrappers. These problems probably don't show up in most cases when people just have the tools installed as root and on their PATH by default? I'm going to be fixing these where I find them. Would it be helpful for me to contribute these tweaks back or would it be better just to raise an issue? Thanks, Clare On Wed, Nov 16, 2011 at 4:33 PM, Clare Sloggett <sloc(a)unimelb.edu.au> wrote: > Looks like it's working! The problem I had run into, in hindsight, was > a) I hadn't set tool_dependency_dir as I didn't know about it > b) if I had set it, I was installing tools to > $SW/tool-name/version-number/ but the default tool wrappers in > galaxy-dist don't have version numbers set, so they will just look in > $SW/tool-name/ . > > Can I suggest this be added to the wiki somewhere under the Admin > pages? Apologies if it's there, I couldn't find it except under News > Briefs at http://wiki.g2.bx.psu.edu/News%20Briefs/2010_11_24?highlight=%28tool_depend… > . > > As well as being on the wiki it would be useful to have it (commented > out by default) in universe_wsgi.ini. I think the tool_dependency_dir > variable isn't in there at all at the moment, at least in the > galaxy-dist I have. It would also be useful to have a brief mention or > link to it on http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup to > save time for people like me who had tools installed in a non-standard > place. > > Thanks again! > Clare > > On Wed, Nov 16, 2011 at 3:01 PM, Clare Sloggett <sloc(a)unimelb.edu.au> wrote: >> Great! Thanks James, this is exactly what I need. >> >> On Wed, Nov 16, 2011 at 2:20 PM, James Taylor <james(a)jamestaylor.org> wrote: >>> On Nov 15, 2011, at 9:59 PM, Clare Sloggett wrote: >>> >>>> If this is the case, what is the best way to install and maintain two >>>> versions of the same tool? I can write code into the wrapper to find >>>> the correct version of the tool in a given case, but I was wondering >>>> if there is a more standard 'galaxy' way to configure this. >>> >>> You should provide "tool_dependency_dir" in the config file and point it at some directory $SW where you will install tools under. >>> >>> With this enabled, when a tool has a dependency, Galaxy will look for it under that directory and attempt to run a script to setup the environment. For example if you have tool with a dependency on foo version 1.3, Galaxy will look for: >>> >>> $SW/foo/1.3/env.sh >>> >>> and if found will source it as part of the job submission script. This usually contains something simple like >>> >>> PATH=$PACKAGE_BASE/bin:$PATH >>> >>> to add the binaries installed with the dependency to the path. >>> >>> Ideally all dependencies used by Galaxy tools would be installed in this way. >>> >>> >> >> >> >> -- >> E: sloc(a)unimelb.edu.au >> P: 03 903 53357 >> M: 0414 854 759 >> > > > > -- > E: sloc(a)unimelb.edu.au > P: 03 903 53357 > M: 0414 854 759 > -- E: sloc(a)unimelb.edu.au P: 03 903 53357 M: 0414 854 759

2 1

Local galaxy Cufflinks problem
by Bidwell, Christopher A. 16 Nov '11

16 Nov '11

Colleagues, We have set up a local Galaxy but I am having trouble running Cufflinks with the annotation gtf file. We have the iGenomes btau4.2 genome in the database and use the iGenomes btau4.2 gtf file in the history. However the –G isn’t showing up in the command line (below) and the output is the same as when I run cufflinks without annotation. Is something not set up correctly? Info: cufflinks v1.0.3 cufflinks -q --no-update-check -I 50000 -F 0.050000 -j 0.050000 -p 4 -N -b /phillip/hiscansq/run00085-09-16-11_D0D1KACXX/110916_H179_0062_AD0D1KACXX/hr00206_OAR13-24/References/Bos_taurus/genome.fa The details page shows the gtf file being supplied from the history Input Parameter Value SAM or BAM file of aligned RNA-Seq reads 26: Tophat for Illumina on data 2 and data 1: accepted_hits Max Intron Length 50000 Min Isoform Fraction 0.05 Pre MRNA Fraction 0.05 Perform quartile normalization Yes Conditional (reference_annotation) 1 Reference Aonnotation 24: Btau42_iGenomes_annot.gtf Conditional (bias_correction) 0 Conditional (seq_source) 0 Conditional (singlePaired) 0 Thanks for your help. Chris

2 1

Re: [galaxy-dev] Problem with the Galaxy tool shed
by Greg Von Kuster 16 Nov '11

16 Nov '11

Hi Marcel, On Nov 15, 2011, at 5:06 PM, Marcel Schumann wrote: > Hi Greg, > > wonderful, thanks a lot! > I actually had a look at the Galaxy tool shed a few hours ago, saw the few http-upload option and used it. It worked well, no problems there. > > But there are indeed two other small issues that I encountered after this: > -- Changing the tool-suite synopsis does not work anymore. I tried to change it in order to reflect the new version number, but after saving the changes, the page still shows the old version-number (i.e. the old synopsis). I am not able to reproduce this problem, assuming you changed the values in either the name, synopsis or detailed description fields on the "Manage repository" page in the tool shed like the screen capture here. Can you clarify that this is what you are trying to do? > -- Selecting an old repository revision and then using "Browse or delete files" will result in the list of files that are in the current revision of the repository. However, downloading works correctly; that is, if you select an old revision, you will really download the old version ... This is by design. Although you can preview tools, inspect tool metadata, data types, etc for older repository revisions, you can browse files contents and delete selected files only within the repository tip. I've made this more clear by changing the "Browse or delete repository files" to now be "Browse or delete repository tip files". The intent here is to make it easy to maintain the latest revision of the repository within the tool shed UI by enabling you to upload new files to the repository tip or delete selected files from the repository tip. This assumes you are not making changes to old versions of your tools and wanting to make them available in the tool shed. In order to push updates of old versions of your tools to the tool shed, you'll have to create a different tool shed repository for each version of your tools. > > > Thanks & best regards, > Marcel > Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

FastQC 0.10.0
by Ross 16 Nov '11

16 Nov '11

Please ignore this message unless you maintain a local Galaxy instance and use FastQC. The most recent FastQC release 0.10.0 fixes some bugs but also completely changes the HTML layout of the report in ways that break the old wrapper. I've updated the wrapper script and tests to cope with this, so after pulling the latest changes from galaxy-central, please also ensure that the current executable release of FastQC from http://www.bioinformatics.bbsrc.ac.uk/projects/download.html#fastqc is unpacked to tool-data/shared/jars to take advantage of the fixes. (You'll also probably need to make tool-data/shared/jars/FastQC/fastqc executable after unpacking - I've notified the author...)

1 0

installing multiple versions of a tool / path configuration
by Clare Sloggett 15 Nov '11

15 Nov '11

Hi all, I am a little confused as to the right way to configure my installed NGS tools. >From the documentation I've found and from looking at the xml/python wrappers, it looks to me like the NGS tool wrappers simply call the tools and assume they will be in the galaxy account's PATH. For instance, bwa_wrapper.py simply calls "bwa" in the shell. So it looks to me like the default assumption is that bwa was installed as root and is available to all users, and will always be on your PATH. Is this right, or am I missing some configuration options that should tell galaxy where to find the bwa binary? If this is the case, what is the best way to install and maintain two versions of the same tool? I can write code into the wrapper to find the correct version of the tool in a given case, but I was wondering if there is a more standard 'galaxy' way to configure this. Sorry for the newbie questions again - I have been looking at the NGS setup and Tools documentation, but I don't think I've found anything on multiple installed versions of a tool. Also if there are any docs I have missed on configuring path environment variables, please let me know! Thanks, Clare -- E: sloc(a)unimelb.edu.au P: 03 903 53357 M: 0414 854 759

2 3

Bootstrapping the Galaxy installation process: populating the database
by Duddy, John 15 Nov '11

15 Nov '11

We want to automate certain aspects of setup. We already create the target database and put it in universe_wsgi.ini, but we want to plug some values into the tables during installation. Currently, the schema is created the first time Galaxy is run. I did not see a way to cause that to happen via the manage_db.sh script. Have I missed something? If not, any pointers on how I might go about adding that (i.e. what stuff to call to just initialize the schema)? John Duddy Sr. Staff Software Engineer Illumina, Inc. E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

2 1