June 2012 - galaxy-user - lists.galaxyproject.org

requirements
by McGowan, Kevin M. 08 Jun '12

08 Jun '12

Galaxy team, Where can I find information about the requirements for installing and running Galaxy on our campus. In particular, I'd like to know what type of computer or server I should purchase. Thanks Kevin McGowan Kevin McGowan Manager Molecular Biology and Cell Culture Shared Resources HHMI-Janelia Farm Research Campus 19700 Helix Drive Ashburn, VA 20147 (571) 209 4681

2 1

Question about Galaxy
by Jennifer Jackson 08 Jun '12

08 Jun '12

Hi Qianli, It looks as if you are using the Extract DNA tool? GTF/GFF files have coordinates with a 1-based start position - and this was your input, but the output from this tool produces coordinates with a BED-style 0-based start position. The genome index files used by the tool are in .2bit format and the source is encoded into the full name. The sequence content of the reference genome is not altered by the tool. Perhaps the interpretation of the coordinates is the source of the difference or you are sourcing a different version? If you have a question about where we sourced a genome, please let us know. To see a quick description of both BED and GFF format together in the UI, click into the tool "Convert Formats -> GFF-to-BED converter". For more in-depth descriptions, please see our wiki: http://wiki.g2.bx.psu.edu/Learn/Datatypes Next time, please send questions directly to our mailing list, with the "to" address as galaxy-user(a)bx.psu.edu. Best, Jen Galaxy team On 6/8/12 7:07 AM, Qianli Shen wrote: > Hello > > I ran a preliminary analysis using test genome and gff file. But the > results is confusing me. > > The fasta file is > > >Gm06 > ATAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTA > AACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAAC > CCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCT > AAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACCCTAAACC > > The gff3 file is > > ##gff-version 3 > Gm06 GDB element 5 15 . + . ID=LEVEL1;Name=Glyma06g05000; > > The output is > > >?_Gm06_4_15_+ > ACCCTAAACCC > > > 1 From the gff3 file we can see the start and end position is 5 and 15, > why the start and end coordinate is 4 and 15 in the output file, which > is really confusing. > 2 How does the galaxy treat the genome file, are all the characters > except for ACGT are removed from the genome file? Because I use substr > function in Perl to test the output of galaxy, somethimes, there is > mismatch. > > I really appreciate all your help. > > qianli > > > -- > Qianli Shen > Graduate Student > Ohio State University > Department of Horticulture and Crop Science > Email: qianlipotential(a)gmail.com <mailto:qianlipotential@gmail.com> > shen.361(a)buckeyemail.osu.edu <http://goog_1606040974> > <http://goog_1606040974> > > > -- Jennifer Jackson http://galaxyproject.org

1 0

Updates to CloudMan
by Enis Afgan 08 Jun '12

08 Jun '12

*We just released an update to CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. *This update brings a large number of updates and new features, the most prominent ones being:* - Support for OpenStack and OpenNebula cloud middleware, allowing easy deployment on private, OpenStack or OpenNebula based clouds (see CloudBioLinux <https://github.com/chapmanb/cloudbiolinux> and mi-deployment<https://bitbucket.org/afgane/mi-deployment/>projects for an easy way to deploy CloudMan on any machine image) - Start your instance via biocloudcentral.org on any supported cloud by simply filling out a 4-field web form. See this screencast<http://www.youtube.com/watch?v=AKu_CbbgEj0> for an example of using it with the Australian National Research Cloud, NeCTAR<http://nectar.org.au/research-cloud> - Support for Amazon Spot instances<http://aws.amazon.com/ec2/spot-instances/>, giving you an opportunity to reduce cost of running your cluster on AWS - Ability to mount any S3 bucket as a local file system via the Admin page, giving you instant and easy file-based access to any of your buckets or public buckets, such as the 1000genomes<http://aws.amazon.com/datasets/4383>one *Other notable enhancements and features:* - Added the ability to disable running of jobs on the master instance (via the Admin page), allowing you to (1) run a smaller instance type longer for the same cost and (2) not see the responsiveness of the master instance degrade with jobs being submitted - Significantly enhanced the details pane for individual worker nodes on the main interface, adding support for terminating and restarting individual nodes - Added MPI and SMP parallel environments to SGE; do qconf -spl to see the list and qsub -pe <pe_name> <slots>* *to use it for your cluster jobs - Removal of data volumes now happens in parallel, shortening the cluster shutdown time - Added worker_post_start_script_url and share_string user data options. See the User Data wiki page <http://wiki.g2.bx.psu.edu/CloudMan/UserData>for the complete list. - Added a messaging framework to allow system information to easily and prominently be shown on the main interface. For example, if an instance was restarted in the wrong zone for its data volume - an explicit message will be shown indicating there was an error and what should be done. - Support for Ubuntu 12.04 - Enhancements to logging by progressively reducing the frequency of log output as no user interaction takes place and also introduced log rotation This update comes as a result of 113 code changesets; for a complete list of changes, see the commit messages<https://bitbucket.org/galaxy/cloudman/changesets/tip/151%3A263> . Any new cluster will automatically start using this version of CloudMan. Existing clusters will be given an option to do an automatic update once the main interface page is refreshed.

1 0

Cufflinks GTF
by Richard Linchangco 07 Jun '12

07 Jun '12

Dear Galaxy, I know this issue has been discussed multiple times but I think what I'm trying to do is a little bit different and I wanted to see if it is viable. Some time ago, I used bowtie and the included index for Mus musculus to do an alignment. Now I'm looking to use Cufflinks/compare/diff for expression. I annotated my alignments(SAM files) with the gff's found here: ftp://biomirror.aarnet.edu.au/biomirror/ncbigenomes/M_musculus/special_requests/gff3/ I've tried running Cufflinks but have gotten 0 FPKM for all genes when using a reference like Ensembl, UCSC, etc. If I were to upload the gff's I posted here, concatenate them, and then try and use that as the reference, would it work? I'm going to try but just thought I'd throw it out there first. The main issue is that in my SAM/BAM files the annotations are GenBank Id's, gi|#|, and all other references I tried were with chr1 and the such. Now that I think about it, I might try just downloading the NCBI gtf from iGenomes. Regardless, let me know what you think! Thanks! -- Richard Linchangco PSM in Bioinformatics College of Computing and Informatics University of North Carolina at Charlotte Tel: (630)440-7010 rlinchan(a)uncc.edu, rlinch2(a)gmail.com

2 1

Re: [galaxy-user] Sequences are not currently available for the specified build.
by Jennifer Jackson 06 Jun '12

06 Jun '12

Hi Rich, You could do this or you could switch to this version of the human genome (is in all of Galaxy's tools, but will not be useable at UCSC): Human (Homo sapiens) (b37): hg_g1k_v37 Thanks, Jen Galaxy team On 6/6/12 7:30 AM, Richard Linchangco wrote: > Dear Jen, > > Thanks again! > > Does this mean I should upload a "Custom Reference Genome"? > I have downloaded the genome specified by GATK for hg19. All I must do > is upload to ftp, map it again, and then it should let me use it later > on for "Counting Covariates" and the such? > > Sincerely, > Rich > > On Wed, Jun 6, 2012 at 9:55 AM, Jennifer Jackson<jen(a)bx.psu.edu> wrote: >> Hi Rich, >> >> Additional genomes will be specially sorted, indexed, and added to the GATK >> tool suite as it moves out of beta status. Hg19 is short-listed for addition >> near term. >> >> We do take requests to have genome added to tools and consider these when >> ranking our prioritization lists. Which genome did you want to use? >> >> One small warning when using a custom reference genome with this particular >> tool set - be sure to visit the GATK web site links directly to understand >> the sorting criteria for genomes. It can be different than how Galaxy, UCSC, >> and many of the existing tools already sort or instruct users to sort >> genomes or data. In short, the genome must be sorted in the exact order that >> it was originally released, but even this can be slightly confusing, >> especially if working with a non-human genome as there are few examples. >> Still, the documentation can help and tools are easily tested (if the >> sorting is wrong, the tool will fail and let you know). >> >> If others have requests for GATK native genomes, they are also welcome to >> reply. In general, key model organisms would be ranked highest in priority. >> We also try to get the largest genomes loaded natively first (for purely >> practical reasons). >> >> Good question, thanks! >> >> Jen >> Galaxy team >> >> >> On 6/5/12 8:01 AM, Richard Linchangco wrote: >>> >>> Dear Galaxy, >>> >>> I've been searching the lists for this type of issue and only found >>> one solution thus far which is the use of a custom reference. It >>> doesn't make sense in my situation because the reference I used was >>> from Galaxy itself when I mapped my data. I'm now trying to use GATK >>> to find SNPs but no matter what I've tried I can't get past this >>> issue. I'm trying to use the Count Covariates and the Unified >>> Genotyper but to no avail. The only issue appears to be that >>> "Sequences are not currently available for the specified build." >>> >>> Any help would be much appreciated. Thanks >>> >>> Sincerely, >>> Rich >>> >> >> -- >> Jennifer Jackson >> http://galaxyproject.org > > > -- Jennifer Jackson http://galaxyproject.org

1 0

problems with the uploading
by Monica BALLARINO 06 Jun '12

06 Jun '12

Dear Galaxy team, I'm using Galaxy Main to run paired-end RNAseq data. I'm a new user and I'm in trouble. I have a total of four set of data to compare: 2 negative controls and two samples. I uploaded already three complete histories (as fastq files) which were around 70 Gb each (MBO11, MBO12 and MBO13). I need to upload the last history (MBO14) but unfortunately I just started and apparently I cannot go further 'cause I've used already 100% of my space. Could you suggest please me a way to go on? My login is edoardo(a)igbmc.fr. Let me know if you would need my password to check my account. Thanks a lot for your support. Monica

3 2

Re: [galaxy-user] Genome Browser Histogram Visualization of Accepted Hits
by Jennifer Jackson 06 Jun '12

06 Jun '12

Hi Trent, I realized that part of your question went unanswered: On 5/31/12 1:58 PM, Fowler, Trent wrote: > Dear Jen, > > Well, it took ~4 hours to load but the visualization track is working > and I am able to show 5 tracks on the screen and scan different genes. > All quite good. However, I cannot find how to copy/paste or save the > graphs for presentation in a figure. Afterwards, such a figure could be > lined up with the gene structure a la UCSC GB. On that note, your note > mentioned something… > > If there is a particular UCSC track (or another source mapped to same > reference genome) that you wish to compare your data against, it could > be imported into Galaxy and added to your Trackster visualization session. Any track loaded into any of your histories can be added to a visualization. Click on the "plus" icon to add tracks. See addtrack1.png. You will be able to navigate by default through your active history, but can also move to any of your histories or the Shared Data Libraries, to select datasets (one or more) and import. See addtrack2.png. The visualization tool recently went through an intensive development cycle. Documentation/tutorials to cover the full functionality is coming. Until then, please let us know if you need pointers when exploring the UI. Best, Jen Galaxy team > > Is there a reference somewhere on how to do this? > > Best, > > Trent > > *From:*Jennifer Jackson [mailto:jen@bx.psu.edu] > *Sent:* Wednesday, May 30, 2012 3:17 PM > *To:* Fowler, Trent > *Cc:* galaxy-user(a)bx.psu.edu > *Subject:* Re: [galaxy-user] Genome Browser Histogram Visualization of > Accepted Hits > > Hi Trent, > > Hiram's advice is very good for UCSC display (thanks Hiram!) > > You most likely heard of converting BAM->SAM->Interval->BED->BigBED. > But, Hiram is exactly correct - there will be a great loss of > information (notably the full individual sequence data itself, as only > the coordinates + variation will remain, plus the data will be in BED6, > which means no splicing, just global coordinates - you won't want this > for RNA-seq). > > Since you are examining RNA-seq data, I also wanted to remind you of the > choice to use Trackster for visualization (top Galaxy menu -> > "Visualization"). A local or custom reference genome can be used, > histogram or full read display is possible, and deeply sequenced areas > can be "unpacked" in stages by using the visualization controls (really > nice). Other features such as drag and drop composite track creation, > new datasets from selects within Trackster (to feed more analysis), > strand coloring, etc. are fairly recent enhancements (see: > http://wiki.g2.bx.psu.edu/DevNewsBriefs/2012_03_12#Galaxy_Track_Browser_.28… > http://wiki.g2.bx.psu.edu/DevNewsBriefs/2012_05_11#Galaxy_Track_Browser_.28…) > More undocumented (so far!) features are added all the time, so good to > experiment. A new paper is in the works that will cover the details. And > there will be tutorials, screencasts, etc. later on, but for now during > the active development cycle, just using it is best and is pretty > intuitive. We're interested in what you like or wish was added (this is > true to everyone reading this!). No promises, but is good to get feedback! > > If there is a particular UCSC track (or another source mapped to same > reference genome) that you wish to compare your data against, it could > be imported into Galaxy and added to your Trackster visualization > session. There are no known limits on data size/depth - only the general > Galaxy dataset file size limit of 50G (applies to all datasets) - > although I suppose an extreme corner case could probably be > devised/uncovered. The point is that a routine RNA-seq experiment > shouldn't present a problem. Special tuning to speed up and handle depth > has been a big area of recent development activity. If you do run into > issues with Trackster, feel free to ask questions (I'll probably ask you > to send a share link to the visualization on the pubic main Galaxy > instance and bring Jeremy into the discussion if the solution is not clear). > > Please let us know if you need more help, > > Jen > > On 5/29/12 11:57 AM, Fowler, Trent wrote: > > Hello, > > I am attempting to run accepted hit data from Tophat output into the > UCSC Genome Web Browser for visualization of sequencing hits in specific > genes. However, the BAM files yield tiles and are too large to present > through the browser. Is there a better file format to convert to that > would allow better visualization such as histograms? > > From word of mouth, I have been told to convert BAMs to BEDs and put > BED files through the browser. However, I notice that Galaxy does not > have an option for this and the oft used BEDtools appears to involve > writing code, which is above my computer abilities. > > Any tips or solutions on how to obtain histograms from sequencing data > would be very welcome. > > Thanks > > Trent Fowler > > > > > ___________________________________________________________ > > The Galaxy User list should be used for the discussion of > > Galaxy analysis and other features on the public server > > at usegalaxy.org. Please keep all replies on the list by > > using"reply all" in your mail client. For discussion of > > local Galaxy instances and the Galaxy source code, please > > use the Galaxy Development list: > > > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > > > To manage your subscriptions to this and other Galaxy lists, > > please use the interface at: > > > > http://lists.bx.psu.edu/ > > > > -- > > Jennifer Jackson > > http://galaxyproject.org > -- Jennifer Jackson http://galaxyproject.org

1 0

Re: [galaxy-user] Sequences are not currently available for the specified build.
by Richard Linchangco 06 Jun '12

06 Jun '12

Dear Galaxy, I've been searching the lists for this type of issue and only found one solution thus far which is the use of a custom reference. It doesn't make sense in my situation because the reference I used was from Galaxy itself when I mapped my data. I'm now trying to use GATK to find SNPs but no matter what I've tried I can't get past this issue. I'm trying to use the Count Covariates and the Unified Genotyper but to no avail. The only issue appears to be that "Sequences are not currently available for the specified build." Any help would be much appreciated. Thanks Sincerely, Rich -- Richard Linchangco PSM in Bioinformatics College of Computing and Informatics University of North Carolina at Charlotte Tel: (630)440-7010 rlinchan(a)uncc.edu, rlinch2(a)gmail.com

2 1

Tophat "Mean Inner Distance between Mate Pairs"
by 杨继文 05 Jun '12

05 Jun '12

Hi all, When mapping pair end RNA-seq reads using tophat, we need to type in "Mean Inner Distance between Mate Pairs". In galaxy, we can read the following information: This is the expected (mean) inner distance between mate pairs. For, example, for paired end runs with fragments selected at 300bp, where each end is 50bp, you should set -r to be 200. There is no default, and this parameter is required for paired end runs. I think the size of fragment (here 300bp) includes not only the length of pair end reads, but also the length of adaptors. so, maybe the Mean Inner Distance between Mate Pairs should be : fragment length - pair end read length - adaptor length. Am I right? or did I miss something? Is it a must to type in the accurate value? Looking forward to your reply JIwen

4 4

Re: [galaxy-user] (no subject)
by Jennifer Jackson 04 Jun '12

04 Jun '12

Hi Megan, If some columns of data were empty, then there should be two adjacent tabs in your data, and Galaxy would leave a blank, empty value in that column. You will need to check how your application actually output the data, possibly pad the empty values, and consider to exporting directly as plain, unformulated text if at all possible. You could also try padding these empty values with a null value. Commonly used values are a dot "." and a zero "0". Different tools will expect different null values, see the tool forms for expected input formats. Completely blank values will cause problems with many tools. Probably also would want to avoid " Convert spaces to tabs:", unless you know this will produce the correct output. Best, Jen Galaxy team On 6/4/12 2:15 PM, Estorninho, Megan wrote: > Thank you Jennifer, this is helpful to know. I am experiencing fewer delays of recent. > > One further question, when uploading a .txt file and converting it to tabular format I sometimes find I my file format changes, I think this is due to blanks in some lines of the tab-delimited file, so that it shifts things out of place and items are no longer in the same columns. Is there a way to avoid this shifting? > > Thanks, > Megan > > ________________________________________ > From: Jennifer Jackson [jen(a)bx.psu.edu] > Sent: 04 June 2012 22:02 > To: Estorninho, Megan > Cc: galaxy-user > Subject: Re: [galaxy-user] (no subject) > > Hi Megan, > > I ran a few tests and found that changing the file suffix to .txt when > using the "autodetect" upload type function speed up the loading process > considerably. As the final result is an identical Galaxy dataset to what > is produced with using the existing suffix, this is something I would > recommend that you try next time. > > For my test, I took one of your files and change the suffix directly, no > other changes were made to the content, as it was already a > tab-delimited text file. I didn't continue with the testing to specify > the datatype at upload (tabular would be the correct choice), but this > is a change that may also speed up import slightly, although the .txt > suffix change was dramatic alone and the upload was quick (I ran a > side-by-side comparison of an original and .txt-suffix modified file). > > The general reason behind this is that Galaxy will interpret data to > detect and confirm datatypes during upload to create associated metadata > needed for tool use. Detection is a convenience option that comes at a > cost (compute resource and time). If you can provide this information > instead, the detection portion of the process can be avoided, > confirmation and metadata creation can be started directly, and the > result is a quicker upload. > > Hopefully this helps for next time, > > Jen > Galaxy team > > On 5/22/12 8:49 AM, Estorninho, Megan wrote: >> Yes I am still experiencing problems. My files are only around 80-120MB and are taking hours to load if at all. >> Thanks for your help, >> Megan >> >> Sent from my iPhone >> >> On 22 May 2012, at 14:38, "Jennifer Jackson"<jen(a)bx.psu.edu> wrote: >> >>> Hello Megan, >>> >>> Are you still experiencing problems now? Galaxy may have been busy >>> immediately following the resolution of the cluster problem, although >>> your problem does appear to be unrelated. >>> >>> It sounds like you are uploading file through a browser. A better choice >>> would be to use FTP. This is required for datasets approaching or >>> exceeding 2G in size. >>> >>> Files that are< 2G, really any file over ~ 500MB, can also benefit from >>> FTP upload. An FTP client tracks the progress of an upload and can >>> resume an interrupted transfer. http://wiki.g2.bx.psu.edu/FTPUpload >>> >>> Hopefully this helps, >>> >>> Jen >>> Galaxy team >>> >>> On 5/21/12 10:21 AM, Estorninho, Megan wrote: >>>> I have been unable to upload data files into Galaxy Main since Friday 18th May 2012. Today is my fourth day of attempting uploads. Refreshing and leaving the files to upload overnight does not work. >>>> Although Jennifer has stated the bug has been fixed at 5.30pm today I am still unable to upload data files. I thought I may be exceeding maximum file capacity but I am well below at only 1.8Gb. >>>> ___________________________________________________________ >>>> The Galaxy User list should be used for the discussion of >>>> Galaxy analysis and other features on the public server >>>> at usegalaxy.org. Please keep all replies on the list by >>>> using "reply all" in your mail client. For discussion of >>>> local Galaxy instances and the Galaxy source code, please >>>> use the Galaxy Development list: >>>> >>>> http://lists.bx.psu.edu/listinfo/galaxy-dev >>>> >>>> To manage your subscriptions to this and other Galaxy lists, >>>> please use the interface at: >>>> >>>> http://lists.bx.psu.edu/ >>> -- >>> Jennifer Jackson >>> http://galaxyproject.org >>> > -- > Jennifer Jackson > http://galaxyproject.org > > -- Jennifer Jackson http://galaxyproject.org

1 0