- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] 回复：回复：回复：回复：回复： Speed up the galaxy
by Langhorst, Brad 07 Dec '12

07 Dec '12

Are you sure you nee to interlace? If you align using bowtie you specify two files of fastq data (one for read 1 and one for read2) I don't know how the interlacer works, but some tools depend on being able to parse the read names to understand which reads go together - it is possible that the interlacer may not understand this read name format. Brad On Dec 7, 2012, at 8:31 AM, 泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> wrote: Hi Brad, I know and I tried, but the "filter by quality" report an error and the "Fastq interlacer" still treat the pair-end reads as two unrelated reads. I paste one line of my data, the right-hand of pair-end read. @HWI-ST1106:232:D14FHACXX:6:1101:1131:2250 2:N:0:GCCAAT CACCTAAAACTCCTATGCACTCNNCATAAACTTACAATAGATCAACTTCATCCACTAGCCTTGAGCCAAACTAATAAAACAGTGCCACGAGCCCGAAAGT + @@BFFFDFFFHFHIIGIIJJJJ##3<CFCGGGCHEHGICGEGAHICGEIGFC>GHHGIGGJJIG@HGII@GGIGHDEEA@BE@C<CCC?BDD=?B6<@>D Can you have any idea that at least I can use the "Fastq interlacer"? 发件人： "Langhorst, Brad" <Langhorst(a)neb.com<mailto:Langhorst@neb.com>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Dev" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>> 发送日期： 2012年12月7日, 星期五, 8:40 下午主题: Re: 回复： [galaxy-dev] 回复：回复：回复： Speed up the galaxy <PastedGraphic-2.png> see that pencil icon - that lets you change the settings for a given dataset. then you'll see a datatype tab Brad On Dec 7, 2012, at 7:23 AM, 泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> wrote: Hi Brad, I'm sorry, I can not quite follow you. The "pencil" and the "datatype tab" are tools in galaxy? 发件人： "Langhorst, Brad" <Langhorst(a)neb.com<mailto:Langhorst@neb.com>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Dev" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>> 发送日期： 2012年12月7日, 星期五, 7:53 下午主题: Re: [galaxy-dev] 回复：回复：回复： Speed up the galaxy You can just change the format of the data from fastq to fastqsanger if you're sure about the error format (use the pencil, then datatype tab) note: fastqsanger !=fastqcsanger Brad On Dec 7, 2012, at 3:20 AM, 泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> wrote: Hi Alex Now is an another problem. I now deal with two fastq files, there are Illumina enconding 1.8 and pair-end, so I don't need to groom. But the fact is, I need to use the "filter by quality" and "Fastq interlacer" and without groom thses two tools can not regonize the files. Any idea to solve this problem? 发件人： "Bossers, Alex" <Alex.Bossers(a)wur.nl<mailto:Alex.Bossers@wur.nl>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>> 发送日期： 2012年12月5日, 星期三, 4:02 上午主题: RE: 回复： [galaxy-dev] 回复： Speed up the galaxy It seems old Illumina encoding 1.5 So yes some tools requiring the fastqsanger would need grooming. If you are up to programming you can seriously speed this up by using a precalculated transfer or hash table. That way you do not have to do any calculation but just translate each quality line using generic regexp/grep/sed like tools or $seq =~ t/STARTSCORES/SANGERSCORES/g in perl. The wiki table could be an alternative option. You still have to set the upload type to fastsanger likely since it will probably sniff the header and see its fastq but not 1.8+ encoding. Good luck! Alex ________________________________ Van: 泽蔡 [caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>] Verzonden: dinsdag 4 december 2012 16:41 To: Bossers, Alex Cc: mailto:galaxy-dev@lists.bx.psu.edu Onderwerp: 回复： [galaxy-dev] 回复： Speed up the galaxy Hi Alex I look the page of wikipedia, but I have a little confused. We sequenced with solexa. I paste a read of my data, can you tell me whether I need run Groom? The read is like this:@HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 AGAAGTACATCGCGATGCCGTTNCCNNCGAAGGCGATAGNNNACAAGNCCAAATGNTTCTNCATCNNNCNCGAGNNGNCGAGGNCGCCGTGCGACCCTGC +HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 Ya^a`edddeddc\c`a`dc]\Ba^BBZ]ZZ`ZZZ]a]]BBB^[`\UB_V[V\`ZBSZX^BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB 发件人： "Bossers, Alex" <Alex.Bossers(a)wur.nl<mailto:Alex.Bossers@wur.nl>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>>; Nate Coraor <nate(a)bx.psu.edu<mailto:nate@bx.psu.edu>> 发送日期： 2012年12月4日, 星期二, 下午 11:21 主题: RE: [galaxy-dev] 回复： Speed up the galaxy Hi I presume the best way to optimise your current problem is to evaluate whether you really need to groom your data!? If its old data presumably yes, but if it is recent data in Illumina 1.8+ encoding(http://en.wikipedia.org/wiki/FASTQ_format) it is not necessary.... speedup 100% :-) Groom would take long at our servers as well but due to the new Illumina format we didn’t bother to optimise it further by parallelisation for instance... Alex Van: mailto:galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu<mailto:galaxy-dev-bounces@lists.bx.psu.edu>] Namens ? ? Verzonden: dinsdag 4 december 2012 16:08 Aan: Nate Coraor CC: mailto:galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] 回复： Speed up the galaxy Hi， I read the page of the document. But I don't think there is anything I need. Now, I installed a local instance of galaxy, and I just run it on a PC. I need to deal with large data, if I just use the default configure of galaxy, every task would take long time. For example I ran the FASTQ Groomer with a large file, the precedure is so slow, and galaxy would not use the potential of my machine. So I want to know how can I get galaxy ran faster. I already can upload files quickly and I just need to know how to ran tools quickly. 发件人： Nate Coraor <nate(a)bx.psu.edu<mailto:nate@bx.psu.edu><mailto:nate@bx.psu.edu<mailto:nate@bx.psu.edu>>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn><mailto:caizexi123@yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>>> 抄送： "galaxy-dev(a)bx.psu.edu<mailto:galaxy-dev@bx.psu.edu><mailto:galaxy-dev@bx.psu.edu<mailto:galaxy-dev@bx.psu.edu>>" <galaxy-dev(a)bx.psu.edu<mailto:galaxy-dev@bx.psu.edu><mailto:galaxy-dev@bx.psu.edu<mailto:galaxy-dev@bx.psu.edu>>> 发送日期： 2012年12月4日, 星期二, 下午 9:38 主题: Re: [galaxy-dev] Speed up the galaxy On Dec 4, 2012, at 4:36 AM, 泽蔡 wrote: > Hi all, > > How can I speed up the galaxy? Like how to use more cores and memeries. Hi, This is a pretty broad question. However, I would recommend that you start at: http://usegalaxy.org/production --nate > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com> -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com> -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com>

1 0

binary data types get corrupted on upload
by Ido Tamir 07 Dec '12

07 Dec '12

Hi, I am trying to create a sqlite datatype. But whenever I upload the database its corrupted. With the latest galaxy-dist checkout changeset: 8153:5dcbbdfe1087 tag: tip user: Daniel Blankenberg <dan(a)bx.psu.edu> date: Tue Nov 06 12:57:23 2012 -0500 summary: Allow rerun to access hidden datasets. from http://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes I thought that adding subclass="True" in datatypes_conf.xml would be enough to register my new type, but it seems like I stil have to add an entry in binary.py for it to be recognized. But its still corrupted. thank you very much, ido ------ Tool: Upload File Name: hg19.sqlite Created: Dec 07, 2012 Filesize: 11.1 MB Dbkey: hg19 Format: sqlite Tool Version: Tool Standard Output: stdout Tool Standard Error: stderr Tool Exit Code: 0 ------ Input Parameter Value File Format sqlite Genome File Format sqlite Inheritance Chain hg19.sqlite ------ I tried combinations of: datatypes_conf.xml <datatype extension="sqlite" type="galaxy.datatypes.binary:SQLite" mimetype="application/octet-stream" display_in_upload="true" subclass="True"/> with/without: in lib/galaxy/datatypes/binary.py class SQLite(Binary): """SQLite file""" file_ext = "sqlite" def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "sqlite file" dataset.blurb = data.nice_size( dataset.get_size() ) else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "sqlite file (%s)" % ( data.nice_size( dataset.get_size() ) ) Binary.register_unsniffable_binary_ext("sqlite")

2 2

Data Libraries dataset information
by Christos Kannas 07 Dec '12

07 Dec '12

Dear all, I would like to know in which table(s), Galaxy stores metadata information about the datasets which are shared via Data Libraries. With metadata information I mean both the metadata of the dataset but also any information that is added via the use of template forms. Regards, Christos -- Christos Kannas Researcher Ph.D Student e-Health Laboratory <http://www.medinfo.cs.ucy.ac.cy/> kannas.christos(a)ucy.ac.cy kannas.christos(a)cs.ucy.ac.cy chriskannas(a)gmail.com Mob: (+357) 99530608

1 0

Which tools do I need to install?
by greg 06 Dec '12

06 Dec '12

Hi guys, I've finished a local galaxy install on our cluster but I'm confused as to which tools are not included and how I need to install them. I saw this page: http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies But how do I tell what is not installed yet? And where do I install everything I'll need? In galaxy-dist somewhere? Or will that interfere with code updates? Thanks, Greg

1 0

Problem in Manage data Library
by Sachit Adhikari 06 Dec '12

06 Dec '12

We have found that, in Galaxy, when we add dataset to a library under Manage data libraries, Galaxy sometimes detects the type of the file and sometimes does not. What I have noticed is that it detects the type of the file when we select "Copy file into Galaxy" under "Copy data into Galaxy?" option and does not detect it when we select "Link to files ..." Is this a known issue? Regards, Sachit

2 1

Permission denied error for velveth
by Andreas Kuntzagk 06 Dec '12

06 Dec '12

Hi, I enabled velvet on our local install. But when I try to use velveth, I get an "Permission denied: '/data/galaxy/galaxy-dist/database/files/027/dataset_27903_files'" This file does not exist. Only file there is /data/galaxy/galaxy-dist/database/files/027/dataset_27903_dat regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich

4 7

RPy testsuite some tests passing - enough for Galaxy?
by Guest, Simon 06 Dec '12

06 Dec '12

Dear Galaxy devs, We are in the process of deploying Galaxy internally here at AgResearch, and I am currently installing required dependencies. My question relates to RPy. It seems that RPy2 is not yet supported by Galaxy, so we're stuck with RPy 1.0.3a, and this seems to have rusted quite badly. (I note however that the remarks previously made on the Galaxy lists about changes required to RPy appear to have been rolled into 1.0.3a.) After installing RPy 1.0.3a, I ran the bundled test suite. Many tests succeed, but there are several errors, in the following categories: 1. Errors about numpy, e.g. object of type numpy.int64 has no len() (Maybe numpy has changed since RPy 1.0.3a was written. I am using numpy 1.4.1 on CentOS 6.) 2. Errors due to missing files, e.g. logit.r and table.txt test data files do not appear to ship with RPy-1.0.3a (!) 3. An error in testLapackLoad, which may be because of (2), I'm not sure yet. 4. An R object index out of range error in testGetItemSlice, which looks quite serious. So here's my question. Just how much of RPy has to work, for its use in Galaxy to be OK? I attach my RPy test suite output in case anyone is (a) interested, (b) has any useful comments. cheers, Simon Simon Guest Senior UNIX Technical Consultant AgResearch, NZ ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

3 4

Improved Workflows through Multiple File Datasets
by John Chilton 06 Dec '12

06 Dec '12

Here is my video-based sales pitch for multiple file datasets (specifically pull request 86 and 87 and subsequent future pull requests). http://www.youtube.com/watch?v=DxJzEkOasu4 I am still committed to helping develop a comprehensive tag-based approach to enhanced workflows with normal datasets, but that seems months off at best and this current approach has the great feature of tracking "sample" names throughout complex workflows. Thanks for your time, -John ------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 Bitbucket: https://bitbucket.org/jmchilton Github: https://github.com/jmchilton Web: http://jmchilton.net

2 1

local instance upload data failed
by 刘阳 06 Dec '12

06 Dec '12

I just installed postgres, and modified the database_connection option in universe_wsgi.ini to database_connection = postgres://postgres:postgres@localhost:5432/galaxy Then I tried to upload a fasta format file (about 4KB), what confused me is that it is uploading on and on. I think the uploading process is pending. The console gave a lot of message like this: 127.0.0.1 - - [06/Dec/2012:12:10:01 +0800] "GET /api/histories/f2db41e1fa331b3e HTTP/1.1" 200 - " http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/536.11 (KHTML, like Gecko) Ubuntu/12.04 Chromium/20.0.1132.47 Chrome/20.0.1132.47 Safari/536.11" Is it caused by the wrong configuration of postgres? I can't figure it out. Does anyone encountered similar problem? Jacky Lau

1 0

Tool runners don't "listen" to settings in [galaxy:tool_runners]
by Anthonius deBoer 06 Dec '12

06 Dec '12

Hi, In my fresh copy of galaxy-dist I am having a problem with tools being run only with the default tool runner and not with the assigned tool runner I created in the [galaxy:tool_runners] section... This is the setting for bwa_wrapper in my universe_wsgi.ini: bwa_wrapper = drmaa://-V -q long.q -pe smp 4/ and this is what the job admin page shows: It's in a different queue and does not use the smp setting I ask for...(It's the default setting) any ideas? 7 X 26 minutes ago toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3 running python /mnt/ngs/analysis/svcgalaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/ffa8aaa14f7c/bwa_wrappers/bwa_wrapper.py --threads="4" --fileSource=indexed --ref="/mnt/ngs/analysis/svcgalaxy/DATA/galaxy-indexes/bwa-indexes/human_g1k_v37_decoy.fasta" --do_not_build_index --input1=/home/tdeboer/P/MUT190-MAIN/fastq-2_205/2_205_1_AAACAT_R1.fastq --input2=/home/tdeboer/P/MUT190-MAIN/fastq-2_205/2_205_1_AAACAT_R2.fastq --output=/mnt/ngs/analysis/svcgalaxy/DATA/files/000/dataset_776.dat --genAlignType=paired --params=pre_set --suppressHeader=false drmaa://-V -q all.q -pe smp 1/ 4440144

4 6