- galaxy-dev - lists.galaxyproject.org

Update some tools
by 泽蔡 09 Dec '12

09 Dec '12

Hi all, I need to deal with fastq encoding 1.8+ data, so I need the FASTX-Toolkit v0.0.13. So how to update the FASTX-Toolkit in the galaxy? By the way, I run a local instance of galaxy.

1 0

file upload/unzip issues
by Michael Moore 08 Dec '12

08 Dec '12

Hello, I and other members of my lab are encountering issues uploading (some) files to the Main public Galaxy server. We routinely upload from our local server to main.g2.bx.psu.edu via ftp using Cyberduck. In my case, on 12/6/12 I uploaded 12 zipped fastq files. Of these, 9 were completely fine. For the remaining 3 files, the transfer through Cyberduck appeared to work fine, and the files appeared (with the correct file size) on the upload screen under "Get data" as usual. However, once the files were uploaded into a Galaxy history, they were empty with a message saying "Problem decompressing gzipped data." An example is entry #45: '1PositiveRFP92112Pool41_ATCACG_L002_R1_001.fastq.gz' in my history called "Tcell_120812." My account uses this email address as the login ID (mmoore(a)rockefeller.edu) All of these files are pretty large (from 5 to 10 GB once unzipped), but the failure/success did not appear to correlate to the size of the file. I am not exceeding my space quota. In addition, I was able to successfully upload the file listed above (1PositiveRFP92112Pool41_ATCACG_L002_R1_001.fastq.gz) to our local Galaxy installation, so I don't think there is anything wrong with the file itself. However, it has failed to upload to the public galaxy server multiple times. The experience of my lab mates is similar; some of their files are uploading correctly, and others are having the same problem I described above. We can't seem to find anything common among the files that are failing to upload. For instance, they span different sequencing runs and were deposited on our local server at different times Has anyone else been encountering similar issues ? Please let me know if I should direct this question elsewhere or if I can provide any further information. Thanks very much for your time. Michael Moore Rockefeller University

1 0

counting number of dataset files on the file system
by Shantanu Pavgi (Campus) 07 Dec '12

07 Dec '12

I am recovering lost dataset files from the file system and I need to verify number of recovered files with corresponding dataset records count in the database. Should number of dataset files on the file system match count obtained using following query? $ select count(*) from dataset where deleted='f'; Appreciate any help on further improving this check. -- Thanks, Shantanu

1 0

FW: newbie uploader
by Banday, Abdul 07 Dec '12

07 Dec '12

________________________________________ From: Banday, Abdul Sent: Friday, December 07, 2012 2:57 PM To: nate(a)bx.psu.edu Subject: RE: [galaxy-dev] newbie uploader Hi Nate, I appreciate your response. I just loaded the file named s_6_1_sequence.txt.gz under the login name abdul.banday(a)uconn.edu. Could you check the file and reply me back about the concerns if any? Regards Rouf ________________________________________ From: Smith, Therese [tms08012(a)engr.uconn.edu] Sent: Friday, December 07, 2012 11:59 AM To: Nate Coraor Subject: RE: [galaxy-dev] newbie uploader Hi Nate, Will do. Thank you. ________________________________________ From: Nate Coraor [nate(a)bx.psu.edu] Sent: Friday, December 07, 2012 11:39 AM To: Smith, Therese Cc: galaxy-dev(a)bx.psu.edu Subject: Re: [galaxy-dev] newbie uploader On Dec 6, 2012, at 7:16 PM, Smith, Therese wrote: > Dear Sir or Madam, > We are using DOS prompt ftp, we connected, we sent (8Gbytes), and we received a succcessful completion message from ftp, we saw the file appearing on your page as having been ftp'd to you, however, we received an upload error message, namely: > > 34: s_8_1_sequence.txt.gz > error > An error occurred running this job: Traceback (most recent call last): File "/galaxy/home/g2main/galaxy_main/tools/data_source/upload.py", line 384, in__main__() File "/galaxy/home/g2main/galaxy_main/tools/data_source/upload.py", line 373, in __main__ add_file( dataset > > We are trying again, with binary channel in ftp,and also would be glad of your advice. > > Our Galaxy login is: abdul.banday(a)uconn.edu Hi Therese, If you could send back an email after your file has finished uploading, but not run the upload tool on it, we can check to make sure it's okay on our filesystem before you attempt to import, and also test importing it ourselves. Thanks, --nate > > Sincerely, > Rouf and Therese > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/

1 0

Cannot see data when selecting "Display Data in Browser"
by Dave Walton 07 Dec '12

07 Dec '12

I'm seeing some weird behavior in our local galaxy instance and am wondering if anyone has seen anything similar. I have a custom tool that I've written that generates a tab-delimted file. When I click on the item name in the history, I can see the contents in the "preview" window. However, when I click on the eye (Display Data in Browser), it comes up empty. Same thing if I download the file. However, if I go to the database directory to examine the file on disk, the content is exactly what I would expect. The permissions on the file are galaxy user/group and the file has permissions of 644. I don't see any reason why I shouldn't be able to view or download this file through galaxy. Any ideas? Thanks, Dave --- Dave Walton Manager Scientific Computing The Jackson Laboratory Bar Harbor, Maine Dave.walton(a)jax.org The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

2 1

Defunct munge processes using Torque PBS
by Matthew Shirley 07 Dec '12

07 Dec '12

I am fairly new to PBS management, so I can't rule out some misconfiguration, but I have a strange issue when running Galaxy with the PBS job runner. It seems that munge spawns a bunch of defunct processes after running Galaxy on my cluster: `ps axjf`: 1 25992 25991 25991 ? -1 Sl 77777 8:48 python ./scripts/paster.py serve universe_wsgi.ini --daemon 25992 26032 25991 25991 ? -1 Z 77777 0:00 \_ [munge] <defunct> 25992 26034 25991 25991 ? -1 Z 77777 0:00 \_ [munge] <defunct> 25992 26036 25991 25991 ? -1 Z 77777 0:00 \_ [munge] <defunct> Now, these processes are being spawned by Galaxy, and I can't figure out why. Can anyone provide some insight or clues about where to start debugging this? Thanks, Matt

2 2

error when looking at tool error
by Andreas Kuntzagk 07 Dec '12

07 Dec '12

If I click on the small "bug" icon for a dataset in error state, I get this message in the log: URL: http://bbc.mdc-berlin.de/galaxy/dataset/errors?id=e96c6ce43a3d239a File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/dataset.py', line 171 in errors hda = trans.sa_session.query( model.HistoryDatasetAssociation ).get( id ) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py', line 507 in get return self._get(key, ident) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py', line 1500 in _get return q.all()[0] File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py', line 1267 in all return list(self) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py', line 1361 in __iter__ return self._execute_and_instances(context) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py', line 1364 in _execute_and_instances result = self.session.execute(querycontext.statement, params=self._params, mapper=self._mapper_zero_or_none()) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py', line 755 in execute clause, params or {}) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py', line 824 in execute return Connection.executors[c](self, object, multiparams, params) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py', line 874 in _execute_clauseelement return self.__execute_context(context) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py', line 896 in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py', line 950 in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py', line 931 in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) DataError: (DataError) invalid input syntax for integer: "e96c6ce43a3d239a" 'SELECT history_dataset_association.id AS history_dataset_association_id, history_dataset_association.history_id AS history_dataset_association_history_id, history_dataset_association.dataset_id AS history_dataset_association_dataset_id, history_dataset_association.create_time AS history_dataset_association_create_time, history_dataset_association.update_time AS history_dataset_association_update_time, history_dataset_association.state AS history_dataset_association_state, history_dataset_association.copied_from_history_dataset_association_id AS history_dataset_association_copied_from_history_dataset_a_1, history_dataset_association.copied_from_library_dataset_dataset_association_id AS history_dataset_association_copied_from_library_dataset_d_2, history_dataset_association.hid AS history_dataset_association_hid, history_dataset_association.name AS history_dataset_association_name, history_dataset_association.info AS history_dataset_association_info, history_dataset_association.blurb AS history_dataset_association_blurb, history_dataset_association.peek AS history_dataset_association_peek, history_dataset_association.tool_version AS history_dataset_association_tool_version, history_dataset_association.extension AS history_dataset_association_extension, history_dataset_association.metadata AS history_dataset_association_metadata, history_dataset_association.parent_id AS history_dataset_association_parent_id, history_dataset_association.designation AS history_dataset_association_designation, history_dataset_association.deleted AS history_dataset_association_deleted, history_dataset_association.purged AS history_dataset_association_purged, history_dataset_association.visible AS history_dataset_association_visible, dataset_1.id AS dataset_1_id, dataset_1.create_time AS dataset_1_create_time, dataset_1.update_time AS dataset_1_update_time, dataset_1.state AS dataset_1_state, dataset_1.deleted AS dataset_1_deleted, dataset_1.purged AS dataset_1_purged, dataset_1.purgable AS dataset_1_purgable, dataset_1.object_store_id AS dataset_1_object_store_id, dataset_1.external_filename AS dataset_1_external_filename, dataset_1._extra_files_path AS dataset_1__extra_files_path, dataset_1.file_size AS dataset_1_file_size, dataset_1.total_size AS dataset_1_total_size \nFROM history_dataset_association LEFT OUTER JOIN dataset AS dataset_1 ON dataset_1.id = history_dataset_association.dataset_id \nWHERE history_dataset_association.id = %(param_1)s' {'param_1': u'e96c6ce43a3d239a'} Looks like the id for the dataset is wrong. How can this happen and how can I fix this? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich

2 1

newbie uploader
by Smith, Therese 07 Dec '12

07 Dec '12

Dear Sir or Madam, We are using DOS prompt ftp, we connected, we sent (8Gbytes), and we received a succcessful completion message from ftp, we saw the file appearing on your page as having been ftp'd to you, however, we received an upload error message, namely: 34: s_8_1_sequence.txt.gz error An error occurred running this job: Traceback (most recent call last): File "/galaxy/home/g2main/galaxy_main/tools/data_source/upload.py", line 384, in __main__() File "/galaxy/home/g2main/galaxy_main/tools/data_source/upload.py", line 373, in __main__ add_file( dataset We are trying again, with binary channel in ftp,and also would be glad of your advice. Our Galaxy login is: abdul.banday(a)uconn.edu<mailto:abdul.banday@uconn.edu> Sincerely, Rouf and Therese

2 1

How to Install FastQC?
by greg 07 Dec '12

07 Dec '12

I'm guessing FastQC isn't installed by default? This page give these instructions: http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies FastQC (0.10.0)NGS: QC and manipulationFastQC from Babraham. The main fastqc perl script must be executable from tool-data/shared/jars/FastQC/fastqc But I don't understand what it means. Do I need to download something, change permissions? I don't have anything inside the jars folder. Thanks, Greg

2 2

Re: [galaxy-dev] 回复：回复：回复：回复：回复： Speed up the galaxy
by Langhorst, Brad 07 Dec '12

07 Dec '12

Are you sure you nee to interlace? If you align using bowtie you specify two files of fastq data (one for read 1 and one for read2) I don't know how the interlacer works, but some tools depend on being able to parse the read names to understand which reads go together - it is possible that the interlacer may not understand this read name format. Brad On Dec 7, 2012, at 8:31 AM, 泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> wrote: Hi Brad, I know and I tried, but the "filter by quality" report an error and the "Fastq interlacer" still treat the pair-end reads as two unrelated reads. I paste one line of my data, the right-hand of pair-end read. @HWI-ST1106:232:D14FHACXX:6:1101:1131:2250 2:N:0:GCCAAT CACCTAAAACTCCTATGCACTCNNCATAAACTTACAATAGATCAACTTCATCCACTAGCCTTGAGCCAAACTAATAAAACAGTGCCACGAGCCCGAAAGT + @@BFFFDFFFHFHIIGIIJJJJ##3<CFCGGGCHEHGICGEGAHICGEIGFC>GHHGIGGJJIG@HGII@GGIGHDEEA@BE@C<CCC?BDD=?B6<@>D Can you have any idea that at least I can use the "Fastq interlacer"? 发件人： "Langhorst, Brad" <Langhorst(a)neb.com<mailto:Langhorst@neb.com>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Dev" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>> 发送日期： 2012年12月7日, 星期五, 8:40 下午主题: Re: 回复： [galaxy-dev] 回复：回复：回复： Speed up the galaxy <PastedGraphic-2.png> see that pencil icon - that lets you change the settings for a given dataset. then you'll see a datatype tab Brad On Dec 7, 2012, at 7:23 AM, 泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> wrote: Hi Brad, I'm sorry, I can not quite follow you. The "pencil" and the "datatype tab" are tools in galaxy? 发件人： "Langhorst, Brad" <Langhorst(a)neb.com<mailto:Langhorst@neb.com>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Dev" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>> 发送日期： 2012年12月7日, 星期五, 7:53 下午主题: Re: [galaxy-dev] 回复：回复：回复： Speed up the galaxy You can just change the format of the data from fastq to fastqsanger if you're sure about the error format (use the pencil, then datatype tab) note: fastqsanger !=fastqcsanger Brad On Dec 7, 2012, at 3:20 AM, 泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> wrote: Hi Alex Now is an another problem. I now deal with two fastq files, there are Illumina enconding 1.8 and pair-end, so I don't need to groom. But the fact is, I need to use the "filter by quality" and "Fastq interlacer" and without groom thses two tools can not regonize the files. Any idea to solve this problem? 发件人： "Bossers, Alex" <Alex.Bossers(a)wur.nl<mailto:Alex.Bossers@wur.nl>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>> 发送日期： 2012年12月5日, 星期三, 4:02 上午主题: RE: 回复： [galaxy-dev] 回复： Speed up the galaxy It seems old Illumina encoding 1.5 So yes some tools requiring the fastqsanger would need grooming. If you are up to programming you can seriously speed this up by using a precalculated transfer or hash table. That way you do not have to do any calculation but just translate each quality line using generic regexp/grep/sed like tools or $seq =~ t/STARTSCORES/SANGERSCORES/g in perl. The wiki table could be an alternative option. You still have to set the upload type to fastsanger likely since it will probably sniff the header and see its fastq but not 1.8+ encoding. Good luck! Alex ________________________________ Van: 泽蔡 [caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>] Verzonden: dinsdag 4 december 2012 16:41 To: Bossers, Alex Cc: mailto:galaxy-dev@lists.bx.psu.edu Onderwerp: 回复： [galaxy-dev] 回复： Speed up the galaxy Hi Alex I look the page of wikipedia, but I have a little confused. We sequenced with solexa. I paste a read of my data, can you tell me whether I need run Groom? The read is like this:@HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 AGAAGTACATCGCGATGCCGTTNCCNNCGAAGGCGATAGNNNACAAGNCCAAATGNTTCTNCATCNNNCNCGAGNNGNCGAGGNCGCCGTGCGACCCTGC +HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 Ya^a`edddeddc\c`a`dc]\Ba^BBZ]ZZ`ZZZ]a]]BBB^[`\UB_V[V\`ZBSZX^BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB 发件人： "Bossers, Alex" <Alex.Bossers(a)wur.nl<mailto:Alex.Bossers@wur.nl>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>> 抄送： "galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>" <galaxy-dev(a)lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu>>; Nate Coraor <nate(a)bx.psu.edu<mailto:nate@bx.psu.edu>> 发送日期： 2012年12月4日, 星期二, 下午 11:21 主题: RE: [galaxy-dev] 回复： Speed up the galaxy Hi I presume the best way to optimise your current problem is to evaluate whether you really need to groom your data!? If its old data presumably yes, but if it is recent data in Illumina 1.8+ encoding(http://en.wikipedia.org/wiki/FASTQ_format) it is not necessary.... speedup 100% :-) Groom would take long at our servers as well but due to the new Illumina format we didn’t bother to optimise it further by parallelisation for instance... Alex Van: mailto:galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu<mailto:galaxy-dev-bounces@lists.bx.psu.edu>] Namens ? ? Verzonden: dinsdag 4 december 2012 16:08 Aan: Nate Coraor CC: mailto:galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] 回复： Speed up the galaxy Hi， I read the page of the document. But I don't think there is anything I need. Now, I installed a local instance of galaxy, and I just run it on a PC. I need to deal with large data, if I just use the default configure of galaxy, every task would take long time. For example I ran the FASTQ Groomer with a large file, the precedure is so slow, and galaxy would not use the potential of my machine. So I want to know how can I get galaxy ran faster. I already can upload files quickly and I just need to know how to ran tools quickly. 发件人： Nate Coraor <nate(a)bx.psu.edu<mailto:nate@bx.psu.edu><mailto:nate@bx.psu.edu<mailto:nate@bx.psu.edu>>> 收件人：泽蔡 <caizexi123(a)yahoo.com.cn<mailto:caizexi123@yahoo.com.cn><mailto:caizexi123@yahoo.com.cn<mailto:caizexi123@yahoo.com.cn>>> 抄送： "galaxy-dev(a)bx.psu.edu<mailto:galaxy-dev@bx.psu.edu><mailto:galaxy-dev@bx.psu.edu<mailto:galaxy-dev@bx.psu.edu>>" <galaxy-dev(a)bx.psu.edu<mailto:galaxy-dev@bx.psu.edu><mailto:galaxy-dev@bx.psu.edu<mailto:galaxy-dev@bx.psu.edu>>> 发送日期： 2012年12月4日, 星期二, 下午 9:38 主题: Re: [galaxy-dev] Speed up the galaxy On Dec 4, 2012, at 4:36 AM, 泽蔡 wrote: > Hi all, > > How can I speed up the galaxy? Like how to use more cores and memeries. Hi, This is a pretty broad question. However, I would recommend that you start at: http://usegalaxy.org/production --nate > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com> -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com> -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com>

1 0