- galaxy-dev - lists.galaxyproject.org

Additional eggs
by Chorny, Ilya 30 Sep '11

30 Sep '11

Could you add PyX and Matplotlib(http://sourceforge.net/projects/matplotlib/files/matplotlib/matp… to the eggs? They would be very useful for creating reports. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

1 0

2 questions about Galaxy's functional testing support
by Paniagua, Eric 30 Sep '11

30 Sep '11

Hi all, I've read the Wiki page on Writing Functional Tests (http://wiki.g2.bx.psu.edu/Admin/Tools/Writing%20Tests) and I've been looking through test/base and test/functional and I am left with two questions: * Is it possible to write a test to validate metadata directly on an (optionally composite) output dataset? Everything described on the above page is file oriented. I see that there is TwillTestCase.check_metadata_for_string, but as far as I can tell this is a bit nonspecific since it appears to just do a text search on the Edit page. I don't yet fully understand the context in which tests run, but is there some way to access a "live" dataset's metadata directly, either as a dictionary or just as attributes? Or even to get the actual dataset object? * Does the test harness support retaining output files only for failed tests? Ideally with a cap on how much output data to save. If not, would this be difficult to configure? Thanks, Eric

1 0

Re: [galaxy-dev] GATK unified genotyper
by Chorny, Ilya 29 Sep '11

29 Sep '11

To prevent user error in mixing genomes, would you be interested in a feature in which the unified genotyper choses the genome in a fashion similar to how samtools know which which genome to use? <when value="cached"> <param name="input1" type="data" format="sam" metadata_name="dbkey" label="SAM File to Convert"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_file" filename="sam_fa_indices.loc" metadata_name="dbkey" metadata_column="1" message="Sequences are not currently available for the specified build." line_startswith="index" /> </param> </when> Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 1

Re: [galaxy-dev] GTF files in Cufflinks and Tophat
by Chorny, Ilya 29 Sep '11

29 Sep '11

Jeremy, 1.) Would you be interested in a feature in which the gtf files could be "locally cached" instead of only "from history"? 2.) Would you be interested in a feature in which cufflinks choses the genome in a fashion similar to how samtools know which which genome to use? i.e.: <when value="cached"> <param name="input1" type="data" format="sam" metadata_name="dbkey" label="SAM File to Convert"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_file" filename="sam_fa_indices.loc" metadata_name="dbkey" metadata_column="1" message="Sequences are not currently available for the specified build." line_startswith="index" /> </param> </when> I am happy on working on these changes if they will be put into galaxy-central. Best, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

1 1

Attribute Error when uploading data on local
by Colleen Doherty 29 Sep '11

29 Sep '11

Hi, I get the following error when trying to upload data to Galaxy AttributeError: 'NoneType' object has no attribute 'lower' Same error for many different types of files (fastq, sam, bam) some of which uploaded last week. What I've tried: Deleting browser cache Deleting cookie Deleting database/compiled_templates Deleting eggs/ Please let me know what else I should try / what I am doing wrong. More info below. Thank you! More information: TRACEBACK: URL: http://127.0.0.1:8080/datasets/59c76a119581e190/display/?preview=True File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/lib/galaxy/web/framework/base.py', line 145 in __call__ body = method( trans, **kwargs ) File '/home/colleen/Work/workspace/ChIPSeq/tools/galaxy-dist/lib/galaxy/web/controllers/dataset.py', line 361 in display mime = trans.app.datatypes_registry.get_mimetype_by_extension( data.extension.lower() ) AttributeError: 'NoneType' object has no attribute 'lower' Extra Data CGI Variables CONTENT_LENGTH '0' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'en-us,en;q=0.5' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'galaxysession=c6ca0ddb55be603a2c2796e41769bc2412b1c87332f54f840b1876eab065f3113e1ca20ff6a4ab5c' HTTP_HOST '127.0.0.1:8080' HTTP_REFERER 'http://127.0.0.1:8080/history' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:6.0.2) Gecko/20100101 Firefox/6.0.2' PATH_INFO '/datasets/59c76a119581e190/display/' QUERY_STRING 'preview=True' REMOTE_ADDR '127.0.0.1' REQUEST_METHOD 'GET' SERVER_NAME '127.0.0.1' SERVER_PORT '8080' SERVER_PROTOCOL 'HTTP/1.1' Extra Data WSGI Variables application <paste.debug.prints.PrintDebugMiddleware object at 0x7834350> paste.cookies (<SimpleCookie: galaxysession='c6ca0ddb55be603a2c2796e41769bc2412b1c87332f54f840b1876eab065f3113e1ca20ff6a4ab5c'>, 'galaxysession=c6ca0ddb55be603a2c2796e41769bc2412b1c87332f54f840b1876eab065f3113e1ca20ff6a4ab5c') paste.evalexception <weberror.evalexception.middleware.EvalException object at 0x6dd11d0> paste.evalexception.debug_count 1316648566 paste.expected_exceptions [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions <paste.httpexceptions.HTTPExceptionHandler object at 0x7823090> paste.httpserver.thread_pool <paste.httpserver.ThreadPool object at 0x2b66f90> paste.printdebug_listeners [<cStringIO.StringO object at 0x6e24fb8>, <open file '<stderr>', mode 'w' at 0x7fb208c901e0>] paste.recursive.forward <paste.recursive.Forwarder from /> paste.recursive.include <paste.recursive.Includer from /> paste.recursive.include_app_iter <paste.recursive.IncluderAppIter from /> paste.recursive.script_name '' paste.remove_printdebug <function remove_printdebug at 0x6e23a28> paste.throw_errors True weberror.evalexception <weberror.evalexception.middleware.EvalException object at 0x6dd11d0> weberror.evalexception.debug_count 1316648566 webob._parsed_query_vars (MultiDict([('preview', 'True')]), 'preview=True') wsgi process 'Multithreaded'

2 2

OpenID Issues
by Greenway, Matthew [BSD] - MED 29 Sep '11

29 Sep '11

Hello Galaxy Developers: I noticed a few issues with the OpendID system. The first is that attempting to log in with an OpenID not associated with a galaxy account gives a "Server Error" page. Also using Flickr as an OP gives: "Failed to begin OpenID authentication: No usable OpenID services found for http://flickr.com" I haven't looked into whether Flickr has a different OP URL or none at all. I was wondering if someone was working on these or if they had been fixed in a different version. If not then I will work on fixing them and try to make my code available for others. Thanks, Matt Greenway This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal. Thank you.

2 1

database_engine_option_pool_size
by Simon Taylor 29 Sep '11

29 Sep '11

Hi, What exactly are the two options database_engine_option_pool_size and database_engine_option_max_overflow in universe_wsgi.ini? Occasionally I have a problem with galaxy timing out with an exception such as: TimeoutError: QueuePool limit of size 30 overflow 40 reached, connection timed out, timeout 30 Checking the process list I don't see more than a couple of galaxy jobs running so wouldn't expect 30+ connections to the db. The timeout doesn't seem to resolve itself either and the only way to get the site back is to kill and restart the process. Are there plans to handle the exception in any way and maybe reset the db connection, or at least display a user friendly error message? Thanks, Simon | | Systems utvikler | Centre for Integrative Genetics (CIGENE) | UNIVERSITETET FOR MILJØ- OG BIOVITENSKAP

2 1

October 2011 Events
by Dave Clements 29 Sep '11

29 Sep '11

Hello all, There are (at least) three Galaxy related events<http://wiki.g2.bx.psu.edu/Events>going on next month that you may want to consider attending. See the Events <http://wiki.g2.bx.psu.edu/Events> page for a full listing of upcoming events. If you have a Galaxy related event please send it to outreach(a)galaxyproject.org and we will help you get the word out. Date Topic/Event Venue/Location Comments Contact October 9-12 *Exploit your Data: Galaxy and the GMOD Tool Suite<http://www.amata.org.au/index.php/2011-conference/workshops> * AMATA 2011 <http://www.amata.org.au/index.php/2011-conference>, Canberra, Australia This workshop will cover both Galaxy and provide an overview of the GMOD<http://gmod.org/wiki/>tool suite. Many thanks to CSIRO and AMATA for sponsoring this workshop Dave Clements <http://wiki.g2.bx.psu.edu/DaveClements> October 12-14 *October 2011 GMOD Meeting and Galaxy Workshop<http://gmod.org/wiki/October_2011_GMOD_Meeting> * Ontario Institute for Cancer Research (OICR), Toronto, Canada Dan will give a Galaxy update at the meeting, and then lead a satellite workshop on Galaxy the day following the meeting. (Or, if you already know all about Galaxy you could attend workshops on BioMart<http://gmod.org/wiki/BioMart>and InterMine <http://gmod.org/wiki/InterMine>, two data sources for Galaxy.) Register now <http://gmod.org/wiki/October_2011_GMOD_Meeting> as the meeting and associated workshop have limited space. Dan Blankenberg October 23-24 *The Galaxy Track Browser: Transforming the Genome Browser from Visualization Tool to Analysis Tool<http://www.biovis.net/papers_abstracts/papers/111.html> * BioVis 2011 <http://www.biovis.net/>, Providence, Rhode Island, United States Learn about Trackster and other biological visualization. This is part of VisWeek 2011 <http://visweek.org/>. Jeremy Goecks Thanks, Dave Clements <http://wiki.g2.bx.psu.edu/DaveClements> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://galaxyproject.org/wiki/

1 0

Re: [galaxy-dev] Proxy login issues
by Alex R Bigelow 29 Sep '11

29 Sep '11

Fixed: just for fun I tried using Safari instead of Chrome, and it worked fine - this is probably a Chrome bug, not a Galaxy one. -Alex

1 0

Re: [galaxy-dev] uploading binary files checksum changes, Galaxy doing something to file?
by Paniagua, Eric 29 Sep '11

29 Sep '11

Hi everyone, I am experiencing a similar problem to that of Leandro. I have defined a new datatype deriving Data, and having extension "foobar". The simplest and first thing I'd like to do with it is create a dataset of this new type "Foobar" by uploading from my local machine to my Galaxy development server. When I do the upload, I specify the file format as "foobar" rather than "Auto-detect". The file itself is a zip archive containing a folder containing other files, but the extension is ".foobar" and not ".zip". I encounter at least 2 (I believe separate) problems while trying to upload. 1. I can see that on upload the first non-directory entry in the ".foobar" file is being extracted and replacing the original. It seems this is how *unknown* zip archives are "supposed" to be handled by Galaxy, but as to why that is, I haven't a clue. 2. (This is what's similar to Leandro's case.) The dataset_XXX.dat file produced by the upload is not actually the same as the file it is "copied" from in the uploaded archive. The checksums are different, and the sizes are different (the dataset is 1 byte longer). I did a diff of the hexdumps of the dataset file and the corresponding file from the uploaded archive, and discovered the following: 1. Every occurrence of '^M' (aka '\r' aka 0x0D) has been replaced with '\n' (aka 0x0A). 2. A newline ('\n' aka 0x0A) is added to the end of the file. This file from the archive is not a text file, it is binary, so any code in Galaxy that tries to fix line endings shouldn't be doing this. (Where) Are there such places? Leandro, have you solved your problem? If not, what do you see when you do this kind of comparison? I am unable to reproduce those changes by stepping through the code in upload.py which handles zip files (and replaces them by their first file member) using the same python installation used for running Galaxy. This suggests the problem is elsewhere. Does anyone know why this '\r' -> '\n' mapping is affecting this file? Does anyone know why the default behaviour for uploading zip archives is to keep one file arbitrarily and throw out the rest? Even with an argument in favor of this behaviour, why is there not a "unzippable_file_formats" list for exceptions to be made like there is for sniffing? Any enlightenment on these matters would be greatly appreciated. Best, Eric ________________________________________ From: Leandro Hermida [softdev(a)leandrohermida.com] Sent: Friday, September 16, 2011 10:03 AM To: Paniagua, Eric Subject: Re: [galaxy-dev] uploading binary files checksum changes, Galaxy doing something to file? Hi Eric, On Fri, Sep 16, 2011 at 3:58 PM, Paniagua, Eric <epaniagu(a)cshl.edu> wrote: > Hi Leandro, > > Is there an entry in your history for the upload? What file format does it show? Is there any chance your original file was zipped? If Galaxy detected it as a zip file on upload, it may have unzipped it and taken the first file in it as the dataset. Yes there is an history entry for the upload. The format it shows is the new datatype I created (in datatypes_conf.xml, subclassing Binary) which I selected in the drop-down menu before uploading the file in the Get Data form. It is not a zip file. > That's at least the version of your problem that I've run into before. Specifying the file format manually (rather than choosing Auto-detect) may help if it's a similar problem. I suspect the correct solution is to write a sniffer for your datatype to help ensure it is identified correctly by Galaxy, but I haven't tried this yet. > Essentially the basic question is, how do you tell Galaxy not to do or touch absolutely *anything* with an uploaded binary file??? The checksums should always match. > Best of luck, > Eric > ________________________________________ > From: galaxy-dev-bounces(a)lists.bx.psu.edu [galaxy-dev-bounces(a)lists.bx.psu.edu] on behalf of Leandro Hermida [softdev(a)leandrohermida.com] > Sent: Friday, September 16, 2011 9:42 AM > To: Galaxy Dev > Subject: [galaxy-dev] uploading binary files checksum changes, Galaxy doing something to file? > > Hi all, > > We tried to find something in the docs and mailing list no luck. We > created a new datatype the is a straight subclass of Binary and then > when we upload such a file in the Galaxy UI and check the checksums > between the original file and the file located in the Galaxy > database/files/... directory their checksums are different! > > What are we doing wrong? We simply want Galaxy to upload and no touch > the file at all. > > regards, > Leandro > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ >

2 1