- galaxy-dev - lists.galaxyproject.org

egg distribution error when running galaxy-central
by Clare Sloggett 23 Aug '12

23 Aug '12

Hi all, I'm trying to run galaxy-central on my laptop in order to play around with some changes, and I'm having trouble getting it to run. I can run galaxy-dist without problems and have been working with that (so its eggs are all installed already), but now I want to create a pull request so want to run galaxy-dist. I'm not trying to install any extra tools or data, just the code. I'm running on OSX 10.7.4 and using virtualenv. I have Enthought installed, and I assume I will be using its version of python by default. The default python seems to be 2.7.3. I'm using the same virtualenv environment for galaxy-dist and galaxy-central (though it doesn't seem to matter if I give galaxy-central its own environment, I see the same error). So the steps were: - create a virtualenv environment and activate it - get galaxy-dist and call run.sh - it asked me to build quite a lot of dependencies myself, which was just a matter of running the requested commands, and then it worked with no problems. - shut down galaxy-dist, and in another directory, get galaxy-central and call run.sh. I think it asked me to build a couple of dependencies, but then it gives up with the following: (galaxy_env)Clares-MacBook-Pro:galaxy-central clare$ sh run.sh --reload Some eggs are out of date, attempting to fetch... Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched Warning: pycrypto (a dependent egg of Fabric) cannot be fetched Warning: simplejson (a dependent egg of WebHelpers) cannot be fetched Fetched http://eggs.g2.bx.psu.edu/ssh/ssh-1.7.14-py2.7.egg One of Galaxy's managed eggs depends on something which is missing, this is almost certainly a bug in the egg distribution. Dependency "ssh" requires "pycrypto>=2.1,!=2.4" Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 30, in <module> c.resolve() # Only fetch eggs required by the config File "/Users/clare/galaxy/galaxy-central/lib/galaxy/eggs/__init__.py", line 345, in resolve egg.resolve() File "/Users/clare/galaxy/galaxy-central/lib/galaxy/eggs/__init__.py", line 168, in resolve dists = pkg_resources.working_set.resolve( ( self.distribution.as_requirement(), ), env, self.fetch ) File "/Users/clare/galaxy/galaxy_env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 569, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (ssh 1.7.14 (/Users/clare/galaxy/galaxy-central/eggs/ssh-1.7.14-py2.7.egg), Requirement.parse('pycrypto>=2.1,!=2.4')) Fetch failed. Any idea what is causing this? Thanks, Clare -- Clare Sloggett Research Fellow / Bioinformatician Life Sciences Computation Centre Victorian Life Sciences Computation Initiative University of Melbourne, Parkville Campus 187 Grattan Street, Carlton, Melbourne Victoria 3010, Australia Ph: 03 903 53357 M: 0414 854 759

4 8

metaphlan + krona = working
by Langhorst, Brad 23 Aug '12

23 Aug '12

Well.. it turned out to be pretty easy thanks to the efforts of Dannon and Brian Ondov I've added the metaphlan2krona.py script to the metaphlan toolshed repo. diff -r e88fa24fa837 metaphlan.xml --- a/metaphlan.xml Wed Jun 06 10:41:23 2012 -0400 +++ b/metaphlan.xml Wed Aug 22 21:36:21 2012 -0400 @@ -14,7 +14,7 @@ </command> <inputs> - <param format="fasta" name="input" type="data" label="Input metagenome (multi-fasta of metagenomic reads, loaded with the Get Data module, see below for an example)"></param> + <param format="fasta,fastq,fastqsanger" name="input" type="data" label="Input metagenome (multi-fasta of metagenomic reads, loaded with the Get Data module, see below for an example)"></param> <param name="PresetsForBowtie2" type="select" format="text"> <label>Sensitivity options for read-marker similarity (as described by BowTie2)</label> <option value="very-sensitive-local">Very Sensitive Local</option> diff -r e88fa24fa837 metaphlan2krona.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/metaphlan2krona.py Wed Aug 22 21:36:21 2012 -0400 @@ -0,0 +1,49 @@ +#!/usr/bin/env python + +# ============================================================================== +# Conversion script: from MetaPhlAn output to Krona text input file +# Author: Daniel Brami (daniel.brami(a)gmail.com<mailto:daniel.brami@gmail.com>) +# ============================================================================== + +import sys +import optparse +import re + +def main(): + #Parse Command Line + parser = optparse.OptionParser() + parser.add_option( '-p', '--profile', dest='profile', default='', action='store', help='The input file is the MetaPhlAn standard result file' ) + parser.add_option( '-k', '--krona', dest='krona', default='krona.out', action='store', help='the Krons output file name' ) + ( options, spillover ) = parser.parse_args() + + if not options.profile or not options.krona: + parser.print_help() + sys.exit() + + re_candidates = re.compile(r"s__|unclassified\t") + re_replace = re.compile(r"\w__") + re_bar = re.compile(r"\|") + + metaPhLan = list() + with open(options.profile,'r') as f: + metaPhLan = f.readlines() + f.close() + + krona_tmp = options.krona + metaPhLan_FH = open(krona_tmp, 'w') + + for aline in (metaPhLan): + if(re.search(re_candidates, aline)): + x=re.sub(re_replace, '\t', aline) + x=re.sub(re_bar, '', x) + + x_cells = x.split('\t') + lineage = '\t'.join(x_cells[0:(len(x_cells) -1)]) + abundance = float(x_cells[-1].rstrip('\n')) + + metaPhLan_FH.write('%s\n'%(str(abundance) + '\t' + lineage)) + + metaPhLan_FH.close() + +if __name__ == '__main__': + main() diff -r e88fa24fa837 metaphlan_to_krona.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/metaphlan_to_krona.xml Wed Aug 22 21:36:21 2012 -0400 @@ -0,0 +1,17 @@ +<tool id="meta_to_krona" name="MetaPhlAn to Krona" version="1.0.0"> + <description>Converter</description> + <command interpreter="python"> +metaphlan2krona.py -p $input -k $output + </command> + <inputs> + <param name="input" type="data" format="tabular" label="Input MetaPhlAn File"/> + </inputs> + <outputs> + <data format="tabular" name="output" label="${tool.name} on ${on_string}" /> + </outputs> + <tests> + </tests> + <help> + MetaPhlAn to Krona Converter + </help> +</tool> To get it to work with krona i extended the krona repo like this diff -r 42c899125802 krona/krona.xml --- a/krona/krona.xml Mon Mar 19 17:39:59 2012 -0400 +++ b/krona/krona.xml Wed Aug 22 21:39:17 2012 -0400 @@ -7,15 +7,18 @@ ktImportBLAST -e ${type.factor} ${type.random} + #else if $type.program == 'krona': + ktImportText #else ktImportTaxonomy ${type.summary} #end if - -o $output + -o $output ${type.include} + #unless $type.program == 'krona' + -d $depth + #end unless - -d $depth - ## uncomment for isolated intranets (see README) ## ##-u /static/krona @@ -36,6 +39,7 @@ <option value="galaxy">Galaxy taxonomic representation</option> <option value="blast">Tabular BLAST results</option> <option value="taxonomy">Taxonomy ID list</option> + <option value="krona">Krona format</option> </param> <when value="galaxy"> <param name="factor" type="hidden" value=""/> @@ -51,6 +55,10 @@ <param name="summary" type="boolean" truevalue="-m 1" falsevalue="" label="Summarized" help="The first column is counts rather than query IDs."/> <param name="include" type="boolean" truevalue="-i" falsevalue="" label="Include reads with no hits"/> </when> + <when value="krona"> + <param name="factor" type="hidden" value=""/> + <param name="include" type="hidden" value=""/> + </when> </conditional> <param name="depth" label="Depth" type="integer" help="The maximum depth to show in the chart (0 for unlimited)." value="0"/> <conditional name="datasets"> now i see this kind of graph [cid:D121407D-64B6-4D03-AE23-9DC53B52B5DC@home.langhorst.com] one more item.. had to add sanitize_all_html = False to universe_wsgi.ini brad -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com> 978-380-7564

1 0

metaphlan + krona = ?
by Langhorst, Brad 22 Aug '12

22 Aug '12

I have installed both krona and metphlan from the toolshed to my local insttance. both seem to work ok, but I don't seen an easy way to get the metaphlan output into a form that krona can accept. has anybody solved this problem already? Brad -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com> 978-380-7564

1 0

Re: [galaxy-dev] [galaxy-user] LimitInternalRecursion
by Nate Coraor 22 Aug '12

22 Aug '12

On Aug 21, 2012, at 4:44 PM, Adhemar wrote: > Hi, > There's this error message in my apache log: > > [error] Request exceeded the limit of 10 subrequest nesting levels due to probable confguration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace., referer: ... > > > When I turn the apache debug log on it also shows: > > > [debug] core.c(3072): r->uri = /galaxy/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:8080/root/history_item_updates, referer: ... > > > It seems like mod_rewrite is somehow misconfigured, but I couldb't figure out what it is. > > Here is my httpd.con mod_rewrite conf: > > > RewriteEngine on > RewriteRule ^/galaxy$ /galaxy/ [R] > RewriteRule ^/galaxy/static/style/(.*) /opt/bioinformatics/share/galaxy-central/static/june_2007_style/blue/$1 [L] > RewriteRule ^/galaxy/static/scripts/(.*) /opt/bioinformatics/share/galaxy-central/static/scripts/packed/$1 [L] > RewriteRule ^/galaxy/static/(.*) /opt/bioinformatics/share/galaxy-central/static/$1 [L] > RewriteRule ^/galaxy/favicon.ico /opt/bioinformatics/share/galaxy-central/static/favicon.ico [L] > RewriteRule ^/galaxy/robots.txt /opt/bioinformatics/share/galaxy-central/static/robots.txt [L] > RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P] Hi Adhemar, I've moved this over to the galaxy-dev list since it's concerning a local installation. Are there any other RewriteRules defined? Are these directives inside another block, or just inside a <VirtualHost>? It would be useful to see your entire VirtualHost config. --nate > > > Could you please help me to debug this? > > Thank you, > Adhemar > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/

3 4

Re: [galaxy-dev] galaxy-dev Digest, Vol 74, Issue 21
by Adhemar 22 Aug '12

22 Aug '12

Here is a copy of the whole VirtualHost section. Thanks, Adhemar <VirtualHost lbas046.cnptia.embrapa.br:443> ServerName lbas046.cnptia.embrapa.br ErrorLog logs/galaxy_ssl_error_log TransferLog logs/galaxy_ssl_access_log LogLevel warn SSLEngine on SSLCertificateFile /etc/pki/tls/certs/lbas046.cnptia.embrapa.br.crt SSLCertificateKeyFile /etc/pki/tls/private/lbas046.cnptia.embrapa.br.key.insecure SSLCACertificateFile /etc/pki/tls/certs/cacert.pem RewriteEngine on RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /opt/bioinformatics/share/galaxy-central/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /opt/bioinformatics/share/galaxy-central/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /opt/bioinformatics/share/galaxy-central/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /opt/bioinformatics/share/galaxy-central/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /opt/bioinformatics/share/galaxy-central/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P] <Location "/galaxy"> AuthType Basic AuthName Galaxy_pam AuthPAM_Enabled on Require valid-user RewriteEngine On RewriteCond %{LA-U:REMOTE_USER} (.+) RewriteRule . - [E=RU:%1] RequestHeader set REMOTE_USER "%{RU}e" env=RU </Location> </VirtualHost> 9. Re: [galaxy-user] LimitInternalRecursion (Nate Coraor) ------------------------------ > > Message: 9 > Date: Wed, 22 Aug 2012 10:46:30 -0400 > From: Nate Coraor <nate(a)bx.psu.edu> > To: azneto(a)gmail.com > Cc: galaxy-dev(a)lists.bx.psu.edu > Subject: Re: [galaxy-dev] [galaxy-user] LimitInternalRecursion > Message-ID: <BD7E4946-0184-4887-9E3D-17946ADCE79A(a)bx.psu.edu> > Content-Type: text/plain; charset=iso-8859-1 > > On Aug 21, 2012, at 4:44 PM, Adhemar wrote: > > > Hi, > > There's this error message in my apache log: > > > > [error] Request exceeded the limit of 10 subrequest nesting levels due > to probable confguration error. Use 'LimitInternalRecursion' to increase > the limit if necessary. Use 'LogLevel debug' to get a backtrace., referer: > ... > > > > > > When I turn the apache debug log on it also shows: > > > > > > [debug] core.c(3072): r->uri = /galaxy/proxy: > http://localhost:8080/proxy:http://localhost:8080/proxy:http://localhost:80…, > referer: ... > > > > > > It seems like mod_rewrite is somehow misconfigured, but I couldb't > figure out what it is. > > > > Here is my httpd.con mod_rewrite conf: > > > > > > RewriteEngine on > > RewriteRule ^/galaxy$ /galaxy/ [R] > > RewriteRule ^/galaxy/static/style/(.*) > /opt/bioinformatics/share/galaxy-central/static/june_2007_style/blue/$1 [L] > > RewriteRule ^/galaxy/static/scripts/(.*) > /opt/bioinformatics/share/galaxy-central/static/scripts/packed/$1 [L] > > RewriteRule ^/galaxy/static/(.*) > /opt/bioinformatics/share/galaxy-central/static/$1 [L] > > RewriteRule ^/galaxy/favicon.ico > /opt/bioinformatics/share/galaxy-central/static/favicon.ico [L] > > RewriteRule ^/galaxy/robots.txt > /opt/bioinformatics/share/galaxy-central/static/robots.txt [L] > > RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P] > > Hi Adhemar, > > I've moved this over to the galaxy-dev list since it's concerning a local > installation. > > Are there any other RewriteRules defined? Are these directives inside > another block, or just inside a <VirtualHost>? It would be useful to see > your entire VirtualHost config. > > --nate > > > > > > > Could you please help me to debug this? > > > > Thank you, > > Adhemar

1 0

Re: [galaxy-dev] admin_user not being seen
by Paul-Michael Agapow 22 Aug '12

22 Aug '12

> A peculiar quirk - I'm configuring a new Galaxy system for production > use and can't get it to recognise my account as an admin, i.e. the admin > menu doesn't appear after I've added my email address to the admin_user > line in the universe file. Of course, I've restarted the instance > multiple times, checked spelling and so forth. It's just not happening. > Complication: we're using one Galaxy instance as the webserver with > another as a job runner. I don't know if this might cause issues. Thanks to Hans-Rudolf & Peter for possible solutions. The answer actually turned out to be my fault - as said, we're running Galaxy as a one db / two installation with a webserver/job runner ... and I was putting the admin_user in the wrong universe file. Mea culpa, mea maxima culpa. ------ Paul Agapow (paul-michael.agapow(a)hpa.org.uk) Bioinformatics, Health Protection Agency (UK) ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************

1 0

FTP upload - symlink to uploaded data
by Joachim Jacob 22 Aug '12

22 Aug '12

Dear devs, Question about FTP upload: is it correct that Galaxy copies the data from the FTP upload directory to galaxy database instead of symlinking? Can I set somewhere the option to symlink instead of copying? I run galaxy in a small virtual machine. The Galaxy database is on a network share. With 'get data' - 'Upload file', when importing the files I have uploaded with FTP, there is a process filling up the /tmp directory with the file I want to import. Kind regards, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 5

Unable to read builds file: publicbuilds.txt
by Andrew Warren 21 Aug '12

21 Aug '12

Hello all, With recent additions to lib/galaxy/util/__init__.py I have started getting the following in the std out of bowtie and cufflinks (and probably other tools as well): ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ensembl/builds.txt' Indeed I do not have files for publicbuilds.txt or builds.txt created. Is the best way to deal with this just to create these files (even though I am not using them)? Are these files being read every time a web transaction goes down due to the property in lib/galaxy/web/framework/__init__.py ? @property def ucsc_builds( self ): Thanks, Andrew Warren

1 0

Re: [galaxy-dev] [galaxy-user] Aggregate collections of files (example: Heatmaps)
by Peter Cock 21 Aug '12

21 Aug '12

On Tue, Aug 21, 2012 at 4:46 PM, Ted Goldstein <ted(a)soe.ucsc.edu> wrote: > I am developing several tools that will need to read and write multiple > data files at once. For example, Eisen Cluster produces a heatmap > which consists of three files: a .cdt file, .atr file and a gtr file which are > the underlying heatmap and the array tree and the gene tree. All three > files need to be kept together. I guess I could wrap them in a zip file > and pack and unpack them. The heatmap is not just a view only > object. Some tools, such as cuttree, would extract one tree and then > aggregate genes (or arrays) below a certain depth and create a new > trio of files. Is there support for (or plans for creating) any aggregate > data types? Hi Ted, There is support for composite datatypes, so this should be possible. http://wiki.g2.bx.psu.edu/Admin/Datatypes/Composite%20Datatypes This kind of discussion is normally directed to the galaxy-dev list (CC'd) Peter

1 0

Fwd: Re: [galaxy-user] Linking to Compressed Data
by Hans-Rudolf Hotz 21 Aug '12

21 Aug '12

moving to dev...... Hi Jen and Brandon well, I don't agree (or maybe I misunderstood the question/answer): we work a lot with compressed data (ie g'zipped fastq files from the sequencer), and link them as a Galaxy data library. We preselect 'fastq' as file format. And then, the first tool in our NGS analysis expects a gzipped fastq file. Regards, Hans -------- Original Message -------- Subject: Re: [galaxy-user] Linking to Compressed Data Date: Mon, 20 Aug 2012 10:32:26 -0700 From: Jennifer Jackson <jen(a)bx.psu.edu> To: Branden Timm <btimm(a)wisc.edu> CC: <galaxy-user(a)lists.bx.psu.edu> Hi Branden, Data in a library needs to be uncompressed in order for Galaxy to index and have access to it correctly. I updated our wiki for the "upload by filesystem link" option: http://wiki.g2.bx.psu.edu/Admin/Data%20Libraries/Uploading%20Library%20Files Good question! Jen Galaxy team On 8/15/12 9:09 AM, Branden Timm wrote: > Hi All, > Is it possible to link to compressed files in a Galaxy data library? > We receive all of our NGS data in bz2 or gzip format for obvious > reasons, just wondering if I have to decompress it on the filesystem > before I link to it or not. Thanks! > > -- > Branden Timm > btimm(a)glbrc.wisc.edu > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

4 5