September 2013 - galaxy-dev - lists.galaxyproject.org

(no subject)
by Maria Hoffman 21 Sep '13

21 Sep '13

Hello, Galaxy will not load my histories and as a result I can not run the analysis that I need to run. I have tried repeatedly since 7:00 AM EST to load them with no luck. I worked on this history yesterday and everything was fine. Maria

1 0

testtoolshed.g2.bx.psu.edu not compatible with galaxy-central stable
by Richard Burhans 21 Sep '13

21 Sep '13

my galaxy instance: parent: 10587:1f34ec186fcf Pack script. branch: stable commit: 3 modified, 91 unknown update: (current) testtoolshed.g2.bx.psu.edu: parent: 10592:ea0c57f1cdf0 tip Fix for unicode values in rst_to_html. branch: default commit: 34 unknown (clean) update: (current) Try to install this repository: http://testtoolshed.g2.bx.psu.edu/view/rico/genome_diversity URL: https://oocyte.bx.psu.edu/admin_toolshed/prepare_for_install? tool_shed_url=http://testtoolshed.g2.bx.psu.edu/ &repository_ids=d43c67507430e220&changeset_revisions=b63d27fdf040 File '/scratch/galaxy/home/oocyte/galaxy_oocyte/eggs/WebError-0.8a- py2.7.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/eggs/Paste-1.7.5.1- py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/galaxy/web/ framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/eggs/Paste-1.7.5.1- py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/galaxy/web/ framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/galaxy/web/ framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/galaxy/web/ framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/galaxy/webapps/ galaxy/controllers/admin_toolshed.py', line 911 in prepare_for_install includes_tool_dependencies ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/tool_shed/util/ common_install_util.py', line 85 in get_dependencies_for_repository installed_rd, missing_rd = get_installed_and_missing_repository_dependencies_for_new_install ( trans, repo_info_tuple ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/tool_shed/util/ common_install_util.py', line 214 in get_installed_and_missing_repository_dependencies_for_new_install tool_shed, name, owner, changeset_revision, prior_installation_required = suc.parse_repository_dependency_tuple ( rd_tup ) File '/scratch/galaxy/home/oocyte/galaxy_oocyte/lib/tool_shed/util/ shed_util_common.py', line 1197 in parse_repository_dependency_tuple prior_installation_required = str( prior_installation_required ) UnboundLocalError: local variable 'prior_installation_required' referenced before assignment contains_error = False repository_dependency_tuple = ['http://testtoolshed.g2.bx.psu.edu/', 'package_clapack_3_2_1', 'rico', '56a949e5f998', 'True', 'False'] -rico

3 4

datacache & bowtie2 for mm9 ?
by Curtis Hendrickson (Campus) 20 Sep '13

20 Sep '13

Folks, First, I wanted to thank you for making the datacache available (http://wiki.galaxyproject.org/Admin/Data%20Integration; rsync://datacache.g2.bx.psu.edu) It's a great resource. However, what is the best way to stay abreast of changes to what's in datacache, and understand how these indexes are computed? We are currently upgrading to bowtie2, but I notice that the bowtie2 indices for mm9, which used to be in rsync://datacache.g2.bx.psu.edu/indexes/mm9/mm9*/bowtie2_index have been removed, and only the hg19 genome has bowtie2 indices. Why only that one, and not the others? Where are the scripts you use to make these indices, in case I want to create bowtie2 indices for other So, how do I find out *why* they were removed? (Can I safely use the copy I have, or was there a problem with them?) More generally, how do I understand the policies and logic behind the datacache indices, and be notified of changes, short of running my own periodic rsync/diff? Finally, since I'm doing "reproducible research" is anything planned for systematically versioning genome indices, so I can easily tell what version of a system (ie, what BWA version) was used to create the index, and be sure that an index will not suddenly disappear. Thanks, Curtis Research Associate/CTSA-Informatics Team University of Alabama at Birmingham

2 1

Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
by Carlos Borroto 20 Sep '13

20 Sep '13

On Thu, Sep 19, 2013 at 5:15 PM, Björn Grüning <bjoern.gruening(a)pharmazie.uni-freiburg.de> wrote: > Hi Peter and Carlos, > >> On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto >> <carlos.borroto(a)gmail.com> wrote: >> > I did an extra test. Started with a clean 'galaxy-dist'. This time >> > both repositories fail with the same error. I guess before something >> > was cached for the repository with version 0.1.4. >> > >> > I used biopython repository as a guide to write my tool dependency definition: >> > http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 >> > >> > I can confirm biopython repository is failing to install for me with >> > exactly the same error. I wonder if a recently addition in the test >> > toolshed broke the treatment of "prior_installation_required". >> > >> > Thanks, >> > Carlos >> >> Could be - note that currently Biopython isn't currently >> installing properly on the Test Tool Shed due to ALTAS >> failing (a requirement of NumPy which is a requirement >> of Biopython). Dave and Bjoern are I think looking at this >> already... > > I can't do much I tested it again and for me its working fine on my > computers I have at hand ... sorry. > In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: $ tail -n 3 ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log x ATLAS/tune/lapack/lanbsrch.c tar: Error exit delayed from previous errors. ############################################# This is the relevant part I can find in Galaxy's log: [localhost] local: tar xfvj atlas3.10.1.tar.bz2 Warning: local() encountered an error (return code 1) while executing 'tar xfvj atlas3.10.1.tar.bz2' After noticing this I got what I'm guessing is the original file from sourceforge: http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10… I can confirm that trying to untar this file fails with the exact same error. However, on Ubuntu 13.04 untaring this file works just fine. On Ubuntu 13.04 the error I see is: STDERR It appears you have cpu throttling enabled, which makes timings unreliable and an ATLAS install nonsensical. Aborting. See ATLAS/INSTALL.txt for further information ############################################# Björn, you said cpu throttling can be easily disable on Ubuntu. Can you comment how? Do I need to disable it completely, or should the install script do it when installing? Thanks, Carlos

4 10

Using tool_dependencies.xml files directly via Galaxy API
by Peter Cock 20 Sep '13

20 Sep '13

Hello all, I'm looking at how to automatically script setting up a Galaxy instance with selected tools with dependencies as part of automated testing [*]. Can the Galaxy API be used to run the dependency instructions in a tool_dependencies.xml file (without using a Tool Shed)? Or more generally, can the Galaxy API be used to install a Tool Shed package from disk (i.e. point it at the files which would normally be uploaded to a Tool Shed)? Or, for an automated installation of local tools must I first setup a local Tool Shed and upload the tools there? Thanks, Peter [*] I'm using the TravisCI system for automated testing of my Galaxy tools now hosted on GitHub, which works by running the tests on a clean virtual machine: http://blastedbio.blogspot.com/2013/09/using-travis-ci-for-testing-galaxy-t… This means I need to automate the installation of Galaxy (easy), the tools I wish to test and their dependencies. This configuration would be less effort (and a more useful test) if it could mimic tool and dependency installation from the Tool Shed by reading my dependency XML files.

1 0

UnboundLocalError: local variable 'prior_installation_required' referenced before assignment
by Carlos Borroto 19 Sep '13

19 Sep '13

Hi, I got this error when trying to install a repository I created defining a tool dependency. The odd thing is the exact same definition worked for the previous version of the same tool. This is the repository failing to install, see full trace below: http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_5 This is the content of the only file in the repository: <?xml version="1.0"?> <tool_dependency> <package name="biopython" version="1.61"> <repository name="package_biopython_1_61" owner="biopython" prior_installation_required="True" /> </package> <package name="ngs-tools" version="0.1.5"> <install version="1.0"> <actions> <action type="setup_virtualenv">ngs-tools==0.1.5</action> </actions> </install> </package> </tool_dependency> This is the repository for the previous version of this package currently in the test toolshed and can be installed without issues: http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_4 Diff between tool_dependencies.xml files: $ diff package_ngs_tools_0_1_4/tool_dependencies.xml package_ngs_tools_0_1_5/tool_dependencies.xml 6c6 < <package name="ngs-tools" version="0.1.4"> --- > <package name="ngs-tools" version="0.1.5"> 9c9 < <action type="setup_virtualenv">ngs-tools==0.1.4</action> --- > <action type="setup_virtualenv">ngs-tools==0.1.5</action> Full trace: URL: http://localhost:8080/admin_toolshed/prepare_for_install?tool_shed_url=http… File '/home/cborroto/src/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/home/cborroto/src/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/cborroto/src/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/cborroto/src/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/cborroto/src/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/home/cborroto/src/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/home/cborroto/src/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 911 in prepare_for_install includes_tool_dependencies ) File '/home/cborroto/src/galaxy-dist/lib/tool_shed/util/common_install_util.py', line 85 in get_dependencies_for_repository installed_rd, missing_rd = get_installed_and_missing_repository_dependencies_for_new_install( trans, repo_info_tuple ) File '/home/cborroto/src/galaxy-dist/lib/tool_shed/util/common_install_util.py', line 214 in get_installed_and_missing_repository_dependencies_for_new_install tool_shed, name, owner, changeset_revision, prior_installation_required = suc.parse_repository_dependency_tuple( rd_tup ) File '/home/cborroto/src/galaxy-dist/lib/tool_shed/util/shed_util_common.py', line 1183 in parse_repository_dependency_tuple prior_installation_required = str( prior_installation_required ) UnboundLocalError: local variable 'prior_installation_required' referenced before assignment I clearly have to be missing something here. Any help? Thanks, Carlos

4 8

Part Time Job opening at GenePeeks
by Adam Cohn 19 Sep '13

19 Sep '13

Hi everyone, I am posting this position description for a Part Time - Galaxy (BioLinux) Server Administrator / Python Programmer. We are a really great company in an interesting space. Please check out this video for a good overview of what we do. (We also have some full time openings that range from computational geneticist to senior scientist, if you are interested, send me an e-mail and I will send you more information.) GenePeeks: Protecting our Children http://vimeo.com/68860139 Company GenePeeks is a genetic information company with a mission to help families protect the next generation. Our patent-pending technology reduces the risk of more than 600 severe, pediatric diseases by digitally weaving together the DNA of prospective parents and analyzing thousands of "digital babies" pre-pregnancy. Our initial focus is the donor sperm industry, where we help sperm bank clients avoid donor matches that carry a high risk of passing on life-threatening and life-altering diseases. GenePeeks was founded by Princeton genetics professor Lee Silver and Anne Morriss, an entrepreneur and mother whose first child was born with a rare disease that could have been identified with GenePeeks technology. For more information, please visit genepeeks.com. GenePeeks is a unique genetic information company with a mission to help families protect the health of their future children. We are searching for a Part Time Galaxy (BioLinux) Server Administrator to join our growing team and help develop leading-edge technologies in a fast-paced, early-stage environment. Description: Manage and maintain several cloud based BioLinux instances running Galaxy on Amazon Web Services Manage and maintain a source code control baseline on BitBucket Manage and maintain internal wiki site using mediawiki Help install and maintain new cloud based technologies as required Perform code reviews of Python scripts Assist in organizing the code base and proper source code Manage backup and disaster recovery planning Requirements Must be fast learning and responsive Must have experience using BioLinux Must have Python coding experience Must have experience using BitBucket Background in Genetics or Molecular Biology is a plus Sense of humor required Details Position is part time and will bill hourly Hours will fluctuate (estimate about 10 hours a week) Location in the New York area is a plus, but will consider remote working arrangements for the right candidate Hourly Rate Commensurate with experience and efficiency If you are interested please contact me directly at adamcohn(a)genepeeks.com -- Adam S. Cohn Chief Technology Officer / Chief Operations Officer GenePeeks 175 Varick Street, New York, NY (646) 664-4268 (703) 473-8811 (mobile) adamcohn(a)genepeeks.com www.genepeeks.com

1 0

Towards Galaxy Linux (not) ? [was [RFC] Storing of tarballs and patches for tool_dependencies to enable reproducibility]
by Guest, Simon 19 Sep '13

19 Sep '13

Hi Bjoern, I can see man years of effort being spent on solving this problem within Galaxy. I was going to title this email "Danger, Will Robinson", but I didn't want to be disrespectful. I think the path being embarked upon, tool dependency packaging, tool versioning, reproducibility, and long term archive of source tarballs is going to lead inevitably to creation of a new Linux distribution, which I guess will be called Galaxy Linux. The packaging and archival you are talking about is exactly the service provided by a Linux distribution. There's well established infrastructure to handle this, and years of experience have gone into solving the problems well. Surely the number of Linux distributions in the world now exceeds 100, but I don't see that the world will become a better place if we increase that number by one more. We at AgResearch can't be alone in having to pick a Linux distribution to run from the short list supported by our hardware vendor. I can't see Galaxy Linux being on that list anytime soon. So we have to make Galaxy run on the particular distribution we have here. For us that's CentOS 6. Now, I see scary mention of platform independence as a goal for Galaxy packaging, which I interpret as "will run on any Linux distribution". I think that's essentially infeasible. All you can do is write install scripts which you hope are portable (by following as many best practices as you know about), and then work patiently with users on strange platforms, to adapt each install script to work on that platform also. I think this is not a good use of anyone's time. How many Linux distributions do the Galaxy community actually care about today? The RHEL family is surely important, as is Ubuntu LTS. Anything else? I'd be quite interested to understand this, as it provides a context for the discussion, and ensures we're not just solving a hypothetical problem. I'm just starting work on a native packaging infrastructure for Galaxy, that will enable tool dependencies to use defined versions of natively installed packages. That frees me up to make my packages work nicely on the RHEL family. It looks like the RPMs themselves (including SRPMs obviously) will be hosted by the CentOS project before too long. Once they're there, they can easily be archived forever. Anyone else on that platform is welcome to use the same infrastructure. Then, all we really need is someone to handle the packaging effort for the other major Linux distributions (a small number, I hope), and the problem is essentially solved. Getting the Bio-Linux team interested in multi-version packaging would be a great next step. I'll be posting here when I have progress to report on my native packaging effort. cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

5 4

August 2013 Galaxy Update Newsletter is out
by Dave Clements 19 Sep '13

19 Sep '13

2 5

UnicodeDecodeError - suggested note regarding UTF8 encoding in postgres databases
by Guest, Simon 19 Sep '13

19 Sep '13

I've finally joined the happy throng suffering from Unicode errors. In my particular case, running the deseq tool from the main toolshed brought down my Galaxy server, and it wouldn't restart. The traceback in paster.log contained this Traceback (most recent call last): [snip] File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2903, in __init__ self._init_metadata() File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 3306, in _init_metadata self.__buffer_rows() File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 3326, in __buffer_rows self.__rowbuffer = collections.deque(self.cursor.fetchmany(size)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 162: ordinal not in range(128) The same error occurred each time I tried to restart the galaxy server until I restored the galaxy database from last night's postgres backup. The problem was my galaxy database in postgres was using SQL_ASCII encoding, which turns out to be a bad idea if you're ever going to write Unicode characters there. (I'm quite disappointed that this is the default encoding for postgres databases on CentOS 6.) The fix was to dump the galaxy database, recreate it with UTF8 encoding, and reload it. Now everything works, and that one tool doesn't crash my galaxy server. I thought it would be worth noting that postgres databases should be created with UTF8 encoding on this page: http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Swi… I think the right thing to do is to ensure you set the default database encoding for postgres when you run initdb. You only get one chance to get this right. After that, a way to change the default database encoding to UTF8 is given here: https://gist.github.com/ffmike/877447 Scary stuff, but it worked well enough for me. Alternatively, if you didn't think about encodings when you ran initdb, you can think about it every time you run createdb. Hope this helps someone. cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

1 0