- galaxy-dev - lists.galaxyproject.org

web application scaling
by Edward Kirton 13 Apr '10

13 Apr '10

Hello, could I please get some more information on running multiple web server processes? I'd like to set up our local installation on a web server farm. I'm currently using SGE for all the heavy lifting, but am wary of having a single server on the front end. Any guidance would be appreciated. I'm currently using MySql for the db but am not adverse to switching to PostgreSQL if there was a reason to do so. And anything you'd like to share from your experience regarding hardware selection would be appreciated as well; I'm currently thinking of ordering several "inexpensive" servers. Have you noticed an "optimal" number of web servers, before traffic jams at the db server start to affect performance? Thanks, Ed Kirton US DOE JGI

2 1

Changes to galaxysession cookie?
by Assaf Gordon 13 Apr '10

13 Apr '10

Hello, A quick question: If I recall correctly, it was possible to set the galaxy session cookie name with the "session_key" variable in "universe_wsgi.ini". I'm not sure about that, but I think it was possible, because I'm running two instances (at least) from the same server, and I explicitly set the cookie names to by different. Now it seems that the cookie is always "galaxysession" - was that changed recently, or am I missing something? One reason that I ask: The "path" of the cookie is always hard-coded to "/" (probably in lib/galaxy/web/framework/__init__.py:set_cookie as the default parameter) - so when I run two galaxies from the same server (with different "prefix" filter) - they are now forced to share the same session cookie. I think this is causing some history related problems in one of my servers: galaxy always thinks there is no active history selected. Example: 1. I click "list saved histories" 2. Select a history 3. The history pane is updated. 4. I click on "refresh" history 5. a new, "unnamed history" is created, without any datasets. The log shows error like: alaxy.web.framework DEBUG 2010-04-04 23:48:39,512 Error: this request returned None from get_history(): http://rave.cshl.edu/devglx/history/list Any ideas ? Thanks, -gordon

2 2

Hack for HTTPS support for Genome Browser
by Assaf Gordon 13 Apr '10

13 Apr '10

Hi, The files "intervals.py" and "genetics.py" in "/lib/galaxy/datatypes/" contains "hacks" for turning HTTPS into HTTP because the genome-browser didn't support https. I think that recent versions of the genome browser do support HTTPS (at least there's an option for that in the source code), so this hack can be removed, -gordon

2 1

changing "Database/Build" back to 'unspecified'
by Hans-Rudolf Hotz 13 Apr '10

13 Apr '10

Hi I have a small problem with changing the "Database/Build" setting of a dataset: When I create a new history item with the tool 'Upload File from your computer' the "Genome" is unspecified, unless I change the value from 'Click to Search or Select Build' to any of the genomes and/or builds. I can always change it later using the 'pencil' icon. However, once it has changed to any of the genomes and/or builds I can't change it back to "?". This is the case on 'http://main.g2.bx.psu.edu/' as well on our local installation (running changeset 3545). I do have the line "? unspecified (?)" in the file ~/galaxy_dist/tool-data/shared/ucsc/builds.txt Changing back to 'unspecified' was possible before (ie running changeset 2833) when it was a simple drop down menu. Is there a possibility to fix this? Regards, Hans

2 1

Re: [galaxy-dev] Weird problem with importing datasets
by Chris Cole 13 Apr '10

13 Apr '10

On 09/03/10 15:42, Chris Cole wrote: > On 09/03/10 14:53, Nate Coraor wrote: >> Chris Cole wrote: >>> Anyone got any ideas on this? >>> >>> It's still not working (following the most recent updates) and we've >>> got some NGS data to import into Galaxy. >> >> Hi Chris, >> >> To use this feature, the contents of library_import_dir should >> themselves be directories (or symlinks to the same). Selecting that >> directory from the dropdown in the library upload form will then import >> the contents of those directories. > > Ah, right. That's not clear in the docs. > > How come it works with locally symlinked files, then? Just checked it again (sorry I shouldn't have sent the previous email) with a directory and it still isn't working. www-galaxy@ge-002: tmp> cd ~/data_import/ www-galaxy@ge-002: data_import> ls README www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH www-galaxy@ge-002: data_import> ls TOH/ 0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt Dd_inuc_1.txt Dmono_1.txt processed 0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt Dd_inuc_2.txt Dmono_2.txt www-galaxy@ge-002: data_import> ls -l total 96 -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH -> /homes/pschofield/data/TOH I got the TOH option in the 'Upload directory of files', which I selected, but the upload failed. Again, this is the error I get when selecting one of the filenames in the dataset: Traceback (most recent call last): File "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 326, in __main__() File "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 318, in __main__ add_file( dataset, error Thanks for help. Chris

3 7

Automated deployment of Galaxy associated data files
by Brad Chapman 12 Apr '10

12 Apr '10

Galaxy folks; We recently moved our local Galaxy instance to a new server, and in the process developed an automated script to install and integrate associated data libraries and programs. I thought this might be of interest to other developers on the list and is available at: http://github.com/chapmanb/bcbb/blob/master/galaxy/galaxy_fabfile.py This uses Fabric (http://docs.fabfile.org) to: - Install 3rd party tools, focused around next-gen analysis: bowtie, bwa, maq, samtools, fastx-toolkit, UCSC programs - Download and index genome builds. There are provided classes to handle UCSC, Ensembl and NCBI downloads. - Update associated tool-data files with links to the builds and indexes. It works incrementally, skipping programs and libraries that are already installed, so you can use it to keep data files synchronized across multiple installations. We do this internally with a test and deployment Galaxy, and find it useful to have the exact same genome builds on all of our machines for running off-Galaxy alignments. Happy to hear thoughts and suggestions. My long term goal is to get this to the point that it could be used alongside existing Galaxy egg scrambling automation to automate deploying Galaxy on a bare bones machine, like Amazon EC2 instances. Hope others find this useful, Brad

1 0

sharing histories with a group
by Edward Kirton 09 Apr '10

09 Apr '10

Hello, I'd like to be able to share histories with an entire group at a time, rather than by user. I see this issue has been submitted as a ticket here: http://bitbucket.org/galaxy/galaxy-central/issue/45/data-sharing However, I only see a button to "Share with a user" in the "Share or Publish" history menu. Is a "Share with a group" button planned? Thanks, Ed

2 1

cleanup of old userless histories
by Matthew Eldridge 09 Apr '10

09 Apr '10

Hi, I noticed that the update timestamp for history objects gets "updated" when these are deleted or purged using the cleanup_datasets.py script. When userless histories older than 30 days, for example, are deleted with this script, they cannot then be purged immediately (along with associated datasets) unless the --days 0 option is used. However, I think this would also purge histories that have been recently deleted through the web interface by a user. It seems that to prevent that happening I'd need to wait another 30 days before really freeing up disk space associated with files from the old userless histories. Perhaps I've misunderstood something about the way is supposed to work? Thanks Matt

1 0

removing the "view in GeneTrack" link
by Hotz, Hans-Rudolf 08 Apr '10

08 Apr '10

Hi I fully appreciate the fact that the visualization module is still in development - although, I am looking very much forward to use it. In the meantime, is there a way to remove the link "view in GeneTrack" you get for all the BED files in your history? I am running changeset 3545 and I have the following lines commented in my 'universe_wsgi.ini' file: # Uncomment following line below to enable visualization module #enable_tracks = True # Comma separated list of UCSC / gbrowse browsers to use for viewing #ucsc_display_sites = main #gbrowse_display_sites = main # Define your GeneTrack servers in tool-data/shared/genetrack/genetrack_sites.txt #genetrack_display_sites = Nevertheless, I get a link like: http://galaxy-dev.fmi.ch/display_application/d12c5c8ceaef53e4/genetrack_inte rval/genetrack which (if clicked on) results in an Error with the following message: "Exception: A data display parameter is in the error state: genetrack_file" I am running another course next week promoting the use of our local galaxy server, and people will probably stumble across this broken link Thank you very much for your help Hans

3 2

Having Problems with DRMAA_python
by vill 08 Apr '10

08 Apr '10

L.S. I'm having trouble getting Sun Grid Engine and galaxy to work. The steps I did: #hg clone http://www.bx.psu.edu/hg/galaxy galaxy_dist #cd galaxy_dist #sh setup.sh #SGE_ROOT=/usr/local/gridengine python scripts/scramble.py DRMAA_python I'm getting this error when building the DRMAA egg. Can you help me getting this egg build. There is a .egg verion downloadable from : http://code.google.com/p/drmaa-python/downloads/list When placing this one in the eggs dir it does not work. The SGE 6u4 is installed and working correctly. ##################################################################### # SGE_ROOT=/usr/local/gridengine python scripts/scramble.py DRMAA_python fetch_one(): Trying to fetch: http://eggs.g2.bx.psu.edu/new/DRMAA_python/DRMAA_python-0.2.tar.gz fetch_one(): Fetched to: /usr/local/galaxy_dist/scripts/scramble/archives/DRMAA_python-0.2.tar.gz unpack_source(): Unpacked to: /usr/local/galaxy_dist/scripts/scramble/build/py2.6-linux-x86_64-ucs4/DRMAA_python copy_build_script(): Using build script /usr/local/galaxy_dist/scripts/scramble/scripts/DRMAA_python.py run_scramble_script(): Beginning build run_scramble_script(): Executing in /usr/local/galaxy_dist/scripts/scramble/build/py2.6-linux-x86_64-ucs4/DRMAA_python: /usr/bin/python -ES scramble.py --------------------------------------------------------------------------- This script requires setuptools version 0.6c11 to run (even to display help). I will attempt to download it for you (from http://pypi.python.org/packages/2.6/s/setuptools/) but you may need to enable firewall access for this script first. I will start the download in 8 seconds. (Note: if this machine does not have network access, please obtain the file http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg and place it in this directory before rerunning this script.) --------------------------------------------------------------------------- Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg Traceback (most recent call last): File "scramble.py", line 14, in <module> from scramble_lib import * File "../../../lib/scramble_lib.py", line 146, in <module> use_setuptools( download_delay=8, to_dir=os.path.dirname( __file__ ) ) File "/usr/local/galaxy_dist/scripts/scramble/build/py2.6-linux-x86_64-ucs4/DRMAA_python/ez_setup.py", line 119, in use_setuptools return do_download() File "/usr/local/galaxy_dist/scripts/scramble/build/py2.6-linux-x86_64-ucs4/DRMAA_python/ez_setup.py", line 99, in do_download import setuptools; setuptools.bootstrap_install_from = egg File "/usr/local/galaxy_dist/scripts/scramble/lib/setuptools-0.6c11-py2.6.egg/setuptools/__init__.py", line 2, in <module> File "/usr/local/galaxy_dist/scripts/scramble/lib/setuptools-0.6c11-py2.6.egg/setuptools/extension.py", line 2, in <module> File "/usr/local/galaxy_dist/scripts/scramble/lib/setuptools-0.6c11-py2.6.egg/setuptools/dist.py", line 5, in <module> File "/usr/local/galaxy_dist/scripts/scramble/lib/setuptools-0.6c11-py2.6.egg/setuptools/command/install.py", line 2, in <module> File "/usr/lib/python2.6/distutils/command/install.py", line 21, in <module> from site import USER_BASE File "/usr/local/galaxy_dist/scripts/scramble/lib/setuptools-0.6c11-py2.6.egg/site.py", line 73, in <module> # Enable per user site-packages directory File "/usr/local/galaxy_dist/scripts/scramble/lib/setuptools-0.6c11-py2.6.egg/site.py", line 38, in __boot configuration files, foo.pth and bar.pth. Assume foo.pth contains the ImportError: Couldn't find the real 'site' module Traceback (most recent call last): File "scripts/scramble.py", line 44, in <module> egg.scramble() File "/usr/local/galaxy_dist/lib/galaxy/eggs/scramble.py", line 57, in scramble self.run_scramble_script() File "/usr/local/galaxy_dist/lib/galaxy/eggs/scramble.py", line 206, in run_scramble_script raise ScrambleFailure( self, "%s(): Egg build failed for %s %s" % ( sys._getframe().f_code.co_name, self.name, self.version ) ) galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for DRMAA_python 0.2 ##################################################################### Thanks, //Michel

1 0