- galaxy-dev - lists.galaxyproject.org

Odd (possible latency/race?) problems post-upgrade
by Fields, Christopher J 08 Aug '12

08 Aug '12

I'm seeing a possible latency issue or race condition when starting Galaxy after the latest hg upgrade (July 20) from galaxy-dist; the prior upgrade doesn't have this problem. We have small setup with one job manager/runner and two web front-ends for testing load balancing: …from universe_wsgi.ini: ------------------------------ [server:web0] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:web1] use = egg:Paste#http port = 8081 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 ------------------------------ If I run: GALAXY_RUN_ALL=1 sh run.sh --daemon I will intermittently see the following in the paster log for any of the above services (example below is web1, but I have seen this for manager and web0 as well). The traceback and error is the same in all cases ('File exists: /home/a-m/galaxy/dist-database/tmp/work_tmp'): ------------------------------ galaxy.tool_shed.tool_shed_registry DEBUG 2012-07-30 11:40:10,194 Loading references to tool sheds from tool_sheds_conf.xml galaxy.tool_shed.tool_shed_registry DEBUG 2012-07-30 11:40:10,194 Loaded reference to tool shed: Galaxy main tool shed galaxy.tool_shed.tool_shed_registry DEBUG 2012-07-30 11:40:10,194 Loaded reference to tool shed: Galaxy test tool shed galaxy.model.migrate.check DEBUG 2012-07-30 11:40:10,650 psycopg2 egg successfully loaded for postgres dialect galaxy.model.migrate.check INFO 2012-07-30 11:40:10,845 At database version 103 galaxy.tool_shed.migrate.check DEBUG 2012-07-30 11:40:10,940 psycopg2 egg successfully loaded for postgres dialect galaxy.tool_shed.migrate.check INFO 2012-07-30 11:40:10,986 At migrate_tools version 3 galaxy.model.custom_types DEBUG 2012-07-30 11:40:10,994 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/app.py", line 66, in __init__ self.installed_repository_manager.load_proprietary_datatypes() File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/tool_shed/__init__.py", line 47, in load_proprietary_datatypes installed_repository_dict = galaxy.util.shed_util.load_installed_datatypes( self.app, tool_shed_repository, relative_install_dir ) File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py", line 1269, in load_installed_datatypes work_dir = make_tmp_directory() File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py", line 1305, in make_tmp_directory os.makedirs( work_dir ) File "/usr/lib64/python2.6/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 17] File exists: '/home/a-m/galaxy/dist-database/tmp/work_tmp' Removing PID file web1.pid ------------------------------ I was also seeing this using separate runner/webapp ini files and 'run_multiple_processes.sh --daemon', but we decided to go ahead and migrate over to a unified universe_wsgi.ini file. Anyway, we found a workaround by rerunning 'GALAXY_RUN_ALL=1 sh run.sh --daemon' which skips any running services, but I'm curious whether anyone else has seen this and whether there is a fix (or maybe added config setting we are missing?) chris

3 3

Error on database_connection statement in universe_wsgi.ini file
by Kenneth R. Auerbach 08 Aug '12

08 Aug '12

Hello, Galaxy runs fine 'out of the box' with the default universe_wsgi.ini initialization file. But when I change the "database_connection" statement to connect to my MySql database, I get the errors below. Can you please tell what the problem is? Thanks! -Wolf. Modification to universe_wsgi.ini file: ------------------------------------ #database_connection = sqlite:///./database/universe.sqlite?isolation_level=IMMEDIATE database_connection = mysql:///galaxy?unix_socket=/var/run/mysqld/mysqld.sock ------------------------------------- Then when I type "sh run.sh" I get this: --------- wolf@linuxmint ~/Downloads/galaxy-galaxy-dist-ec29ce8e27a1 $ sh run.sh Traceback (most recent call last): File "./scripts/paster.py", line 34, in <module> command.run() File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run result = self.command() File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command relative_to=base, global_conf=vars) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp **kw) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 224, in loadobj global_conf=global_conf) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 248, in loadcontext global_conf=global_conf) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 278, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 413, in get_context section) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 458, in _context_from_explicit value = import_string(found_expr) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 18, in import_string return pkg_resources.EntryPoint.parse("x="+s).load(False) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/lib/pkg_resources.py", line 1997, in parse src ValueError: ("EntryPoint must be in 'name=module:attrs [extras]' format", 'x=galaxy.web.buildapp:app_factory\ndatabase_connection = mysql:///galaxy?unix_socket=/var/run/mysqld/mysqld.sock') ----------------------------

2 1

How to block anonymous access and disallow user registration‏
by Jennifer Jackson 08 Aug '12

08 Aug '12

Re-Post to galaxy-dev On 8/6/12 11:59 AM, Joel Rosenberg wrote: > Hello, > > I've just set up my own instance of Galaxy running in AWS and had a > security question that I couldn't find an answer to on the wiki. > > I'd like to prevent people from hitting my public Amazon IP and using > Galaxy. Is t! here a way to prevent anonymous users from accessing any > part of Galaxy except a login prompt and have user registration be > invite-only (or perhaps turn it off completely once a handful of > accounts have been created)? > > Thanks, > > -Joel > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > -- Jennifer Jackson http://galaxyproject.org

2 1

Workflows do not open anymore
by Björn Grüning 08 Aug '12

08 Aug '12

Hello, after commit 29680fa5c35e we are not able to open workflows anymore. The error message is: TypeError: <mako.template.Template object at 0x6dc8e10> is not JSON serializable And the problematic changeset is here: https://bitbucket.org/galaxy/galaxy-central/changeset/29680fa5c35e6c22a6b72… Hope that helps, Bjoern -- Björn Grüning Albert-Ludwigs-Universität Freiburg Institute of Pharmaceutical Sciences Pharmaceutical Bioinformatics Hermann-Herder-Strasse 9 D-79104 Freiburg i. Br. Tel.: +49 761 203-4872 Fax.: +49 761 203-97769 E-Mail: bjoern.gruening(a)pharmazie.uni-freiburg.de Web: http://www.pharmaceutical-bioinformatics.org/

2 1

Changing Install Directory
by SHAUN WEBB 08 Aug '12

08 Aug '12

Hi all, I would like to change the name of the directory where my Galaxy instance is installed. What is the best way to go about this without breaking any paths/links. Thanks Shaun -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

Galaxy and Toolshed issues
by SHAUN WEBB 08 Aug '12

08 Aug '12

Hi, I have been trying out the latest version of Galaxy and a newly installed local toolshed and I have a few questions/suggestions/bug reports: 1. Tool Versioning: a) As I understand from the wiki, when I create a new version of a tool in a tool shed I must then install the latest version in my Galaxy instance. This then shows up as two separate tools in my tool menu. Presumably it is best to change the xml description tag to reflect the version number and make tool versions distinguishable in the Galaxy tool menu? b) Would it make more sense to have a single entry for each tool in the tool menu and a drop down to choose which version to run in the tool gui? Similar to when I try to rerun a tool that is no longer installed I get a drop down option to use a derivative of that tool if it exists. d) Say I create a workflow then install a new version of a tool within the workflow. I would like to be able to run that workflow using either version of the tool without creating a new workflow. Is this possible? To me it would make sense to again provide a drop down to select tool versions on the workflow ui just before execution. c) The toolshed interface allows me to change the name of my tool. If I do this, then go to my Galaxy admin ui and try to get updates for my tool then it fails. Does it make sense to change a tools name after it is created? Is it possible to track previous tool names to avoid this happening? e) Email alerts: I could not find an option to receive email alerts about tool updates when installing via the Galaxy admin ui. Perhaps this should be an option during installation. Running toolshed tools: I have no problems installing tools from toolsheds via Galaxy admin interface, however I can't get these tools to run. They stay Grey forever and look like this in the manage jobs table: 65 xxxx(a)xxxx.xx.ac.uk 0 minutes ago toolshed.g2.bx.psu.edu/repos/aaronquinlan/bedtools/bedtools_coveragebed_cou… new None None None Do I have to set a tool runner or some other option for shed_tools? Migrating tools: I upgraded to the latest dist release of galaxy and when I restarted the server I expected to get a message about migrating freebayes tools. I didn't see this message. Will this be because I restart using --daemon? Is it advisable to therefore restart without --daemon after an upgrade to see if there are any tools to be migrated? Genome indexing: 1. I realise this is just a beta feature and it will be great when fully integrated. I tried to generate bowtie2 indexes for an installed build. I can see the job appear below the table in yellow and it contains the message "...added to the job queue to be indexed with bowtie2". However it seems to just sit there, even after restarting. Should I be able to see the job under 'manage jobs'? 2. Where are the generated indexes stored and is the .loc file automatically updated? Workflow menus: Would it be possible to categorise workflows the way we do with tools. I have a long list of workflows and it is sometimes difficult to find the one I need. I realise that workflows are user specific so this is not as easy to implement as with tools. Perhaps it would be possible to have category menus for published workflows at least? Trackster: I have noticed a few bugs with trackster in the latest release (tested on Chrome and Firefox): 1. Overview button does not work 2. If I add tracks to a group and make a composite track, then showindividual tracks again I get new menu options (filters, tool, tool parameter space visualisation). None of these seem to work and trackster often stops working after doing this. Thanks for all the effort put in to these great new features. I hope this lengthy email is helpful! Shaun Webb Edinburgh University -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

4 9

Error on database_connection statement in universe_wsgi.ini file
by Kenneth Auerbach 08 Aug '12

08 Aug '12

Hello, Galaxy runs fine 'out of the box' with the default universe_wsgi.ini initialization file. But when I change the "database_connection" statement to connect to my MySql database, I get the errors below. Can you please tell what the problem is? Thanks! -Wolf. Modification to universe_wsgi.ini file: ------------------------------------ #database_connection = sqlite:///./database/universe.sqlite?isolation_level=IMMEDIATE database_connection = mysql:///galaxy?unix_socket=/var/run/mysqld/mysqld.sock ------------------------------------- Then when I type "sh run.sh" I get this: --------- wolf@linuxmint ~/Downloads/galaxy-galaxy-dist-ec29ce8e27a1 $ sh run.sh Traceback (most recent call last): File "./scripts/paster.py", line 34, in <module> command.run() File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run result = self.command() File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command relative_to=base, global_conf=vars) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp **kw) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 224, in loadobj global_conf=global_conf) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 248, in loadcontext global_conf=global_conf) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 278, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 413, in get_context section) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 458, in _context_from_explicit value = import_string(found_expr) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 18, in import_string return pkg_resources.EntryPoint.parse("x="+s).load(False) File "/home/wolf/Downloads/galaxy-galaxy-dist-ec29ce8e27a1/lib/pkg_resources.py", line 1997, in parse src ValueError: ("EntryPoint must be in 'name=module:attrs [extras]' format", 'x=galaxy.web.buildapp:app_factory\ndatabase_connection = mysql:///galaxy?unix_socket=/var/run/mysqld/mysqld.sock') ----------------------------

3 4

Adding column names to output
by Batsal Devkota 07 Aug '12

07 Aug '12

Hello, I am extracting data from a database and displaying the result in tabular format. I would like to have the header in the output as column names, so that when I do 'join' I know which columns I am referring to. Currently, it only displays as 1, 2, 3. How do I assign column names to the header? Whatever columns I search in the database can be the header in my output. I tried searching around but couldn't find the solution. Thank you. Batsal.

1 0

Galaxy with hadoop
by bluesky8640 07 Aug '12

07 Aug '12

Hi all, As we all known, Galaxy can be configured to run jobs on a cluster with PBS, i'm wondering if Galaxy can run jobs on a Hadoop cluster. Does anyone have done some work to join Galaxy with Hadoop? Thanks, -Jin 2012-08-07 bluesky8640

1 0

Job recovery of non existing job
by Sascha Kastens 07 Aug '12

07 Aug '12

Hi all, I?m running a local instance of Galaxy and every time I start it, Galaxy tries to recover some jobs. These jobs are not running anymore and I get the following information: galaxy.jobs DEBUG 2012-08-07 10:58:08,056 (418) Working directory for job is: /mnt/sec1/bioit/galaxy_working_dir/000/418 galaxy.jobs.handler DEBUG 2012-08-07 10:58:08,089 recovering job 418 in drmaa runner galaxy.jobs.runners.drmaa DEBUG 2012-08-07 10:58:08,090 (418/277560) is still in running state, adding to the DRM queue Is there a way to clean the job handler queue? I tried to start Galaxy with ?enable_job_recovery = False? but that did not solve the problem. Cheers, Sascha

2 2