Fresh galaxy installation tips/requests
Hi all, It's been a long while since I had to install a fresh production galaxy server, and I can offer some tips or requests for minor improvements (but to give credit were it's due - installation is much smoother now, and DRMAA works great with SGE out of the box). These are just minor annoyances, but they can help make new installations easier. --- 1. Debugging REMOTE_USER issues. The mix of apache + authentication + non-root URL + load balancing is a killer. googling "mod_rewrite + mod_auth" shows that I'm not the only one having problems with it... Adding those two lines in "./lib/galaxy/web/framework/middleware/remoteuser.py", line 79 is a wonderful debugging tool, which enables you to see what Apache is passing on to Galaxy: for k,v in environ.items(): sys.stderr.write ( "%s:\t%s\n" % ( k, v ) ) I needed to print all environment variables (not just HTTP_REMOTE_USER) so I could debug mod_rewrite's variables setup. If there was an easy way to enable this debug from the INI file it would be a great help. --- 2. The file name "universe_wsgi.ini" is hard-coded in several places. Most of them are python scripts (like cleanup, ampq, etc.), but one is "./galaxy/eggs/init.py". Not a real problem, except that in the "WebApplicationScaling" wiki page it's kind of implied that "universe_wsgi.ini" is not used any more after creating the two separated ini files (universe_wsgi.runner.ini and universe_wsgi.webapp.ini). If you rename "universe_wsgi.ini" instead of copying it, galaxy will simply not start, even if given a different INI file name. So the best recommendation (in the documents) would probably be to maintain three identical INI files, except for the "[server:main]" part of each (and the "enable_job_running = False" in the runner). Most importantly, make sure that the database connection string is identical in all of them. As a side node, the "cp" command in the "WebApplicationScaling" wiki page is incorrect (you can't copy one file into two files). It'll also be good to mention that the cleanup scripts will always use "universe_wsgi.ini" regardless of what the galaxy server is using. --- 3. Saving SGE/PBS/DRMAA shell scripts if "debug=TRUE" in the jobrunner's INI file - this one is a gem, too bad I discovered it too late. If only it was documented :) --- 4. INI keys that accept multiple comma separated values (e.g. "admin_users") are sensitive to white space. Example: admin_users = gordon@cshl.edu,foobar@cshl.edu works fine, but: admin_users = gordon@cshl.edu , foobar@cshl.edu will not work (spaces around the comma). Same thing with "ucsc_display_sites": ucsc_display_sites = cshl,main will work (and display both "cshl" and "main"), but: ucsc_display_sites = cshl, main will display only "cshl" (because "main" is " main" with a space). It's probably easier to document it than to fix it, but either will do. --- 5. There's no "ensembl_display_sites" in the INI, so disabling the "display at ensembl" link (for BAM files, etc.) requires commenting out the corresponding "<display>" tag in "datatypes_conf.xml", whereas for other servers (gbrowse,ucsc) it's enough to comment out the "ucsc_display_sites" in the INI. --- 6. For galaxy servers running inside an intranet without access to the main/test UCSC Genome browser server, and want to use a local mirror (this might be an obscure setup but that's what I have): disabling both "main" and "test" in "ucsc_display_sites" will disable the authenticated user bypass, because "remouteuser.py" have them hard-coded before setting "self.allow_ucsc_main = True". I needed to patch and add "cshl" as a third option - this is kind of ugly. --- Two development issues not related to production server installation: 1. Debugging complex cheetah templates is very hard. If you have good tricks (beyond #slient and #breakpoint) - examples would be appriciated. If there's a way to see the compiled cheetah code which produced the error - it would be great. One problem is that the error is always reported without a line number (I guess because it's first converted into a string) and without the offending tag. 2. I only half-understand the new "tool_data_table_conf.xml" mechanism - is it explained somewhere ? Looks quite complicated, especially adding in-lined lamba for reading the "__app__" global variable inside the cheetah code of each tool :( --- And user interface quirks: 1. when managing workflows, the "configure your workflow menu" page is a bit uninformative: there's no title, header, instruction or anything else, and the button says "Submit Query" which doesn't mean much ("Update Workflow Menu" would be better). If you click on "configure your workflow menu" and you don't have any workflows, the page is down right mysterious . 2. JQuery Tipsy balloons - please make them go away. Or, make the appearance delay changeable in the INI file (to something like 600 seconds :) ). --- As always, grumbling aside, galaxy is GREAT! so keep up the good work. thanks, -gordon
Hi Assaf, Sorry for the delay in reply. Did you get help/resolve the problems you were having? Or do you still need help? Thanks! Jen Galaxy team On 3/30/11 3:12 PM, Assaf Gordon wrote:
Hi all,
It's been a long while since I had to install a fresh production galaxy server, and I can offer some tips or requests for minor improvements (but to give credit were it's due - installation is much smoother now, and DRMAA works great with SGE out of the box).
These are just minor annoyances, but they can help make new installations easier.
---
1. Debugging REMOTE_USER issues. The mix of apache + authentication + non-root URL + load balancing is a killer. googling "mod_rewrite + mod_auth" shows that I'm not the only one having problems with it...
Adding those two lines in "./lib/galaxy/web/framework/middleware/remoteuser.py", line 79 is a wonderful debugging tool, which enables you to see what Apache is passing on to Galaxy:
for k,v in environ.items(): sys.stderr.write ( "%s:\t%s\n" % ( k, v ) )
I needed to print all environment variables (not just HTTP_REMOTE_USER) so I could debug mod_rewrite's variables setup.
If there was an easy way to enable this debug from the INI file it would be a great help.
---
2. The file name "universe_wsgi.ini" is hard-coded in several places. Most of them are python scripts (like cleanup, ampq, etc.), but one is "./galaxy/eggs/init.py".
Not a real problem, except that in the "WebApplicationScaling" wiki page it's kind of implied that "universe_wsgi.ini" is not used any more after creating the two separated ini files (universe_wsgi.runner.ini and universe_wsgi.webapp.ini). If you rename "universe_wsgi.ini" instead of copying it, galaxy will simply not start, even if given a different INI file name.
So the best recommendation (in the documents) would probably be to maintain three identical INI files, except for the "[server:main]" part of each (and the "enable_job_running = False" in the runner). Most importantly, make sure that the database connection string is identical in all of them.
As a side node, the "cp" command in the "WebApplicationScaling" wiki page is incorrect (you can't copy one file into two files).
It'll also be good to mention that the cleanup scripts will always use "universe_wsgi.ini" regardless of what the galaxy server is using. ---
3. Saving SGE/PBS/DRMAA shell scripts if "debug=TRUE" in the jobrunner's INI file - this one is a gem, too bad I discovered it too late. If only it was documented :)
---
4. INI keys that accept multiple comma separated values (e.g. "admin_users") are sensitive to white space. Example: admin_users = gordon@cshl.edu,foobar@cshl.edu works fine, but: admin_users = gordon@cshl.edu , foobar@cshl.edu will not work (spaces around the comma).
Same thing with "ucsc_display_sites": ucsc_display_sites = cshl,main will work (and display both "cshl" and "main"), but: ucsc_display_sites = cshl, main will display only "cshl" (because "main" is " main" with a space).
It's probably easier to document it than to fix it, but either will do.
---
5. There's no "ensembl_display_sites" in the INI, so disabling the "display at ensembl" link (for BAM files, etc.) requires commenting out the corresponding "<display>" tag in "datatypes_conf.xml", whereas for other servers (gbrowse,ucsc) it's enough to comment out the "ucsc_display_sites" in the INI.
---
6. For galaxy servers running inside an intranet without access to the main/test UCSC Genome browser server, and want to use a local mirror (this might be an obscure setup but that's what I have):
disabling both "main" and "test" in "ucsc_display_sites" will disable the authenticated user bypass, because "remouteuser.py" have them hard-coded before setting "self.allow_ucsc_main = True".
I needed to patch and add "cshl" as a third option - this is kind of ugly.
---
Two development issues not related to production server installation:
1. Debugging complex cheetah templates is very hard. If you have good tricks (beyond #slient and #breakpoint) - examples would be appriciated. If there's a way to see the compiled cheetah code which produced the error - it would be great. One problem is that the error is always reported without a line number (I guess because it's first converted into a string) and without the offending tag.
2. I only half-understand the new "tool_data_table_conf.xml" mechanism - is it explained somewhere ? Looks quite complicated, especially adding in-lined lamba for reading the "__app__" global variable inside the cheetah code of each tool :(
---
And user interface quirks:
1. when managing workflows, the "configure your workflow menu" page is a bit uninformative: there's no title, header, instruction or anything else, and the button says "Submit Query" which doesn't mean much ("Update Workflow Menu" would be better). If you click on "configure your workflow menu" and you don't have any workflows, the page is down right mysterious .
2. JQuery Tipsy balloons - please make them go away. Or, make the appearance delay changeable in the INI file (to something like 600 seconds :) ).
---
As always, grumbling aside, galaxy is GREAT! so keep up the good work.
thanks, -gordon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org
Yes, everything is working fine. Those notes are mostly for possible future improvement ideas. Thanks! Jennifer Jackson wrote, On 05/05/2011 01:52 PM:
Hi Assaf,
Sorry for the delay in reply. Did you get help/resolve the problems you were having? Or do you still need help?
Thanks!
Jen Galaxy team
On 3/30/11 3:12 PM, Assaf Gordon wrote:
Hi all,
It's been a long while since I had to install a fresh production galaxy server, and I can offer some tips or requests for minor improvements (but to give credit were it's due - installation is much smoother now, and DRMAA works great with SGE out of the box).
These are just minor annoyances, but they can help make new installations easier.
---
1. Debugging REMOTE_USER issues. The mix of apache + authentication + non-root URL + load balancing is a killer. googling "mod_rewrite + mod_auth" shows that I'm not the only one having problems with it...
Adding those two lines in "./lib/galaxy/web/framework/middleware/remoteuser.py", line 79 is a wonderful debugging tool, which enables you to see what Apache is passing on to Galaxy:
for k,v in environ.items(): sys.stderr.write ( "%s:\t%s\n" % ( k, v ) )
I needed to print all environment variables (not just HTTP_REMOTE_USER) so I could debug mod_rewrite's variables setup.
If there was an easy way to enable this debug from the INI file it would be a great help.
---
2. The file name "universe_wsgi.ini" is hard-coded in several places. Most of them are python scripts (like cleanup, ampq, etc.), but one is "./galaxy/eggs/init.py".
Not a real problem, except that in the "WebApplicationScaling" wiki page it's kind of implied that "universe_wsgi.ini" is not used any more after creating the two separated ini files (universe_wsgi.runner.ini and universe_wsgi.webapp.ini). If you rename "universe_wsgi.ini" instead of copying it, galaxy will simply not start, even if given a different INI file name.
So the best recommendation (in the documents) would probably be to maintain three identical INI files, except for the "[server:main]" part of each (and the "enable_job_running = False" in the runner). Most importantly, make sure that the database connection string is identical in all of them.
As a side node, the "cp" command in the "WebApplicationScaling" wiki page is incorrect (you can't copy one file into two files).
It'll also be good to mention that the cleanup scripts will always use "universe_wsgi.ini" regardless of what the galaxy server is using. ---
3. Saving SGE/PBS/DRMAA shell scripts if "debug=TRUE" in the jobrunner's INI file - this one is a gem, too bad I discovered it too late. If only it was documented :)
---
4. INI keys that accept multiple comma separated values (e.g. "admin_users") are sensitive to white space. Example: admin_users = gordon@cshl.edu,foobar@cshl.edu works fine, but: admin_users = gordon@cshl.edu , foobar@cshl.edu will not work (spaces around the comma).
Same thing with "ucsc_display_sites": ucsc_display_sites = cshl,main will work (and display both "cshl" and "main"), but: ucsc_display_sites = cshl, main will display only "cshl" (because "main" is " main" with a space).
It's probably easier to document it than to fix it, but either will do.
---
5. There's no "ensembl_display_sites" in the INI, so disabling the "display at ensembl" link (for BAM files, etc.) requires commenting out the corresponding "<display>" tag in "datatypes_conf.xml", whereas for other servers (gbrowse,ucsc) it's enough to comment out the "ucsc_display_sites" in the INI.
---
6. For galaxy servers running inside an intranet without access to the main/test UCSC Genome browser server, and want to use a local mirror (this might be an obscure setup but that's what I have):
disabling both "main" and "test" in "ucsc_display_sites" will disable the authenticated user bypass, because "remouteuser.py" have them hard-coded before setting "self.allow_ucsc_main = True".
I needed to patch and add "cshl" as a third option - this is kind of ugly.
---
Two development issues not related to production server installation:
1. Debugging complex cheetah templates is very hard. If you have good tricks (beyond #slient and #breakpoint) - examples would be appriciated. If there's a way to see the compiled cheetah code which produced the error - it would be great. One problem is that the error is always reported without a line number (I guess because it's first converted into a string) and without the offending tag.
2. I only half-understand the new "tool_data_table_conf.xml" mechanism - is it explained somewhere ? Looks quite complicated, especially adding in-lined lamba for reading the "__app__" global variable inside the cheetah code of each tool :(
---
And user interface quirks:
1. when managing workflows, the "configure your workflow menu" page is a bit uninformative: there's no title, header, instruction or anything else, and the button says "Submit Query" which doesn't mean much ("Update Workflow Menu" would be better). If you click on "configure your workflow menu" and you don't have any workflows, the page is down right mysterious .
2. JQuery Tipsy balloons - please make them go away. Or, make the appearance delay changeable in the INI file (to something like 600 seconds :) ).
---
As always, grumbling aside, galaxy is GREAT! so keep up the good work.
thanks, -gordon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (2)
-
Assaf Gordon -
Jennifer Jackson