Some problems setting up a new cloudman instance on AWS
Hi Galaxy Admin list, This is my first post to this list. Galaxy is a great tool and I am really enjoying working with it. Background, I am setting up a private galaxy instance for my company using AWS. The installation is going very smoothly with a few problems listed below. I apologize if these questions have been asked before. I have been through the documentation and searched the list pretty thoroughly and can't find the answers. Thanks in advance for your help. Question 1: Can't install tools in the tool shed I have added this line to the universe_wsgi.ini tool_config_file = tool_conf.xml,shed_tool_conf.xml Which references a file that contains this tag: <toolbox tool_path="../shed_tools"> </toolbox> I have created this directory at the correct level and set permission to 755. But I am still getting a non specific error loading tools. I saw that this was (is?) a known issue, but that it had been resolved in the latest build. What am I missing? The log does not give me any more specific information about the error. Any suggestions? Question 2: Weird log error. Every minute the log records this error: galaxy.web.framework DEBUG 2013-03-07 13:07:42,207 Error: this request returned None from get_history(): http://127.0.0.1:8080/root 127.0.0.1 - - [07/Mar/2013:13:07:42 +0000] "GET /root HTTP/1.1" 200 - "-" "Python-urllib/2.6" Is this a known issue? How can I find more information about what is causing it? Question 3: Correct Settings: My intent is to allow employees to access galaxy remotely without allowing public access. In order to keep the data private, I have set: require_login = True allow_user_creation = False Other than the id_secret setting (which I am currently researching) are there any big security holes I am missing with the default installation? Thanks again for the help. Adam
On Thu, Mar 7, 2013 at 8:21 AM, Adam Cohn <adamcohn@genepeeks.com> wrote:
Hi Galaxy Admin list,
This is my first post to this list. Galaxy is a great tool and I am really enjoying working with it.
Background, I am setting up a private galaxy instance for my company using AWS. The installation is going very smoothly with a few problems listed below. I apologize if these questions have been asked before. I have been through the documentation and searched the list pretty thoroughly and can't find the answers. Thanks in advance for your help.
*Question 1: Can't install tools in the tool shed*
I have added this line to the universe_wsgi.ini
tool_config_file = tool_conf.xml,shed_tool_conf.xml
Which references a file that contains this tag:
<toolbox tool_path="../shed_tools"> </toolbox>
I have created this directory at the correct level and set permission to 755.
But I am still getting a non specific error loading tools. I saw that this was (is?) a known issue, but that it had been resolved in the latest build. What am I missing? The log does not give me any more specific information about the error.
Any suggestions?
Can you give any more information about the error you're seeing? The only step necessary to install tools out of the toolshed should be to uncomment the single line referencing shed_tool_conf.xml from your universe_wsgi.ini. The creation of the shed_tools directory, etc., should all be automatic beyond that point. Additionally, because of where the shed tools are currently installed, you'll also need to 'persist galaxyTools' in the cloud admin interface once you have your instance how you want it. It's worth mentioning that there's a new cloud release pending, currently being tested, that will streamline this process.
*Question 2: Weird log error.* Every minute the log records this error:
galaxy.web.framework DEBUG 2013-03-07 13:07:42,207 Error: this request returned None from get_history(): http://127.0.0.1:8080/root 127.0.0.1 - - [07/Mar/2013:13:07:42 +0000] "GET /root HTTP/1.1" 200 - "-" "Python-urllib/2.6"
Is this a known issue? How can I find more information about what is causing it?
Safe to ignore this. *Question 3: **Correct Settings*: My intent is to allow employees to
access galaxy remotely without allowing public access. In order to keep the data private, I have set:
require_login = True allow_user_creation = False
Other than the id_secret setting (which I am currently researching) are there any big security holes I am missing with the default installation?
I'm interested to see what other people suggest here, but it's a start. For this use case, I'd probably recommend setting up external authentication (instead of require_login using galaxy's auth) integrated with nginx on the cloud instance -- see more information about this here: http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy -Dannon
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> environ, self.app) Module paste.wsgilib:543 in intercept_output
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> return self.application(environ, start_response) Module galaxy.web.framework.base:128 in __call__
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:184 in handle_request
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> body = method(
Dannon, Thanks for the quick response. I tried to install a tool from the galaxy test too shed and received the following information, though I am not sure what will be useful to you. I will definitely keep my eye out for the new release and will check into the authentication method you suggested. URL: http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio n?changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2 Module paste.exceptions.errormiddleware:144 in __call__ trans, **kwargs ) Module galaxy.webapps.tool_shed.controllers.repository:1439 in install_repositories_by_revision
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> ( web.url_for( '/', qualified=True ), ','.join( util.listify( repository_ids ) ), ','.join( util.listify( changeset_revisions ) ) ) ) Module tool_shed.util.shed_util_common:4029 in url_join
<http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revisio... hangeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2#> parts.append( arg.strip( '/' ) ) AttributeError: 'NoneType' object has no attribute 'strip'
CGI Variables CONTENT_LENGTH'0' HTTP_ACCEPT'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING'gzip, deflate' HTTP_ACCEPT_LANGUAGE'en-us' HTTP_CONNECTION'close' HTTP_DNT'1' HTTP_HOST'testtoolshed.g2.bx.psu.edu' HTTP_REFERER'http://testtoolshed.g2.bx.psu.edu/repository/preview_tools_in_c hangeset?repository_id=5483924c34f005f2&changeset_revision=1d2730e6bb39' HTTP_USER_AGENT'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17' HTTP_X_FORWARDED_FOR'76.100.61.154' HTTP_X_FORWARDED_HOST'testtoolshed.g2.bx.psu.edu' ORGINAL_HTTP_HOST'testtoolshed_app' ORGINAL_REMOTE_ADDR'127.0.0.1' PATH_INFO'/repository/install_repositories_by_revision' QUERY_STRING'changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f 2' REMOTE_ADDR'76.100.61.154' REQUEST_METHOD'GET' SERVER_NAME'127.0.0.1' SERVER_PORT'9930' SERVER_PROTOCOL'HTTP/1.0' WSGI Variables application<paste.debug.prints.PrintDebugMiddleware object at 0x6855910> paste.cookies(<SimpleCookie: >, '') paste.expected_exceptions[<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions<paste.httpexceptions.HTTPExceptionHandler object at 0x6843f90> paste.httpserver.thread_pool<paste.httpserver.ThreadPool object at 0x6a0ccd0> paste.parsed_querystring([('changeset_revisions', '1d2730e6bb39'), ('repository_ids', '5483924c34f005f2')], 'changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2') paste.printdebug_listeners[<cStringIO.StringO object at 0x87178f0>, <galaxy.util.pastescript.serve.LazyWriter object at 0x31694d0>] paste.recursive.forward<paste.recursive.Forwarder from /> paste.recursive.include<paste.recursive.Includer from /> paste.recursive.include_app_iter<paste.recursive.IncluderAppIter from /> paste.recursive.script_name'' paste.remove_printdebug<function remove_printdebug at 0x5868758> paste.throw_errorsTrue webob._parsed_query_vars(MultiDict([('changeset_revisions', '1d2730e6bb39'), ('repository_ids', '5483924c34f005f2')]), 'changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2') wsgi process'Multithreaded' From: Dannon Baker <dannon.baker@gmail.com> Date: Thursday, March 7, 2013 9:03 AM To: Adam Cohn <adamcohn@genepeeks.com> Cc: <galaxy-dev@lists.bx.psu.edu> Subject: Re: [galaxy-dev] Some problems setting up a new cloudman instance on AWS On Thu, Mar 7, 2013 at 8:21 AM, Adam Cohn <adamcohn@genepeeks.com> wrote:
Hi Galaxy Admin list,
This is my first post to this list. Galaxy is a great tool and I am really enjoying working with it.
Background, I am setting up a private galaxy instance for my company using AWS. The installation is going very smoothly with a few problems listed below. I apologize if these questions have been asked before. I have been through the documentation and searched the list pretty thoroughly and can't find the answers. Thanks in advance for your help.
Question 1: Can't install tools in the tool shed
I have added this line to the universe_wsgi.ini
tool_config_file = tool_conf.xml,shed_tool_conf.xml
Which references a file that contains this tag:
<toolbox tool_path="../shed_tools"> </toolbox>
I have created this directory at the correct level and set permission to 755.
But I am still getting a non specific error loading tools. I saw that this was (is?) a known issue, but that it had been resolved in the latest build. What am I missing? The log does not give me any more specific information about the error.
Any suggestions?
Can you give any more information about the error you're seeing? The only step necessary to install tools out of the toolshed should be to uncomment the single line referencing shed_tool_conf.xml from your universe_wsgi.ini. The creation of the shed_tools directory, etc., should all be automatic beyond that point. Additionally, because of where the shed tools are currently installed, you'll also need to 'persist galaxyTools' in the cloud admin interface once you have your instance how you want it. It's worth mentioning that there's a new cloud release pending, currently being tested, that will streamline this process.
Question 2: Weird log error. Every minute the log records this error: galaxy.web.framework DEBUG 2013-03-07 13:07:42,207 Error: this request returned None from get_history(): http://127.0.0.1:8080/root 127.0.0.1 - - [07/Mar/2013:13:07:42 +0000] "GET /root HTTP/1.1" 200 - "-" "Python-urllib/2.6" Is this a known issue? How can I find more information about what is causing it?
Safe to ignore this.
Question 3: Correct Settings: My intent is to allow employees to access galaxy remotely without allowing public access. In order to keep the data private, I have set:
require_login = True allow_user_creation = False
Other than the id_secret setting (which I am currently researching) are there any big security holes I am missing with the default installation?
I'm interested to see what other people suggest here, but it's a start. For this use case, I'd probably recommend setting up external authentication (instead of require_login using galaxy's auth) integrated with nginx on the cloud instance -- see more information about this here: http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy -Dannon
This is probably due to the version of your cloud galaxy installation being slightly older than the currently deployed tool shed version. If you use the cloud admin panel to update to a more recent galaxy-dist (or just do it yourself after ssh'ing in), this should be resolved. -Dannon On Thu, Mar 7, 2013 at 10:31 AM, Adam Cohn <adamcohn@genepeeks.com> wrote:
Dannon, Thanks for the quick response.
I tried to install a tool from the galaxy test too shed and received the following information, though I am not sure what will be useful to you. I will definitely keep my eye out for the new release and will check into the authentication method you suggested.
URL: http://testtoolshed.g2.bx.psu.edu/repository/install_repositories_by_revision?changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2 Module paste.exceptions.errormiddleware:144 in __call__
app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ environ, self.app) Module paste.wsgilib:543 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:128 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:184 in handle_request body = method( trans, **kwargs ) Module galaxy.webapps.tool_shed.controllers.repository:1439 in install_repositories_by_revision ( web.url_for( '/', qualified=True ), ','.join( util.listify( repository_ids ) ), ','.join( util.listify( changeset_revisions ) ) ) ) Module tool_shed.util.shed_util_common:4029 in url_join parts.append( arg.strip( '/' ) ) AttributeError: 'NoneType' object has no attribute 'strip'
CGI Variables CONTENT_LENGTH '0' HTTP_ACCEPT'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING'gzip, deflate' HTTP_ACCEPT_LANGUAGE'en-us' HTTP_CONNECTION'close' HTTP_DNT'1' HTTP_HOST'testtoolshed.g2.bx.psu.edu' HTTP_REFERER'http://testtoolshed.g2.bx.psu.edu/repository/preview_tools_in_changeset?repository_id=5483924c34f005f2&changeset_revision=1d2730e6bb39' HTTP_USER_AGENT'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17' HTTP_X_FORWARDED_FOR'76.100.61.154' HTTP_X_FORWARDED_HOST'testtoolshed.g2.bx.psu.edu' ORGINAL_HTTP_HOST'testtoolshed_app' ORGINAL_REMOTE_ADDR '127.0.0.1' PATH_INFO '/repository/install_repositories_by_revision' QUERY_STRING'changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2' REMOTE_ADDR'76.100.61.154' REQUEST_METHOD'GET' SERVER_NAME '127.0.0.1' SERVER_PORT'9930' SERVER_PROTOCOL'HTTP/1.0'
WSGI Variables application<paste.debug.prints.PrintDebugMiddleware object at 0x6855910> paste.cookies(<SimpleCookie: >, '') paste.expected_exceptions [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions <paste.httpexceptions.HTTPExceptionHandler object at 0x6843f90> paste.httpserver.thread_pool<paste.httpserver.ThreadPool object at 0x6a0ccd0> paste.parsed_querystring([('changeset_revisions', '1d2730e6bb39'), ('repository_ids', '5483924c34f005f2')], 'changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2') paste.printdebug_listeners[<cStringIO.StringO object at 0x87178f0>, <galaxy.util.pastescript.serve.LazyWriter object at 0x31694d0>] paste.recursive.forward<paste.recursive.Forwarder from /> paste.recursive.include<paste.recursive.Includer from /> paste.recursive.include_app_iter<paste.recursive.IncluderAppIter from /> paste.recursive.script_name'' paste.remove_printdebug <function remove_printdebug at 0x5868758> paste.throw_errors True webob._parsed_query_vars(MultiDict([('changeset_revisions', '1d2730e6bb39'), ('repository_ids', '5483924c34f005f2')]), 'changeset_revisions=1d2730e6bb39&repository_ids=5483924c34f005f2') wsgi process'Multithreaded'
From: Dannon Baker <dannon.baker@gmail.com> Date: Thursday, March 7, 2013 9:03 AM To: Adam Cohn <adamcohn@genepeeks.com> Cc: <galaxy-dev@lists.bx.psu.edu> Subject: Re: [galaxy-dev] Some problems setting up a new cloudman instance on AWS
On Thu, Mar 7, 2013 at 8:21 AM, Adam Cohn <adamcohn@genepeeks.com> wrote:
Hi Galaxy Admin list,
This is my first post to this list. Galaxy is a great tool and I am really enjoying working with it.
Background, I am setting up a private galaxy instance for my company using AWS. The installation is going very smoothly with a few problems listed below. I apologize if these questions have been asked before. I have been through the documentation and searched the list pretty thoroughly and can't find the answers. Thanks in advance for your help.
Question 1: Can't install tools in the tool shed
I have added this line to the universe_wsgi.ini
tool_config_file = tool_conf.xml,shed_tool_conf.xml
Which references a file that contains this tag:
<toolbox tool_path="../shed_tools"> </toolbox>
I have created this directory at the correct level and set permission to 755.
But I am still getting a non specific error loading tools. I saw that this was (is?) a known issue, but that it had been resolved in the latest build. What am I missing? The log does not give me any more specific information about the error.
Any suggestions?
Can you give any more information about the error you're seeing? The only step necessary to install tools out of the toolshed should be to uncomment the single line referencing shed_tool_conf.xml from your universe_wsgi.ini. The creation of the shed_tools directory, etc., should all be automatic beyond that point.
Additionally, because of where the shed tools are currently installed, you'll also need to 'persist galaxyTools' in the cloud admin interface once you have your instance how you want it.
It's worth mentioning that there's a new cloud release pending, currently being tested, that will streamline this process.
Question 2: Weird log error. Every minute the log records this error:
galaxy.web.framework DEBUG 2013-03-07 13:07:42,207 Error: this request returned None from get_history(): http://127.0.0.1:8080/root 127.0.0.1 - - [07/Mar/2013:13:07:42 +0000] "GET /root HTTP/1.1" 200 - "-" "Python-urllib/2.6"
Is this a known issue? How can I find more information about what is causing it?
Safe to ignore this.
Question 3: Correct Settings: My intent is to allow employees to access galaxy remotely without allowing public access. In order to keep the data private, I have set:
require_login = True allow_user_creation = False
Other than the id_secret setting (which I am currently researching) are there any big security holes I am missing with the default installation?
I'm interested to see what other people suggest here, but it's a start. For this use case, I'd probably recommend setting up external authentication (instead of require_login using galaxy's auth) integrated with nginx on the cloud instance -- see more information about this here: http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy
-Dannon
Dannon, Thanks again for the response. I tried this and it was not successful. (same error) I am pretty much running CloudMan out of the box, plus one update, plus a few minor config file changes (already described in my first post.) The only thing I did that was different from the documentation is when I initiated SSH, it asked me if I wanted to configure NX client. I said yes and this process executed successfully. I am rapidly reaching the end of my time allocation to spend on this. Do you know of anyone who would be willing to do a couple of hours of paid work to setup and configure a cloud based Galaxy for our use? Thanks, Adam On 3/8/13 5:41 PM, "Dannon Baker" <dannon.baker@gmail.com> wrote:
This is probably due to the version of your cloud galaxy installation being slightly older than the currently deployed tool shed version. If you use the cloud admin panel to update to a more recent galaxy-dist (or just do it yourself after ssh'ing in), this should be resolved.
-Dannon
If anyone is following this thread, it turned out to be an error specific to Safari Web browser. The tool shed functionality works fine on Firefox. It has been identified as an issue and reproduced by the Galaxy team. Thanks Dannon for going above and beyond in tracking this down! Adam
participants (2)
-
Adam Cohn -
Dannon Baker