Hi Colleen, Stopping instances is not currently supported. You must terminate the cluster and do so through the CloudMan web UI. If you simply want to restart an instance, there is that option available on the CloudMan's Admin page. Terminating an instance is the cheaper version anyhow because once an instance is terminated, you're not having to pay for the root EBS volume (which you do if you were to 'stop' an instance). Once you terminate a cluster and want to bring it back up, simply use the same user data you used when first starting/creating it. For example: cluster_name: the same cluster name as used originally password: any_password access_key: <YOUR AWS ACCESS KEY> secret_key: <YOUR AWS SECRET KEY> Let us know if you still have issues, Enis On Mon, Oct 3, 2011 at 11:43 PM, Colleen Doherty <cdoherty@ucsd.edu> wrote:

Hello, When I stop and restart an instance of galaxy I am not able to bring it back up. I believe I have the exact same error described here:

http://gmod.827538.n3.nabble.com/cloud-instance-missing-opt-sge-default-comm... Basically I do not have the following files, therefore sge won't start up, so no galaxy. -bash: /opt/sge/default/common/settings.sh: No such file or directory -bash: /opt/sge/default/common/settings.sh: No such file or directory

1. Is it just not advisable to stop and restart? Should I just start a fresh instance every time? (Is that any difference in cost?) 2. If stopping and restarting are OK- Are there things I should do before shutting down that would prevent this? I did find sge.py from galaxy cloudman (which looks like something I should have done) 3. If I need to start a new instance, will someone provide an example of the line for userdata that allows me to link to my current bucket?

Thanks for any advice (and apologies if this should be in galaxy-users).

output from /mnt/cm/paster.log is below- but I did try to mess around with this a bit (restarting, shutting things down before copying this, so it may not be very diagnostic, sorry).

Thank you, Colleen

[ERROR] misc:514 2011-10-03 20:55:37,222: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 20:55:37,223: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 20:55:37,223: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 20:55:38,059: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 20:55:39,657: Configuring SGE... [DEBUG] sge:104 2011-10-03 20:55:39,658: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 20:55:39,659: Setting up SGE. [ERROR] misc:514 2011-10-03 20:55:39,769: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] galaxy:139 2011-10-03 20:55:47,615: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:55:47,616: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:04,183: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:04,184: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:20,481: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:20,481: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:36,769: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:36,770: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:53,039: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:53,039: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:09,346: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:09,346: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:25,641: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:25,642: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:41,944: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:41,944: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:58,246: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:58,247: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:14,523: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:14,523: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:30,819: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:30,820: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:47,089: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:47,089: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:03,385: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:03,386: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:19,713: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:19,713: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:35,996: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:35,996: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:52,264: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:52,265: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:08,545: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:08,545: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:24,824: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:24,825: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:41,116: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:41,117: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:57,398: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:57,398: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:13,676: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:13,677: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:29,993: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:29,994: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:46,312: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:46,313: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:02,642: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:02,643: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:18,934: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:18,934: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [INFO] sge:28 2011-10-03 21:02:19,073: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:19,079: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 21:02:19,080: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:02:19,080: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:02:19,779: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:02:[ERROR] misc:514 2011-10-03 20:55:37,222: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 20:55:37,223: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 20:55:37,223: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 20:55:38,059: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 20:55:39,657: Configuring SGE... [DEBUG] sge:104 2011-10-03 20:55:39,658: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 20:55:39,659: Setting up SGE. [ERROR] misc:514 2011-10-03 20:55:39,769: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] galaxy:139 2011-10-03 20:55:47,615: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:55:47,616: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:04,183: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:04,184: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:20,481: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:20,481: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:36,769: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:36,770: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:53,039: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:53,039: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:09,346: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:09,346: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:25,641: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:25,642: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:41,944: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:41,944: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:58,246: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:58,247: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:14,523: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:14,523: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:30,819: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:30,820: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:47,089: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:47,089: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:03,385: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:03,386: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:19,713: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:19,713: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:35,996: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:35,996: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:52,264: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:52,265: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:08,545: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:08,545: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:24,824: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:24,825: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:41,116: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:41,117: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:57,398: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:57,398: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:13,676: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:13,677: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:29,993: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:29,994: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:46,312: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:46,313: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:02,642: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:02,643: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:18,934: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:18,934: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [INFO] sge:28 2011-10-03 21:02:19,073: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:19,079: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 21:02:19,080: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:02:19,080: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:02:19,779: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:02:21,313: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:02:21,314: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:02:21,314: Setting up SGE. [ERROR] misc:514 2011-10-03 21:02:21,424: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] root:354 2011-10-03 21:02:30,686: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:02:30,687: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:30,692: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [INFO] sge:28 2011-10-03 21:02:31,978: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:31,983: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 21:02:31,984: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:02:31,984: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:02:32,216: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:02:33,718: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:02:33,719: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:02:33,719: Setting up SGE. [ERROR] misc:514 2011-10-03 21:02:33,830: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] galaxy:139 2011-10-03 21:02:35,204: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:35,205: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:51,478: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:51,478: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:07,771: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:07,772: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:24,109: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:24,109: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:40,384: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:40,384: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:56,651: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:56,651: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:12,920: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:12,921: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:29,215: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:29,215: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:45,524: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:45,524: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:01,832: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:01,832: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:18,100: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:18,101: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:34,374: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:34,374: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:50,636: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:50,637: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:06,984: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:06,985: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:23,253: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:23,254: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:39,536: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:39,537: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:55,787: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:55,788: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] root:354 2011-10-03 21:07:09,086: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:07:09,086: Removing SGE service [ERROR] misc:514 2011-10-03 21:07:09,092: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] root:354 2011-10-03 21:07:10,053: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:07:10,053: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:07:10,053: Shutting down Galaxy... [DEBUG] misc:511 2011-10-03 21:07:11,409: Successfully stopped Galaxy. [DEBUG] master:1510 2011-10-03 21:07:12,062: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Shut down; [DEBUG] root:354 2011-10-03 21:07:13,247: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:07:13,247: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:07:13,247: Shutting down Galaxy... [ERROR] misc:514 2011-10-03 21:07:13,594: Error stopping Galaxy, running command '/bin/su - galaxy -c "sh $GALAXY_HOME/run.sh --stop-daemon"' returned code '1' and following stderr: '' [DEBUG] root:354 2011-10-03 21:07:19,796: Managing services: [<cm.services.apps.postgres.PostgresService object at 0x2af8290>] [INFO] postgres:24 2011-10-03 21:07:19,797: Removing 'Postgres' service [DEBUG] misc:511 2011-10-03 21:07:19,863: Successfully set ownership of Postgres data directory [INFO] postgres:102 2011-10-03 21:07:19,864: Stopping PostgreSQL... [DEBUG] misc:511 2011-10-03 21:07:20,910: Successfully stopped PostgreSQL [DEBUG] root:354 2011-10-03 21:07:25,447: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [DEBUG] sge:61 2011-10-03 21:07:25,447: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:07:25,448: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:07:25,617: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:07:27,074: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:07:27,075: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:07:27,075: Setting up SGE. [ERROR] misc:514 2011-10-03 21:07:27,186: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] master:1510 2011-10-03 21:07:28,269: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:07:44,467: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:00,668: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:16,861: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:33,066: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] root:354 2011-10-03 21:08:44,630: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:08:44,631: Removing SGE service [ERROR] misc:514 2011-10-03 21:08:44,636: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] master:1510 2011-10-03 21:08:49,320: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:05,516: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:21,725: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:37,940: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:54,203: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:10,422: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:26,625: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:42,830: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:59,055: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:15,257: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:31,577: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:47,765: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:03,974: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:20,166: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:36,399: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:52,608: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:08,812: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:25,006: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:41,208: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:57,398: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:13,734: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:29,943: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:46,152: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:02,378: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:18,599: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:34,849: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:51,043: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:07,329: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:23,517: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:39,712: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:56,062: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:17:12,268: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:17:28,489: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] root:354 2011-10-03 21:17:36,296: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:17:36,296: Removing SGE service [ERROR] misc:514 2011-10-03 21:17:36,302: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] root:354 2011-10-03 21:17:37,230: Managing services: [<cm.services.apps.postgres.PostgresService object at 0x2af8290>] [INFO] postgres:24 2011-10-03 21:17:37,230: Removing 'Postgres' service [DEBUG] misc:511 2011-10-03 21:17:37,289: Successfully set ownership of Postgres data directory [INFO] postgres:102 2011-10-03 21:17:37,290: Stopping PostgreSQL... [ERROR] misc:514 2011-10-03 21:17:37,325: Encountered problem while stopping PostgreSQL, running command '/bin/su - postgres -c "/usr/lib/postgresql/8.4/bin/pg_ctl -w -D /mnt/galaxyData/pgsql/data -o\"-p 5840\" stop"' returned code '1' and following stderr: 'pg_ctl: PID file "/mnt/galaxyData/pgsql/data/postmaster.pid" does not exist Is server running? ' [DEBUG] root:354 2011-10-03 21:17:37,887: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:17:37,888: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:17:37,888: Shutting down Galaxy... [ERROR] misc:514 2011-10-03 21:17:38,244: Error stopping Galaxy, running command '/bin/su - galaxy -c "sh $GALAXY_HOME/run.sh --stop-daemon"' returned code '1' and following stderr: '' [ERROR] postgres:138 2011-10-03 21:17:44,686: PostgreSQL daemon NOT running. [DEBUG] master:1510 2011-10-03 21:17:44,686: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Error; Galaxy..Shutting down; Python version: (2, 6) [DEBUG] app:42 2011-10-03 21:18:47,413: Initializing app [DEBUG] ec2:207 2011-10-03 21:18:47,413: Getting boto S3 connection [DEBUG] ec2:209 2011-10-03 21:18:47,413: No S3 Connection, creating a new one. [DEBUG] ec2:212 2011-10-03 21:18:47,413: Got boto S3 connection. [INFO] misc:378 2011-10-03 21:18:47,495: Retrieved file 'persistent_data.yaml' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to 'pd.yaml'. [INFO] app:52 2011-10-03 21:18:47,501: Master starting [DEBUG] master:50 2011-10-03 21:18:47,604: Initializing console manager - cluster start time: 2011-10-03 21:18:47.604445 [DEBUG] connection:648 2011-10-03 21:18:47,606: Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '1.7.2', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL. See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US'] [DEBUG] connection:494 2011-10-03 21:18:47,646: Open OK! known_hosts [domU-12-31-39-01-61-F2:5672] [DEBUG] channel:70 2011-10-03 21:18:47,646: using channel_id: 1 [DEBUG] channel:484 2011-10-03 21:18:47,647: Channel open [DEBUG] comm:35 2011-10-03 21:18:47,649: Successfully established AMQP connection [DEBUG] master:436 2011-10-03 21:18:47,649: Trying to discover any worker instances associated with this cluster... [DEBUG] ec2:177 2011-10-03 21:18:47,649: Establishing boto EC2 connection [DEBUG] ec2:85 2011-10-03 21:18:47,649: Gathering instance zone, attempt 0 [DEBUG] ec2:90 2011-10-03 21:18:47,652: Instance zone is 'us-east-1a' [DEBUG] ec2:198 2011-10-03 21:18:47,933: Got boto EC2 connection for region 'us-east-1' [DEBUG] ec2:66 2011-10-03 21:18:47,981: Gathering instance id, attempt 0 [DEBUG] ec2:71 2011-10-03 21:18:47,983: Instance ID is 'i-90bed4f0' [DEBUG] ec2:228 2011-10-03 21:18:48,053: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'i-90bed4f0' [DEBUG] ec2:228 2011-10-03 21:18:48,094: Adding tag 'role:master' to resource 'i-90bed4f0' [DEBUG] master:147 2011-10-03 21:18:48,160: ud at manager start: {'access_key': 'AKIAILGV2D5H2XA73EZQ', 'data_filesystems': {'galaxyData': [{'vol_id': 'vol-b53762df', 'size': 600}]}, 'cloudman_home': '/mnt/cm', 'static_filesystems': [{'snap_id': u'snap-5b030634', 'filesystem': 'galaxyIndices', 'size': 700}, {'snap_id': u'snap-1688b978', 'filesystem': 'galaxyTools', 'size': 2}], 'cluster_name': 'CJDGalaxy_Cluster', 'services': [{'service': 'SGE'}, {'service': 'Postgres'}, {'service': 'Galaxy'}], 'bucket_default': 'cloudman', 'boot_script_name': 'cm_boot.py', 'bucket_cluster': 'cm-6d20503431d38fb901ad6a550b90e0d4', 'boot_script_path': '/tmp/cm', 'role': 'master', 'secret_key': 'ARHzUdsBnOVFgT0DE7+X7HXdfQjsZ3m4yFo5Q/S/', 'password': 'oclock'} [DEBUG] master:455 2011-10-03 21:18:48,160: Trying to discover any volumes attached to this instance... [DEBUG] base:55 2011-10-03 21:18:48,165: Enabling 'root' controller, class: CM [DEBUG] buildapp:80 2011-10-03 21:18:48,167: Enabling 'httpexceptions' middleware [DEBUG] buildapp:86 2011-10-03 21:18:48,172: Enabling 'recursive' middleware [DEBUG] buildapp:106 2011-10-03 21:18:48,180: Enabling 'print debug' middleware [DEBUG] buildapp:120 2011-10-03 21:18:48,211: Enabling 'error' middleware [DEBUG] buildapp:130 2011-10-03 21:18:48,211: Enabling 'config' middleware [DEBUG] buildapp:134 2011-10-03 21:18:48,211: Enabling 'x-forwarded-host' middleware Starting server in PID 955. [DEBUG] master:461 2011-10-03 21:18:48,307: Attached volumes: [Volume:vol-73bcf219, Volume:vol-5dbaf437, Volume:vol-ffbaf495, Volume:vol-b53762df, Volume:vol-e77e2c8d, Volume:vol-a37e2cc9] [DEBUG] ec2:237 2011-10-03 21:18:48,307: Getting tag 'cluster_name' on resource 'vol-73bcf219' [DEBUG] ec2:237 2011-10-03 21:18:48,307: Getting tag 'cluster_name' on resource 'vol-5dbaf437' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-ffbaf495' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-b53762df' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-e77e2c8d' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-a37e2cc9' [DEBUG] master:180 2011-10-03 21:18:48,308: Adding static filesystem: 'galaxyIndices' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-73bcf219' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-5dbaf437' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-ffbaf495' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-b53762df' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-e77e2c8d' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-a37e2cc9' [DEBUG] master:180 2011-10-03 21:18:48,310: Adding static filesystem: 'galaxyTools' [DEBUG] master:185 2011-10-03 21:18:48,310: Adding a previously existing data filesystem: 'galaxyData' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'SGE' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'Postgres' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'Galaxy' [INFO] master:155 2011-10-03 21:18:48,310: Completed initial cluster configuration. [DEBUG] master:1488 2011-10-03 21:18:48,310: Monitor started; manager started serving on 0.0.0.0:42284 view at http://127.0.0.1:42284 [DEBUG] master:1515 2011-10-03 21:18:52,314: Monitor adding service 'SGE' [DEBUG] __init__:31 2011-10-03 21:18:52,314: Trying to add service 'SGE' [INFO] __init__:49 2011-10-03 21:18:52,321: Prerequisites OK; starting service 'SGE' [DEBUG] sge:61 2011-10-03 21:18:52,322: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:18:52,352: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:18:53,460: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:18:55,627: Configuring SGE... [DEBUG] ec2:136 2011-10-03 21:18:55,628: Gathering instance private hostname, attempt 0 [DEBUG] sge:104 2011-10-03 21:18:55,630: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:18:55,630: Setting up SGE. [ERROR] misc:514 2011-10-03 21:18:55,848: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] master:1515 2011-10-03 21:18:55,848: Monitor adding service 'FS-galaxyIndices' [DEBUG] filesystem:311 2011-10-03 21:18:55,849: Trying to add service 'FS-galaxyIndices' [DEBUG] filesystem:135 2011-10-03 21:18:55,849: Creating a new volume of size '700' in zone 'us-east-1a' from snapshot 'snap-5b030634' [DEBUG] filesystem:139 2011-10-03 21:19:01,945: Created new volume of size '700' from snapshot 'snap-5b030634' with ID 'vol-0d6b3967' in zone 'us-east-1a' [DEBUG] ec2:228 2011-10-03 21:19:01,946: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-0d6b3967' [DEBUG] ec2:228 2011-10-03 21:19:02,001: Adding tag 'filesystem:galaxyIndices' to resource 'vol-0d6b3967' [DEBUG] filesystem:319 2011-10-03 21:19:02,076: Marked volume 'vol-0d6b3967' from file system 'galaxyIndices' as 'static' [DEBUG] filesystem:129 2011-10-03 21:19:02,130: Running on kernel version 2.6.32-308-ec2; set volume device as '/dev/sdg6' and attach device as '/dev/sdg6' [DEBUG] filesystem:167 2011-10-03 21:19:02,130: Attaching volume 'vol-0d6b3967' to instance 'i-90bed4f0' as device '/dev/sdg6' [DEBUG] filesystem:190 2011-10-03 21:19:02,222: Attaching volume 'vol-0d6b3967'; status: attaching (check 0/30) [DEBUG] filesystem:190 2011-10-03 21:19:04,328: Attaching volume 'vol-0d6b3967'; status: attaching (check 1/30) [DEBUG] filesystem:190 2011-10-03 21:19:06,486: Attaching volume 'vol-0d6b3967'; status: attaching (check 2/30) [DEBUG] filesystem:190 2011-10-03 21:19:08,566: Attaching volume 'vol-0d6b3967'; status: attaching (check 3/30) [DEBUG] filesystem:190 2011-10-03 21:19:10,624: Attaching volume 'vol-0d6b3967'; status: attaching (check 4/30) [DEBUG] filesystem:190 2011-10-03 21:19:12,763: Attaching volume 'vol-0d6b3967'; status: attached (check 5/30) [DEBUG] filesystem:192 2011-10-03 21:19:12,764: Volume 'vol-0d6b3967' attached to instance 'i-90bed4f0' as device '/dev/sdg6' [DEBUG] filesystem:426 2011-10-03 21:19:12,832: Path '/dev/sdg6' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:14,066: Successfully mounted file system '/mnt/galaxyIndices' from '/dev/sdg6' [DEBUG] misc:511 2011-10-03 21:19:14,074: Successfully edited '/mnt/galaxyIndices' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:15,161: As part of filesystem 'Filesystem-galaxyIndices' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:15,237: Monitor adding service 'FS-galaxyTools' [DEBUG] filesystem:311 2011-10-03 21:19:15,237: Trying to add service 'FS-galaxyTools' [DEBUG] filesystem:135 2011-10-03 21:19:15,238: Creating a new volume of size '2' in zone 'us-east-1a' from snapshot 'snap-1688b978' [DEBUG] filesystem:139 2011-10-03 21:19:15,671: Created new volume of size '2' from snapshot 'snap-1688b978' with ID 'vol-c56b39af' in zone 'us-east-1a' [DEBUG] ec2:228 2011-10-03 21:19:15,671: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-c56b39af' [DEBUG] ec2:228 2011-10-03 21:19:15,731: Adding tag 'filesystem:galaxyTools' to resource 'vol-c56b39af' [DEBUG] filesystem:319 2011-10-03 21:19:15,778: Marked volume 'vol-c56b39af' from file system 'galaxyTools' as 'static' [DEBUG] filesystem:129 2011-10-03 21:19:15,828: Running on kernel version 2.6.32-308-ec2; set volume device as '/dev/sdg7' and attach device as '/dev/sdg7' [DEBUG] filesystem:167 2011-10-03 21:19:15,828: Attaching volume 'vol-c56b39af' to instance 'i-90bed4f0' as device '/dev/sdg7' [DEBUG] filesystem:190 2011-10-03 21:19:15,920: Attaching volume 'vol-c56b39af'; status: attaching (check 0/30) [DEBUG] filesystem:190 2011-10-03 21:19:18,051: Attaching volume 'vol-c56b39af'; status: attaching (check 1/30) [DEBUG] filesystem:190 2011-10-03 21:19:20,200: Attaching volume 'vol-c56b39af'; status: attaching (check 2/30) [DEBUG] filesystem:190 2011-10-03 21:19:22,259: Attaching volume 'vol-c56b39af'; status: attaching (check 3/30) [DEBUG] filesystem:190 2011-10-03 21:19:24,308: Attaching volume 'vol-c56b39af'; status: attaching (check 4/30) [DEBUG] filesystem:190 2011-10-03 21:19:26,506: Attaching volume 'vol-c56b39af'; status: attached (check 5/30) [DEBUG] filesystem:192 2011-10-03 21:19:26,507: Volume 'vol-c56b39af' attached to instance 'i-90bed4f0' as device '/dev/sdg7' [DEBUG] filesystem:426 2011-10-03 21:19:26,554: Path '/dev/sdg7' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:27,482: Successfully mounted file system '/mnt/galaxyTools' from '/dev/sdg7' [DEBUG] misc:511 2011-10-03 21:19:27,489: Successfully edited '/mnt/galaxyTools' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:28,726: As part of filesystem 'Filesystem-galaxyTools' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:28,825: Monitor adding service 'FS-galaxyData' [DEBUG] filesystem:311 2011-10-03 21:19:28,826: Trying to add service 'FS-galaxyData' [DEBUG] filesystem:143 2011-10-03 21:19:28,950: Tried to create a volume but it is in state 'attached' (volume ID: vol-b53762df) [DEBUG] ec2:228 2011-10-03 21:19:28,950: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-b53762df' [DEBUG] ec2:228 2011-10-03 21:19:28,995: Adding tag 'filesystem:galaxyData' to resource 'vol-b53762df' [DEBUG] filesystem:214 2011-10-03 21:19:29,164: Tried to attach a volume but the volume 'vol-b53762df' is already attached (as device /dev/sdg3) [DEBUG] filesystem:426 2011-10-03 21:19:29,249: Path '/dev/sdg3' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:29,690: Successfully mounted file system '/mnt/galaxyData' from '/dev/sdg3' [DEBUG] misc:511 2011-10-03 21:19:29,702: Successfully edited '/mnt/galaxyData' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:30,804: As part of filesystem 'Filesystem-galaxyData' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:30,854: Monitor adding service 'Postgres' [DEBUG] __init__:31 2011-10-03 21:19:30,855: Trying to add service 'Postgres' [DEBUG] __init__:36 2011-10-03 21:19:30,855: 'Postgres' service checking its prerequisite 'Filesystem:galaxyData' [INFO] __init__:49 2011-10-03 21:19:30,855: Prerequisites OK; starting service 'Postgres' [DEBUG] misc:511 2011-10-03 21:19:31,057: Successfully set ownership of Postgres data directory [ERROR] postgres:138 2011-10-03 21:19:31,058: PostgreSQL daemon NOT running. [ERROR] postgres:138 2011-10-03 21:19:31,058: PostgreSQL daemon NOT running. [INFO] postgres:90 2011-10-03 21:19:31,058: Starting PostgreSQL... [DEBUG] misc:511 2011-10-03 21:19:31,079: Successfully changed owner of postgres data dir [DEBUG] misc:511 2011-10-03 21:19:32,218: Successfully started PostgreSQL server [INFO] postgres:95 2011-10-03 21:19:32,332: Successfully started PostgreSQL. [DEBUG] master:1515 2011-10-03 21:19:32,332: Monitor adding service 'Galaxy' [DEBUG] __init__:31 2011-10-03 21:19:32,332: Trying to add service 'Galaxy' [DEBUG] __init__:36 2011-10-03 [ERROR] misc:514 2011-10-03 20:55:37,222: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 20:55:37,223: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 20:55:37,223: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 20:55:38,059: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 20:55:39,657: Configuring SGE... [DEBUG] sge:104 2011-10-03 20:55:39,658: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 20:55:39,659: Setting up SGE. [ERROR] misc:514 2011-10-03 20:55:39,769: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] galaxy:139 2011-10-03 20:55:47,615: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:55:47,616: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:04,183: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:04,184: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:20,481: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:20,481: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:36,769: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:36,770: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:56:53,039: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:56:53,039: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:09,346: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:09,346: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:25,641: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:25,642: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:41,944: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:41,944: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:57:58,246: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:57:58,247: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:14,523: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:14,523: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:30,819: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:30,820: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:58:47,089: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:58:47,089: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:03,385: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:03,386: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:19,713: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:19,713: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:35,996: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:35,996: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 20:59:52,264: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 20:59:52,265: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:08,545: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:08,545: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:24,824: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:24,825: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:41,116: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:41,117: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:00:57,398: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:00:57,398: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:13,676: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:13,677: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:29,993: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:29,994: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:01:46,312: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:01:46,313: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:02,642: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:02,643: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:18,934: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:18,934: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [INFO] sge:28 2011-10-03 21:02:19,073: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:19,079: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 21:02:19,080: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:02:19,080: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:02:19,779: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:02:21,313: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:02:21,314: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:02:21,314: Setting up SGE. [ERROR] misc:514 2011-10-03 21:02:21,424: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] root:354 2011-10-03 21:02:30,686: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:02:30,687: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:30,692: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [INFO] sge:28 2011-10-03 21:02:31,978: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:31,983: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 21:02:31,984: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:02:31,984: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:02:32,216: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:02:33,718: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:02:33,719: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:02:33,719: Setting up SGE. [ERROR] misc:514 2011-10-03 21:02:33,830: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] galaxy:139 2011-10-03 21:02:35,204: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:35,205: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:51,478: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:51,478: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:07,771: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:07,772: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:24,109: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:24,109: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:40,384: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:40,384: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:56,651: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:56,651: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:12,920: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:12,921: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:29,215: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:29,215: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:45,524: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:45,524: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:01,832: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:01,832: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:18,100: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:18,101: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:34,374: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:34,374: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:50,636: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:50,637: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:06,984: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:06,985: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:23,253: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:23,254: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:39,536: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:39,537: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:55,787: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:55,788: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] root:354 2011-10-03 21:07:09,086: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:07:09,086: Removing SGE service [ERROR] misc:514 2011-10-03 21:07:09,092: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] root:354 2011-10-03 21:07:10,053: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:07:10,053: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:07:10,053: Shutting down Galaxy... [DEBUG] misc:511 2011-10-03 21:07:11,409: Successfully stopped Galaxy. [DEBUG] master:1510 2011-10-03 21:07:12,062: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Shut down; [DEBUG] root:354 2011-10-03 21:07:13,247: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:07:13,247: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:07:13,247: Shutting down Galaxy... [ERROR] misc:514 2011-10-03 21:07:13,594: Error stopping Galaxy, running command '/bin/su - galaxy -c "sh $GALAXY_HOME/run.sh --stop-daemon"' returned code '1' and following stderr: '' [DEBUG] root:354 2011-10-03 21:07:19,796: Managing services: [<cm.services.apps.postgres.PostgresService object at 0x2af8290>] [INFO] postgres:24 2011-10-03 21:07:19,797: Removing 'Postgres' service [DEBUG] misc:511 2011-10-03 21:07:19,863: Successfully set ownership of Postgres data directory [INFO] postgres:102 2011-10-03 21:07:19,864: Stopping PostgreSQL... [DEBUG] misc:511 2011-10-03 21:07:20,910: Successfully stopped PostgreSQL [DEBUG] root:354 2011-10-03 21:07:25,447: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [DEBUG] sge:61 2011-10-03 21:07:25,447: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:07:25,448: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:07:25,617: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:07:27,074: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:07:27,075: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:07:27,075: Setting up SGE. [ERROR] misc:514 2011-10-03 21:07:27,186: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] master:1510 2011-10-03 21:07:28,269: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:07:44,467: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:00,668: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:16,861: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:33,066: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] root:354 2011-10-03 21:08:44,630: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:08:44,631: Removing SGE service [ERROR] misc:514 2011-10-03 21:08:44,636: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] master:1510 2011-10-03 21:08:49,320: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:05,516: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:21,725: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:37,940: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:54,203: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:10,422: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:26,625: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:42,830: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:59,055: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:15,257: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:31,577: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:47,765: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:03,974: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:20,166: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:36,399: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:52,608: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:08,812: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:25,006: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:41,208: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:57,398: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:13,734: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:29,943: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:46,152: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:02,378: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:18,599: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:34,849: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:51,043: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:07,329: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:23,517: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:39,712: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:56,062: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:17:12,268: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:17:28,489: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] root:354 2011-10-03 21:17:36,296: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:17:36,296: Removing SGE service [ERROR] misc:514 2011-10-03 21:17:36,302: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] root:354 2011-10-03 21:17:37,230: Managing services: [<cm.services.apps.postgres.PostgresService object at 0x2af8290>] [INFO] postgres:24 2011-10-03 21:17:37,230: Removing 'Postgres' service [DEBUG] misc:511 2011-10-03 21:17:37,289: Successfully set ownership of Postgres data directory [INFO] postgres:102 2011-10-03 21:17:37,290: Stopping PostgreSQL... [ERROR] misc:514 2011-10-03 21:17:37,325: Encountered problem while stopping PostgreSQL, running command '/bin/su - postgres -c "/usr/lib/postgresql/8.4/bin/pg_ctl -w -D /mnt/galaxyData/pgsql/data -o\"-p 5840\" stop"' returned code '1' and following stderr: 'pg_ctl: PID file "/mnt/galaxyData/pgsql/data/postmaster.pid" does not exist Is server running? ' [DEBUG] root:354 2011-10-03 21:17:37,887: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:17:37,888: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:17:37,888: Shutting down Galaxy... [ERROR] misc:514 2011-10-03 21:17:38,244: Error stopping Galaxy, running command '/bin/su - galaxy -c "sh $GALAXY_HOME/run.sh --stop-daemon"' returned code '1' and following stderr: '' [ERROR] postgres:138 2011-10-03 21:17:44,686: PostgreSQL daemon NOT running. [DEBUG] master:1510 2011-10-03 21:17:44,686: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Error; Galaxy..Shutting down; Python version: (2, 6) [DEBUG] app:42 2011-10-03 21:18:47,413: Initializing app [DEBUG] ec2:207 2011-10-03 21:18:47,413: Getting boto S3 connection [DEBUG] ec2:209 2011-10-03 21:18:47,413: No S3 Connection, creating a new one. [DEBUG] ec2:212 2011-10-03 21:18:47,413: Got boto S3 connection. [INFO] misc:378 2011-10-03 21:18:47,495: Retrieved file 'persistent_data.yaml' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to 'pd.yaml'. [INFO] app:52 2011-10-03 21:18:47,501: Master starting [DEBUG] master:50 2011-10-03 21:18:47,604: Initializing console manager - cluster start time: 2011-10-03 21:18:47.604445 [DEBUG] connection:648 2011-10-03 21:18:47,606: Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '1.7.2', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL. See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US'] [DEBUG] connection:494 2011-10-03 21:18:47,646: Open OK! known_hosts [domU-12-31-39-01-61-F2:5672] [DEBUG] channel:70 2011-10-03 21:18:47,646: using channel_id: 1 [DEBUG] channel:484 2011-10-03 21:18:47,647: Channel open [DEBUG] comm:35 2011-10-03 21:18:47,649: Successfully established AMQP connection [DEBUG] master:436 2011-10-03 21:18:47,649: Trying to discover any worker instances associated with this cluster... [DEBUG] ec2:177 2011-10-03 21:18:47,649: Establishing boto EC2 connection [DEBUG] ec2:85 2011-10-03 21:18:47,649: Gathering instance zone, attempt 0 [DEBUG] ec2:90 2011-10-03 21:18:47,652: Instance zone is 'us-east-1a' [DEBUG] ec2:198 2011-10-03 21:18:47,933: Got boto EC2 connection for region 'us-east-1' [DEBUG] ec2:66 2011-10-03 21:18:47,981: Gathering instance id, attempt 0 [DEBUG] ec2:71 2011-10-03 21:18:47,983: Instance ID is 'i-90bed4f0' [DEBUG] ec2:228 2011-10-03 21:18:48,053: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'i-90bed4f0' [DEBUG] ec2:228 2011-10-03 21:18:48,094: Adding tag 'role:master' to resource 'i-90bed4f0' [DEBUG] master:147 2011-10-03 21:18:48,160: ud at manager start: {'access_key': 'AKIAILGV2D5H2XA73EZQ', 'data_filesystems': {'galaxyData': [{'vol_id': 'vol-b53762df', 'size': 600}]}, 'cloudman_home': '/mnt/cm', 'static_filesystems': [{'snap_id': u'snap-5b030634', 'filesystem': 'galaxyIndices', 'size': 700}, {'snap_id': u'snap-1688b978', 'filesystem': 'galaxyTools', 'size': 2}], 'cluster_name': 'CJDGalaxy_Cluster', 'services': [{'service': 'SGE'}, {'service': 'Postgres'}, {'service': 'Galaxy'}], 'bucket_default': 'cloudman', 'boot_script_name': 'cm_boot.py', 'bucket_cluster': 'cm-6d20503431d38fb901ad6a550b90e0d4', 'boot_script_path': '/tmp/cm', 'role': 'master', 'secret_key': 'ARHzUdsBnOVFgT0DE7+X7HXdfQjsZ3m4yFo5Q/S/', 'password': 'oclock'} [DEBUG] master:455 2011-10-03 21:18:48,160: Trying to discover any volumes attached to this instance... [DEBUG] base:55 2011-10-03 21:18:48,165: Enabling 'root' controller, class: CM [DEBUG] buildapp:80 2011-10-03 21:18:48,167: Enabling 'httpexceptions' middleware [DEBUG] buildapp:86 2011-10-03 21:18:48,172: Enabling 'recursive' middleware [DEBUG] buildapp:106 2011-10-03 21:18:48,180: Enabling 'print debug' middleware [DEBUG] buildapp:120 2011-10-03 21:18:48,211: Enabling 'error' middleware [DEBUG] buildapp:130 2011-10-03 21:18:48,211: Enabling 'config' middleware [DEBUG] buildapp:134 2011-10-03 21:18:48,211: Enabling 'x-forwarded-host' middleware Starting server in PID 955. [DEBUG] master:461 2011-10-03 21:18:48,307: Attached volumes: [Volume:vol-73bcf219, Volume:vol-5dbaf437, Volume:vol-ffbaf495, Volume:vol-b53762df, Volume:vol-e77e2c8d, Volume:vol-a37e2cc9] [DEBUG] ec2:237 2011-10-03 21:18:48,307: Getting tag 'cluster_name' on resource 'vol-73bcf219' [DEBUG] ec2:237 2011-10-03 21:18:48,307: Getting tag 'cluster_name' on resource 'vol-5dbaf437' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-ffbaf495' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-b53762df' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-e77e2c8d' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-a37e2cc9' [DEBUG] master:180 2011-10-03 21:18:48,308: Adding static filesystem: 'galaxyIndices' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-73bcf219' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-5dbaf437' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-ffbaf495' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-b53762df' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-e77e2c8d' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-a37e2cc9' [DEBUG] master:180 2011-10-03 21:18:48,310: Adding static filesystem: 'galaxyTools' [DEBUG] master:185 2011-10-03 21:18:48,310: Adding a previously existing data filesystem: 'galaxyData' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'SGE' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'Postgres' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'Galaxy' [INFO] master:155 2011-10-03 21:18:48,310: Completed initial cluster configuration. [DEBUG] master:1488 2011-10-03 21:18:48,310: Monitor started; manager started serving on 0.0.0.0:42284 view at http://127.0.0.1:42284 [DEBUG] master:1515 2011-10-03 21:18:52,314: Monitor adding service 'SGE' [DEBUG] __init__:31 2011-10-03 21:18:52,314: Trying to add service 'SGE' [INFO] __init__:49 2011-10-03 21:18:52,321: Prerequisites OK; starting service 'SGE' [DEBUG] sge:61 2011-10-03 21:18:52,322: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:18:52,352: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:18:53,460: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:18:55,627: Configuring SGE... [DEBUG] ec2:136 2011-10-03 21:18:55,628: Gathering instance private hostname, attempt 0 [DEBUG] sge:104 2011-10-03 21:18:55,630: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:18:55,630: Setting up SGE. [ERROR] misc:514 2011-10-03 21:18:55,848: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] master:1515 2011-10-03 21:18:55,848: Monitor adding service 'FS-galaxyIndices' [DEBUG] filesystem:311 2011-10-03 21:18:55,849: Trying to add service 'FS-galaxyIndices' [DEBUG] filesystem:135 2011-10-03 21:18:55,849: Creating a new volume of size '700' in zone 'us-east-1a' from snapshot 'snap-5b030634' [DEBUG] filesystem:139 2011-10-03 21:19:01,945: Created new volume of size '700' from snapshot 'snap-5b030634' with ID 'vol-0d6b3967' in zone 'us-east-1a' [DEBUG] ec2:228 2011-10-03 21:19:01,946: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-0d6b3967' [DEBUG] ec2:228 2011-10-03 21:19:02,001: Adding tag 'filesystem:galaxyIndices' to resource 'vol-0d6b3967' [DEBUG] filesystem:319 2011-10-03 21:19:02,076: Marked volume 'vol-0d6b3967' from file system 'galaxyIndices' as 'static' [DEBUG] filesystem:129 2011-10-03 21:19:02,130: Running on kernel version 2.6.32-308-ec2; set volume device as '/dev/sdg6' and attach device as '/dev/sdg6' [DEBUG] filesystem:167 2011-10-03 21:19:02,130: Attaching volume 'vol-0d6b3967' to instance 'i-90bed4f0' as device '/dev/sdg6' [DEBUG] filesystem:190 2011-10-03 21:19:02,222: Attaching volume 'vol-0d6b3967'; status: attaching (check 0/30) [DEBUG] filesystem:190 2011-10-03 21:19:04,328: Attaching volume 'vol-0d6b3967'; status: attaching (check 1/30) [DEBUG] filesystem:190 2011-10-03 21:19:06,486: Attaching volume 'vol-0d6b3967'; status: attaching (check 2/30) [DEBUG] filesystem:190 2011-10-03 21:19:08,566: Attaching volume 'vol-0d6b3967'; status: attaching (check 3/30) [DEBUG] filesystem:190 2011-10-03 21:19:10,624: Attaching volume 'vol-0d6b3967'; status: attaching (check 4/30) [DEBUG] filesystem:190 2011-10-03 21:19:12,763: Attaching volume 'vol-0d6b3967'; status: attached (check 5/30) [DEBUG] filesystem:192 2011-10-03 21:19:12,764: Volume 'vol-0d6b3967' attached to instance 'i-90bed4f0' as device '/dev/sdg6' [DEBUG] filesystem:426 2011-10-03 21:19:12,832: Path '/dev/sdg6' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:14,066: Successfully mounted file system '/mnt/galaxyIndices' from '/dev/sdg6' [DEBUG] misc:511 2011-10-03 21:19:14,074: Successfully edited '/mnt/galaxyIndices' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:15,161: As part of filesystem 'Filesystem-galaxyIndices' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:15,237: Monitor adding service 'FS-galaxyTools' [DEBUG] filesystem:311 2011-10-03 21:19:15,237: Trying to add service 'FS-galaxyTools' [DEBUG] filesystem:135 2011-10-03 21:19:15,238: Creating a new volume of size '2' in zone 'us-east-1a' from snapshot 'snap-1688b978' [DEBUG] filesystem:139 2011-10-03 21:19:15,671: Created new volume of size '2' from snapshot 'snap-1688b978' with ID 'vol-c56b39af' in zone 'us-east-1a' [DEBUG] ec2:228 2011-10-03 21:19:15,671: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-c56b39af' [DEBUG] ec2:228 2011-10-03 21:19:15,731: Adding tag 'filesystem:galaxyTools' to resource 'vol-c56b39af' [DEBUG] filesystem:319 2011-10-03 21:19:15,778: Marked volume 'vol-c56b39af' from file system 'galaxyTools' as 'static' [DEBUG] filesystem:129 2011-10-03 21:19:15,828: Running on kernel version 2.6.32-308-ec2; set volume device as '/dev/sdg7' and attach device as '/dev/sdg7' [DEBUG] filesystem:167 2011-10-03 21:19:15,828: Attaching volume 'vol-c56b39af' to instance 'i-90bed4f0' as device '/dev/sdg7' [DEBUG] filesystem:190 2011-10-03 21:19:15,920: Attaching volume 'vol-c56b39af'; status: attaching (check 0/30) [DEBUG] filesystem:190 2011-10-03 21:19:18,051: Attaching volume 'vol-c56b39af'; status: attaching (check 1/30) [DEBUG] filesystem:190 2011-10-03 21:19:20,200: Attaching volume 'vol-c56b39af'; status: attaching (check 2/30) [DEBUG] filesystem:190 2011-10-03 21:19:22,259: Attaching volume 'vol-c56b39af'; status: attaching (check 3/30) [DEBUG] filesystem:190 2011-10-03 21:19:24,308: Attaching volume 'vol-c56b39af'; status: attaching (check 4/30) [DEBUG] filesystem:190 2011-10-03 21:19:26,506: Attaching volume 'vol-c56b39af'; status: attached (check 5/30) [DEBUG] filesystem:192 2011-10-03 21:19:26,507: Volume 'vol-c56b39af' attached to instance 'i-90bed4f0' as device '/dev/sdg7' [DEBUG] filesystem:426 2011-10-03 21:19:26,554: Path '/dev/sdg7' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:27,482: Successfully mounted file system '/mnt/galaxyTools' from '/dev/sdg7' [DEBUG] misc:511 2011-10-03 21:19:27,489: Successfully edited '/mnt/galaxyTools' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:28,726: As part of filesystem 'Filesystem-galaxyTools' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:28,825: Monitor adding service 'FS-galaxyData' [DEBUG] filesystem:311 2011-10-03 21:19:28,826: Trying to add service 'FS-galaxyData' [DEBUG] filesystem:143 2011-10-03 21:19:28,950: Tried to create a volume but it is in state 'attached' (volume ID: vol-b53762df) [DEBUG] ec2:228 2011-10-03 21:19:28,950: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-b53762df' [DEBUG] ec2:228 2011-10-03 21:19:28,995: Adding tag 'filesystem:galaxyData' to resource 'vol-b53762df' [DEBUG] filesystem:214 2011-10-03 21:19:29,164: Tried to attach a volume but the volume 'vol-b53762df' is already attached (as device /dev/sdg3) [DEBUG] filesystem:426 2011-10-03 21:19:29,249: Path '/dev/sdg3' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:29,690: Successfully mounted file system '/mnt/galaxyData' from '/dev/sdg3' [DEBUG] misc:511 2011-10-03 21:19:29,702: Successfully edited '/mnt/galaxyData' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:30,804: As part of filesystem 'Filesystem-galaxyData' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:30,854: Monitor adding service 'Postgres' [DEBUG] __init__:31 2011-10-03 21:19:30,855: Trying to add service 'Postgres' [DEBUG] __init__:36 2011-10-03 21:19:30,855: 'Postgres' service checking its prerequisite 'Filesystem:galaxyData' [INFO] __init__:49 2011-10-03 21:19:30,855: Prerequisites OK; starting service 'Postgres' [DEBUG] misc:511 2011-10-03 21:19:31,057: Successfully set ownership of Postgres data directory [ERROR] postgres:138 2011-10-03 21:19:31,058: PostgreSQL daemon NOT running. [ERROR] postgres:138 2011-10-03 21:19:31,058: PostgreSQL daemon NOT running. [INFO] postgres:90 2011-10-03 21:19:31,058: Starting PostgreSQL... [DEBUG] misc:511 2011-10-03 21:19:31,079: Successfully changed owner of postgres data dir [DEBUG] misc:511 2011-10-03 21:19:32,218: Successfully started PostgreSQL server [INFO] postgres:95 2011-10-03 21:19:32,332: Successfully started PostgreSQL. [DEBUG] master:1515 2011-10-03 21:19:32,332: Monitor adding service 'Galaxy' [DEBUG] __init__:31 2011-10-03 21:19:32,332: Trying to add service 'Galaxy' [DEBUG] __init__:36 2011-10-03 21:19:32,333: 'Galaxy' service checking its prerequisite 'Postgres:None' [DEBUG] __init__:36 2011-10-03 21:19:32,333: 'Galaxy' service checking its prerequisite 'Filesystem:galaxyTools' [INFO] __init__:49 2011-10-03 21:19:32,333: Prerequisites OK; starting service 'Galaxy' [INFO] galaxy:50 2011-10-03 21:19:32,333: Setting up Galaxy application [DEBUG] ec2:207 2011-10-03 21:19:32,333: Getting boto S3 connection [INFO] misc:378 2011-10-03 21:19:32,943: Retrieved file 'universe_wsgi.ini.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/universe_wsgi.ini'. [INFO] misc:378 2011-10-03 21:19:34,459: Retrieved file 'tool_conf.xml.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/tool_conf.xml'. [INFO] misc:378 2011-10-03 21:19:34,565: Retrieved file 'tool_data_table_conf.xml.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/tool_data_table_conf.xml.cloud'. [DEBUG] misc:511 2011-10-03 21:19:36,238: Started FTP server [INFO] galaxy:113 2011-10-03 21:19:36,238: Starting Galaxy... [DEBUG] misc:511 2011-10-03 21:19:43,828: Successfully initiated Galaxy start. [DEBUG] galaxy:139 2011-10-03 21:19:43,852: Galaxy UI does not seem to be accessible. [DEBUG] ec2:207 2011-10-03 21:19:43,852: Getting boto S3 connection [INFO] misc:394 2011-10-03 21:19:44,143: Saved file 'persistent_data.yaml' to bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' [DEBUG] master:1462 2011-10-03 21:19:44,195: Instance post start script (/mnt/cm/post_start_script) does not exist? [DEBUG] galaxy:139 2011-10-03 21:19:48,544: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:19:48,545: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:04,821: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:04,822: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:21,089: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:21,089: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:37,369: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:37,370: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; 21:19:32,333: 'Galaxy' service checking its prerequisite 'Postgres:None' [DEBUG] __init__:36 2011-10-03 21:19:32,333: 'Galaxy' service checking its prerequisite 'Filesystem:galaxyTools' [INFO] __init__:49 2011-10-03 21:19:32,333: Prerequisites OK; starting service 'Galaxy' [INFO] galaxy:50 2011-10-03 21:19:32,333: Setting up Galaxy application [DEBUG] ec2:207 2011-10-03 21:19:32,333: Getting boto S3 connection [INFO] misc:378 2011-10-03 21:19:32,943: Retrieved file 'universe_wsgi.ini.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/universe_wsgi.ini'. [INFO] misc:378 2011-10-03 21:19:34,459: Retrieved file 'tool_conf.xml.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/tool_conf.xml'. [INFO] misc:378 2011-10-03 21:19:34,565: Retrieved file 'tool_data_table_conf.xml.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/tool_data_table_conf.xml.cloud'. [DEBUG] misc:511 2011-10-03 21:19:36,238: Started FTP server [INFO] galaxy:113 2011-10-03 21:19:36,238: Starting Galaxy... [DEBUG] misc:511 2011-10-03 21:19:43,828: Successfully initiated Galaxy start. [DEBUG] galaxy:139 2011-10-03 21:19:43,852: Galaxy UI does not seem to be accessible. [DEBUG] ec2:207 2011-10-03 21:19:43,852: Getting boto S3 connection [INFO] misc:394 2011-10-03 21:19:44,143: Saved file 'persistent_data.yaml' to bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' [DEBUG] master:1462 2011-10-03 21:19:44,195: Instance post start script (/mnt/cm/post_start_script) does not exist? [DEBUG] galaxy:139 2011-10-03 21:19:48,544: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:19:48,545: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:04,821: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:04,822: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:21,089: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:21,089: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:37,369: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:37,370: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; 21,313: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:02:21,314: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:02:21,314: Setting up SGE. [ERROR] misc:514 2011-10-03 21:02:21,424: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] root:354 2011-10-03 21:02:30,686: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:02:30,687: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:30,692: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [INFO] sge:28 2011-10-03 21:02:31,978: Removing SGE service [ERROR] misc:514 2011-10-03 21:02:31,983: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] sge:61 2011-10-03 21:02:31,984: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:02:31,984: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:02:32,216: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:02:33,718: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:02:33,719: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:02:33,719: Setting up SGE. [ERROR] misc:514 2011-10-03 21:02:33,830: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] galaxy:139 2011-10-03 21:02:35,204: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:35,205: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:02:51,478: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:02:51,478: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:07,771: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:07,772: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:24,109: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:24,109: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:40,384: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:40,384: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:03:56,651: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:03:56,651: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:12,920: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:12,921: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:29,215: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:29,215: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:04:45,524: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:04:45,524: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:01,832: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:01,832: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:18,100: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:18,101: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:34,374: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:34,374: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:05:50,636: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:05:50,637: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:06,984: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:06,985: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:23,253: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:23,254: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:39,536: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:39,537: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:06:55,787: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:06:55,788: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] root:354 2011-10-03 21:07:09,086: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:07:09,086: Removing SGE service [ERROR] misc:514 2011-10-03 21:07:09,092: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] root:354 2011-10-03 21:07:10,053: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:07:10,053: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:07:10,053: Shutting down Galaxy... [DEBUG] misc:511 2011-10-03 21:07:11,409: Successfully stopped Galaxy. [DEBUG] master:1510 2011-10-03 21:07:12,062: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Shut down; [DEBUG] root:354 2011-10-03 21:07:13,247: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:07:13,247: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:07:13,247: Shutting down Galaxy... [ERROR] misc:514 2011-10-03 21:07:13,594: Error stopping Galaxy, running command '/bin/su - galaxy -c "sh $GALAXY_HOME/run.sh --stop-daemon"' returned code '1' and following stderr: '' [DEBUG] root:354 2011-10-03 21:07:19,796: Managing services: [<cm.services.apps.postgres.PostgresService object at 0x2af8290>] [INFO] postgres:24 2011-10-03 21:07:19,797: Removing 'Postgres' service [DEBUG] misc:511 2011-10-03 21:07:19,863: Successfully set ownership of Postgres data directory [INFO] postgres:102 2011-10-03 21:07:19,864: Stopping PostgreSQL... [DEBUG] misc:511 2011-10-03 21:07:20,910: Successfully stopped PostgreSQL [DEBUG] root:354 2011-10-03 21:07:25,447: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [DEBUG] sge:61 2011-10-03 21:07:25,447: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:07:25,448: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:07:25,617: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:07:27,074: Configuring SGE... [DEBUG] sge:104 2011-10-03 21:07:27,075: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:07:27,075: Setting up SGE. [ERROR] misc:514 2011-10-03 21:07:27,186: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] master:1510 2011-10-03 21:07:28,269: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:07:44,467: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:00,668: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:16,861: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:08:33,066: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] root:354 2011-10-03 21:08:44,630: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:08:44,631: Removing SGE service [ERROR] misc:514 2011-10-03 21:08:44,636: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] master:1510 2011-10-03 21:08:49,320: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:05,516: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:21,725: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:37,940: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:09:54,203: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:10,422: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:26,625: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:42,830: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:10:59,055: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:15,257: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:31,577: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:11:47,765: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:03,974: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:20,166: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:36,399: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:12:52,608: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:08,812: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:25,006: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:41,208: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:13:57,398: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:13,734: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:29,943: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:14:46,152: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:02,378: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:18,599: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:34,849: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:15:51,043: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:07,329: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:23,517: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:39,712: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:16:56,062: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:17:12,268: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] master:1510 2011-10-03 21:17:28,489: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Shut down; Galaxy..Shutting down; [DEBUG] root:354 2011-10-03 21:17:36,296: Managing services: [<cm.services.apps.sge.SGEService object at 0x2904f90>] [INFO] sge:28 2011-10-03 21:17:36,296: Removing SGE service [ERROR] misc:514 2011-10-03 21:17:36,302: Problems stopping SGE master, running command 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -km' returned code '2' and following stderr: '.: 1: Can't open /opt/sge/default/common/settings.sh ' [DEBUG] root:354 2011-10-03 21:17:37,230: Managing services: [<cm.services.apps.postgres.PostgresService object at 0x2af8290>] [INFO] postgres:24 2011-10-03 21:17:37,230: Removing 'Postgres' service [DEBUG] misc:511 2011-10-03 21:17:37,289: Successfully set ownership of Postgres data directory [INFO] postgres:102 2011-10-03 21:17:37,290: Stopping PostgreSQL... [ERROR] misc:514 2011-10-03 21:17:37,325: Encountered problem while stopping PostgreSQL, running command '/bin/su - postgres -c "/usr/lib/postgresql/8.4/bin/pg_ctl -w -D /mnt/galaxyData/pgsql/data -o\"-p 5840\" stop"' returned code '1' and following stderr: 'pg_ctl: PID file "/mnt/galaxyData/pgsql/data/postmaster.pid" does not exist Is server running? ' [DEBUG] root:354 2011-10-03 21:17:37,887: Managing services: [<cm.services.apps.galaxy.GalaxyService object at 0x2af8250>] [INFO] galaxy:30 2011-10-03 21:17:37,888: Removing 'Galaxy' service [INFO] galaxy:122 2011-10-03 21:17:37,888: Shutting down Galaxy... [ERROR] misc:514 2011-10-03 21:17:38,244: Error stopping Galaxy, running command '/bin/su - galaxy -c "sh $GALAXY_HOME/run.sh --stop-daemon"' returned code '1' and following stderr: '' [ERROR] postgres:138 2011-10-03 21:17:44,686: PostgreSQL daemon NOT running. [DEBUG] master:1510 2011-10-03 21:17:44,686: S&S: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..Error; Galaxy..Shutting down; Python version: (2, 6) [DEBUG] app:42 2011-10-03 21:18:47,413: Initializing app [DEBUG] ec2:207 2011-10-03 21:18:47,413: Getting boto S3 connection [DEBUG] ec2:209 2011-10-03 21:18:47,413: No S3 Connection, creating a new one. [DEBUG] ec2:212 2011-10-03 21:18:47,413: Got boto S3 connection. [INFO] misc:378 2011-10-03 21:18:47,495: Retrieved file 'persistent_data.yaml' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to 'pd.yaml'. [INFO] app:52 2011-10-03 21:18:47,501: Master starting [DEBUG] master:50 2011-10-03 21:18:47,604: Initializing console manager - cluster start time: 2011-10-03 21:18:47.604445 [DEBUG] connection:648 2011-10-03 21:18:47,606: Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '1.7.2', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL. See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US'] [DEBUG] connection:494 2011-10-03 21:18:47,646: Open OK! known_hosts [domU-12-31-39-01-61-F2:5672] [DEBUG] channel:70 2011-10-03 21:18:47,646: using channel_id: 1 [DEBUG] channel:484 2011-10-03 21:18:47,647: Channel open [DEBUG] comm:35 2011-10-03 21:18:47,649: Successfully established AMQP connection [DEBUG] master:436 2011-10-03 21:18:47,649: Trying to discover any worker instances associated with this cluster... [DEBUG] ec2:177 2011-10-03 21:18:47,649: Establishing boto EC2 connection [DEBUG] ec2:85 2011-10-03 21:18:47,649: Gathering instance zone, attempt 0 [DEBUG] ec2:90 2011-10-03 21:18:47,652: Instance zone is 'us-east-1a' [DEBUG] ec2:198 2011-10-03 21:18:47,933: Got boto EC2 connection for region 'us-east-1' [DEBUG] ec2:66 2011-10-03 21:18:47,981: Gathering instance id, attempt 0 [DEBUG] ec2:71 2011-10-03 21:18:47,983: Instance ID is 'i-90bed4f0' [DEBUG] ec2:228 2011-10-03 21:18:48,053: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'i-90bed4f0' [DEBUG] ec2:228 2011-10-03 21:18:48,094: Adding tag 'role:master' to resource 'i-90bed4f0' [DEBUG] master:147 2011-10-03 21:18:48,160: ud at manager start: {'access_key': 'AKIAILGV2D5H2XA73EZQ', 'data_filesystems': {'galaxyData': [{'vol_id': 'vol-b53762df', 'size': 600}]}, 'cloudman_home': '/mnt/cm', 'static_filesystems': [{'snap_id': u'snap-5b030634', 'filesystem': 'galaxyIndices', 'size': 700}, {'snap_id': u'snap-1688b978', 'filesystem': 'galaxyTools', 'size': 2}], 'cluster_name': 'CJDGalaxy_Cluster', 'services': [{'service': 'SGE'}, {'service': 'Postgres'}, {'service': 'Galaxy'}], 'bucket_default': 'cloudman', 'boot_script_name': 'cm_boot.py', 'bucket_cluster': 'cm-6d20503431d38fb901ad6a550b90e0d4', 'boot_script_path': '/tmp/cm', 'role': 'master', 'secret_key': 'ARHzUdsBnOVFgT0DE7+X7HXdfQjsZ3m4yFo5Q/S/', 'password': 'oclock'} [DEBUG] master:455 2011-10-03 21:18:48,160: Trying to discover any volumes attached to this instance... [DEBUG] base:55 2011-10-03 21:18:48,165: Enabling 'root' controller, class: CM [DEBUG] buildapp:80 2011-10-03 21:18:48,167: Enabling 'httpexceptions' middleware [DEBUG] buildapp:86 2011-10-03 21:18:48,172: Enabling 'recursive' middleware [DEBUG] buildapp:106 2011-10-03 21:18:48,180: Enabling 'print debug' middleware [DEBUG] buildapp:120 2011-10-03 21:18:48,211: Enabling 'error' middleware [DEBUG] buildapp:130 2011-10-03 21:18:48,211: Enabling 'config' middleware [DEBUG] buildapp:134 2011-10-03 21:18:48,211: Enabling 'x-forwarded-host' middleware Starting server in PID 955. [DEBUG] master:461 2011-10-03 21:18:48,307: Attached volumes: [Volume:vol-73bcf219, Volume:vol-5dbaf437, Volume:vol-ffbaf495, Volume:vol-b53762df, Volume:vol-e77e2c8d, Volume:vol-a37e2cc9] [DEBUG] ec2:237 2011-10-03 21:18:48,307: Getting tag 'cluster_name' on resource 'vol-73bcf219' [DEBUG] ec2:237 2011-10-03 21:18:48,307: Getting tag 'cluster_name' on resource 'vol-5dbaf437' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-ffbaf495' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-b53762df' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-e77e2c8d' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-a37e2cc9' [DEBUG] master:180 2011-10-03 21:18:48,308: Adding static filesystem: 'galaxyIndices' [DEBUG] ec2:237 2011-10-03 21:18:48,308: Getting tag 'cluster_name' on resource 'vol-73bcf219' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-5dbaf437' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-ffbaf495' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-b53762df' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-e77e2c8d' [DEBUG] ec2:237 2011-10-03 21:18:48,309: Getting tag 'cluster_name' on resource 'vol-a37e2cc9' [DEBUG] master:180 2011-10-03 21:18:48,310: Adding static filesystem: 'galaxyTools' [DEBUG] master:185 2011-10-03 21:18:48,310: Adding a previously existing data filesystem: 'galaxyData' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'SGE' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'Postgres' [DEBUG] master:193 2011-10-03 21:18:48,310: Adding service: 'Galaxy' [INFO] master:155 2011-10-03 21:18:48,310: Completed initial cluster configuration. [DEBUG] master:1488 2011-10-03 21:18:48,310: Monitor started; manager started serving on 0.0.0.0:42284 view at http://127.0.0.1:42284 [DEBUG] master:1515 2011-10-03 21:18:52,314: Monitor adding service 'SGE' [DEBUG] __init__:31 2011-10-03 21:18:52,314: Trying to add service 'SGE' [INFO] __init__:49 2011-10-03 21:18:52,321: Prerequisites OK; starting service 'SGE' [DEBUG] sge:61 2011-10-03 21:18:52,322: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-10-03 21:18:52,352: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-10-03 21:18:53,460: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-10-03 21:18:55,627: Configuring SGE... [DEBUG] ec2:136 2011-10-03 21:18:55,628: Gathering instance private hostname, attempt 0 [DEBUG] sge:104 2011-10-03 21:18:55,630: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-10-03 21:18:55,630: Setting up SGE. [ERROR] misc:514 2011-10-03 21:18:55,848: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) ' [DEBUG] master:1515 2011-10-03 21:18:55,848: Monitor adding service 'FS-galaxyIndices' [DEBUG] filesystem:311 2011-10-03 21:18:55,849: Trying to add service 'FS-galaxyIndices' [DEBUG] filesystem:135 2011-10-03 21:18:55,849: Creating a new volume of size '700' in zone 'us-east-1a' from snapshot 'snap-5b030634' [DEBUG] filesystem:139 2011-10-03 21:19:01,945: Created new volume of size '700' from snapshot 'snap-5b030634' with ID 'vol-0d6b3967' in zone 'us-east-1a' [DEBUG] ec2:228 2011-10-03 21:19:01,946: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-0d6b3967' [DEBUG] ec2:228 2011-10-03 21:19:02,001: Adding tag 'filesystem:galaxyIndices' to resource 'vol-0d6b3967' [DEBUG] filesystem:319 2011-10-03 21:19:02,076: Marked volume 'vol-0d6b3967' from file system 'galaxyIndices' as 'static' [DEBUG] filesystem:129 2011-10-03 21:19:02,130: Running on kernel version 2.6.32-308-ec2; set volume device as '/dev/sdg6' and attach device as '/dev/sdg6' [DEBUG] filesystem:167 2011-10-03 21:19:02,130: Attaching volume 'vol-0d6b3967' to instance 'i-90bed4f0' as device '/dev/sdg6' [DEBUG] filesystem:190 2011-10-03 21:19:02,222: Attaching volume 'vol-0d6b3967'; status: attaching (check 0/30) [DEBUG] filesystem:190 2011-10-03 21:19:04,328: Attaching volume 'vol-0d6b3967'; status: attaching (check 1/30) [DEBUG] filesystem:190 2011-10-03 21:19:06,486: Attaching volume 'vol-0d6b3967'; status: attaching (check 2/30) [DEBUG] filesystem:190 2011-10-03 21:19:08,566: Attaching volume 'vol-0d6b3967'; status: attaching (check 3/30) [DEBUG] filesystem:190 2011-10-03 21:19:10,624: Attaching volume 'vol-0d6b3967'; status: attaching (check 4/30) [DEBUG] filesystem:190 2011-10-03 21:19:12,763: Attaching volume 'vol-0d6b3967'; status: attached (check 5/30) [DEBUG] filesystem:192 2011-10-03 21:19:12,764: Volume 'vol-0d6b3967' attached to instance 'i-90bed4f0' as device '/dev/sdg6' [DEBUG] filesystem:426 2011-10-03 21:19:12,832: Path '/dev/sdg6' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:14,066: Successfully mounted file system '/mnt/galaxyIndices' from '/dev/sdg6' [DEBUG] misc:511 2011-10-03 21:19:14,074: Successfully edited '/mnt/galaxyIndices' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:15,161: As part of filesystem 'Filesystem-galaxyIndices' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:15,237: Monitor adding service 'FS-galaxyTools' [DEBUG] filesystem:311 2011-10-03 21:19:15,237: Trying to add service 'FS-galaxyTools' [DEBUG] filesystem:135 2011-10-03 21:19:15,238: Creating a new volume of size '2' in zone 'us-east-1a' from snapshot 'snap-1688b978' [DEBUG] filesystem:139 2011-10-03 21:19:15,671: Created new volume of size '2' from snapshot 'snap-1688b978' with ID 'vol-c56b39af' in zone 'us-east-1a' [DEBUG] ec2:228 2011-10-03 21:19:15,671: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-c56b39af' [DEBUG] ec2:228 2011-10-03 21:19:15,731: Adding tag 'filesystem:galaxyTools' to resource 'vol-c56b39af' [DEBUG] filesystem:319 2011-10-03 21:19:15,778: Marked volume 'vol-c56b39af' from file system 'galaxyTools' as 'static' [DEBUG] filesystem:129 2011-10-03 21:19:15,828: Running on kernel version 2.6.32-308-ec2; set volume device as '/dev/sdg7' and attach device as '/dev/sdg7' [DEBUG] filesystem:167 2011-10-03 21:19:15,828: Attaching volume 'vol-c56b39af' to instance 'i-90bed4f0' as device '/dev/sdg7' [DEBUG] filesystem:190 2011-10-03 21:19:15,920: Attaching volume 'vol-c56b39af'; status: attaching (check 0/30) [DEBUG] filesystem:190 2011-10-03 21:19:18,051: Attaching volume 'vol-c56b39af'; status: attaching (check 1/30) [DEBUG] filesystem:190 2011-10-03 21:19:20,200: Attaching volume 'vol-c56b39af'; status: attaching (check 2/30) [DEBUG] filesystem:190 2011-10-03 21:19:22,259: Attaching volume 'vol-c56b39af'; status: attaching (check 3/30) [DEBUG] filesystem:190 2011-10-03 21:19:24,308: Attaching volume 'vol-c56b39af'; status: attaching (check 4/30) [DEBUG] filesystem:190 2011-10-03 21:19:26,506: Attaching volume 'vol-c56b39af'; status: attached (check 5/30) [DEBUG] filesystem:192 2011-10-03 21:19:26,507: Volume 'vol-c56b39af' attached to instance 'i-90bed4f0' as device '/dev/sdg7' [DEBUG] filesystem:426 2011-10-03 21:19:26,554: Path '/dev/sdg7' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:27,482: Successfully mounted file system '/mnt/galaxyTools' from '/dev/sdg7' [DEBUG] misc:511 2011-10-03 21:19:27,489: Successfully edited '/mnt/galaxyTools' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:28,726: As part of filesystem 'Filesystem-galaxyTools' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:28,825: Monitor adding service 'FS-galaxyData' [DEBUG] filesystem:311 2011-10-03 21:19:28,826: Trying to add service 'FS-galaxyData' [DEBUG] filesystem:143 2011-10-03 21:19:28,950: Tried to create a volume but it is in state 'attached' (volume ID: vol-b53762df) [DEBUG] ec2:228 2011-10-03 21:19:28,950: Adding tag 'clusterName:CJDGalaxy_Cluster' to resource 'vol-b53762df' [DEBUG] ec2:228 2011-10-03 21:19:28,995: Adding tag 'filesystem:galaxyData' to resource 'vol-b53762df' [DEBUG] filesystem:214 2011-10-03 21:19:29,164: Tried to attach a volume but the volume 'vol-b53762df' is already attached (as device /dev/sdg3) [DEBUG] filesystem:426 2011-10-03 21:19:29,249: Path '/dev/sdg3' checked and exists. [DEBUG] misc:511 2011-10-03 21:19:29,690: Successfully mounted file system '/mnt/galaxyData' from '/dev/sdg3' [DEBUG] misc:511 2011-10-03 21:19:29,702: Successfully edited '/mnt/galaxyData' in '/etc/exports' for NFS. [DEBUG] misc:511 2011-10-03 21:19:30,804: As part of filesystem 'Filesystem-galaxyData' update, successfully restarted NFS server [DEBUG] master:1515 2011-10-03 21:19:30,854: Monitor adding service 'Postgres' [DEBUG] __init__:31 2011-10-03 21:19:30,855: Trying to add service 'Postgres' [DEBUG] __init__:36 2011-10-03 21:19:30,855: 'Postgres' service checking its prerequisite 'Filesystem:galaxyData' [INFO] __init__:49 2011-10-03 21:19:30,855: Prerequisites OK; starting service 'Postgres' [DEBUG] misc:511 2011-10-03 21:19:31,057: Successfully set ownership of Postgres data directory [ERROR] postgres:138 2011-10-03 21:19:31,058: PostgreSQL daemon NOT running. [ERROR] postgres:138 2011-10-03 21:19:31,058: PostgreSQL daemon NOT running. [INFO] postgres:90 2011-10-03 21:19:31,058: Starting PostgreSQL... [DEBUG] misc:511 2011-10-03 21:19:31,079: Successfully changed owner of postgres data dir [DEBUG] misc:511 2011-10-03 21:19:32,218: Successfully started PostgreSQL server [INFO] postgres:95 2011-10-03 21:19:32,332: Successfully started PostgreSQL. [DEBUG] master:1515 2011-10-03 21:19:32,332: Monitor adding service 'Galaxy' [DEBUG] __init__:31 2011-10-03 21:19:32,332: Trying to add service 'Galaxy' [DEBUG] __init__:36 2011-10-03 21:19:32,333: 'Galaxy' service checking its prerequisite 'Postgres:None' [DEBUG] __init__:36 2011-10-03 21:19:32,333: 'Galaxy' service checking its prerequisite 'Filesystem:galaxyTools' [INFO] __init__:49 2011-10-03 21:19:32,333: Prerequisites OK; starting service 'Galaxy' [INFO] galaxy:50 2011-10-03 21:19:32,333: Setting up Galaxy application [DEBUG] ec2:207 2011-10-03 21:19:32,333: Getting boto S3 connection [INFO] misc:378 2011-10-03 21:19:32,943: Retrieved file 'universe_wsgi.ini.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/universe_wsgi.ini'. [INFO] misc:378 2011-10-03 21:19:34,459: Retrieved file 'tool_conf.xml.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/tool_conf.xml'. [INFO] misc:378 2011-10-03 21:19:34,565: Retrieved file 'tool_data_table_conf.xml.cloud' from bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' to '/mnt/galaxyTools/galaxy-central/tool_data_table_conf.xml.cloud'. [DEBUG] misc:511 2011-10-03 21:19:36,238: Started FTP server [INFO] galaxy:113 2011-10-03 21:19:36,238: Starting Galaxy... [DEBUG] misc:511 2011-10-03 21:19:43,828: Successfully initiated Galaxy start. [DEBUG] galaxy:139 2011-10-03 21:19:43,852: Galaxy UI does not seem to be accessible. [DEBUG] ec2:207 2011-10-03 21:19:43,852: Getting boto S3 connection [INFO] misc:394 2011-10-03 21:19:44,143: Saved file 'persistent_data.yaml' to bucket 'cm-6d20503431d38fb901ad6a550b90e0d4' [DEBUG] master:1462 2011-10-03 21:19:44,195: Instance post start script (/mnt/cm/post_start_script) does not exist? [DEBUG] galaxy:139 2011-10-03 21:19:48,544: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:19:48,545: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:04,821: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:04,822: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:21,089: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:21,089: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] galaxy:139 2011-10-03 21:20:37,369: Galaxy UI does not seem to be accessible. [DEBUG] master:1510 2011-10-03 21:20:37,370: S&S: SGE..Starting; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/