Unable to start galaxy either via ami or https://main.g2.bx.psu.edu/cloudlaunch - galaxy-user

26 Jun 2013


      I'm trying to get an instance launched and I'm not seeing an instance launch.
When I start an ami directly :  (ami-da58aab3) it never answers when I try and browser to it via the web (nor does it respond when I do curl localhost)
When I use cloud launch,  I can browse to it on the web but on the console it never shows galaxy or postgres as starting (when I try and start postgres it says Cannot find Postgres service – same with starting Galaxy)
When I look at the cloud man log – I can see a couple of what appear to be errors in it :
[INFO] sge:122 2013-06-25 22:15:08,979: Successfully setup SGE; configuring SGE
[DEBUG] sge:123 2013-06-25 22:15:08,979: Adding parallel environments
[DEBUG] misc:577 2013-06-25 22:15:08,993: 'cd /opt/sge; ./bin/lx24-amd64/qconf -Ap /tmp/SMP_PE' command OK
[DEBUG] misc:577 2013-06-25 22:15:09,006: 'cd /opt/sge; ./bin/lx24-amd64/qconf -Ap /tmp/MPI_PE' command OK
[DEBUG] sge:130 2013-06-25 22:15:09,006: Creating queue 'all.q'
[DEBUG] sge:135 2013-06-25 22:15:09,006: Created SGE all.q template as file '/opt/sge/all.q.conf'
[DEBUG] misc:577 2013-06-25 22:15:09,021: Successfully modified all.q
[DEBUG] sge:137 2013-06-25 22:15:09,021: Configuring users' SGE profiles
[DEBUG] master:1676 2013-06-25 22:15:09,022: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:09,022: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] misc:113 2013-06-25 22:15:10,054: Checking if bucket 'cm-81a2ec9d11e15570ea7136a31533170e' exists... it does not.
[DEBUG] misc:124 2013-06-25 22:15:10,222: Created bucket 'cm-81a2ec9d11e15570ea7136a31533170e'.
[INFO] misc:455 2013-06-25 22:15:10,637: Saved file 'persistent_data.yaml' to bucket 'cm-81a2ec9d11e15570ea7136a31533170e'
[DEBUG] master:1601 2013-06-25 22:15:10,666: Saving current instance boot script (/tmp/cm/cm_boot.py) to cluster bucket 'cm-81a2ec9d11e15570ea7136a31533170e' as 'cm_boot.py'
[INFO] misc:455 2013-06-25 22:15:10,713: Saved file 'cm_boot.py' to bucket 'cm-81a2ec9d11e15570ea7136a31533170e'
[DEBUG] master:1605 2013-06-25 22:15:10,740: Saving CloudMan source (/mnt/cm/cm.tar.gz) to cluster bucket 'cm-81a2ec9d11e15570ea7136a31533170e' as 'cm.tar.gz'
[INFO] misc:455 2013-06-25 22:15:11,120: Saved file 'cm.tar.gz' to bucket 'cm-81a2ec9d11e15570ea7136a31533170e'
[DEBUG] misc:527 2013-06-25 22:15:11,120: Setting metadata 'revision' for file 'cm.tar.gz' in bucket 'cm-81a2ec9d11e15570ea7136a31533170e'
[DEBUG] master:1619 2013-06-25 22:15:11,314: Saving '/mnt/cm/bz_test.clusterName' file to cluster bucket 'cm-81a2ec9d11e15570ea7136a31533170e' as 'bz_test.clusterName'
[INFO] misc:455 2013-06-25 22:15:11,347: Saved file 'bz_test.clusterName' to bucket 'cm-81a2ec9d11e15570ea7136a31533170e'
[DEBUG] sge:405 2013-06-25 22:15:15,747: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.64     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:15:15,748: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:15:15,748: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:15,748: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:19,758: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:19,758: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:23,768: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:23,768: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] sge:405 2013-06-25 22:15:28,166: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.64     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:15:28,167: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:15:28,167: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:28,167: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:32,178: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:32,178: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:36,188: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:36,188: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] sge:405 2013-06-25 22:15:40,584: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.64     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:15:40,585: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:15:40,585: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:40,585: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:44,588: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:44,588: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:48,598: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:48,598: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] sge:405 2013-06-25 22:15:52,995: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.75     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:15:52,996: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:15:52,996: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:52,996: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:15:56,998: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:15:56,998: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:16:01,008: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:01,008: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] sge:405 2013-06-25 22:16:05,406: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.75     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:16:05,407: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:16:05,407: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:05,407: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:16:09,418: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:09,418: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:16:13,422: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:13,422: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] sge:405 2013-06-25 22:16:17,821: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.75     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:16:17,821: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:16:17,821: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:17,822: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:16:21,832: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:21,832: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1676 2013-06-25 22:16:25,842: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:25,842: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] sge:405 2013-06-25 22:16:30,241: qstat: ['all.q@ip-10-60-121-222.ec2.int BIP   0/0/2          0.76     lx24-amd64    ']
[DEBUG] master:1650 2013-06-25 22:16:30,242: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-06-25 22:16:30,242: Monitor adding service 'post_start_script'
[DEBUG] pss:65 2013-06-25 22:16:30,242: Not adding post_start_script svc; it already ran (False) or the cluster was not yet initialized (None)
[DEBUG] master:1247 2013-06-25 22:16:32,585: Checking for new version of CloudMan
[DEBUG] misc:512 2013-06-25 22:16:32,585: Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cm-81a2ec9d11e15570ea7136a31533170e'
[DEBUG] misc:512 2013-06-25 22:16:33,639: Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cloudman'
[DEBUG] master:1251 2013-06-25 22:16:33,690: Revision number for user's CloudMan: '262'; revision number for default CloudMan: '262'
[DEBUG] ec2:48 2013-06-25 22:16:33,690: Gathering instance type, attempt 0
…
[DEBUG] pss:74 2013-06-25 22:18:00,864: post_start_script service prerequisites OK (i.e., all other services running), checking if post_start_script was provided...
[DEBUG] pss:91 2013-06-25 22:18:00,899: Cluster bucket 'cm-81a2ec9d11e15570ea7136a31533170e' found; looking for post start script 'post_start_script'
[DEBUG] misc:440 2013-06-25 22:18:00,960: Failed to get file 'post_start_script' from bucket 'cm-81a2ec9d11e15570ea7136a31533170e': S3ResponseError: 404 Not Found
NoSuchKeyThe specified key does not exist.post_start_scriptC0097A621B19D07CJtBjACdzRU2Ko+A4xAe5hCPYd3p1EfXw6Hx0Y5DjXLPU7XS3CZyinJs6CMbxxXUS
[DEBUG] pss:102 2013-06-25 22:18:00,960: post_start_script does not exist or could not be downloaded; continuing without running it.
Any idea how I can get this started and running?
Thanks!
-Brad