CloudMan from Galaxy Admin | Report bugs | Wiki | Screencast Only up to the most recent 200 lines of the file (paster.log) are shown. Show all | Show less | Show more | Back to admin view 2015-06-19 20:42:32,573 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:43:36,824 DEBUG sge:552 qstat: ['all.q@ip-172-31-2-194.ec2.inte BIP 0/0/2 0.13 lx24-amd64 '] 2015-06-19 20:43:37,425 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:44:41,676 DEBUG sge:552 qstat: ['all.q@ip-172-31-2-194.ec2.inte BIP 0/0/2 0.16 lx24-amd64 '] 2015-06-19 20:44:42,432 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:45:42,692 DEBUG sge:552 qstat: ['all.q@ip-172-31-2-194.ec2.inte BIP 0/0/2 0.12 lx24-amd64 '] 2015-06-19 20:45:43,248 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:46:47,500 DEBUG sge:552 qstat: ['all.q@ip-172-31-2-194.ec2.inte BIP 0/0/2 0.11 lx24-amd64 '] 2015-06-19 20:46:48,223 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:47:01,584 DEBUG sge:498 Removing instance 'i-fd327802' with FQDN '172.31.2.194' from SGE execution host list (including @allhosts) 2015-06-19 20:47:01,585 DEBUG sge:414 to_add: 'None' 2015-06-19 20:47:01,585 DEBUG sge:415 to_remove: '172.31.2.194' 2015-06-19 20:47:01,585 DEBUG sge:418 Composing SGE's @allhosts group config file /tmp/ah_remove_20_47_01: 2015-06-19 20:47:01,585 DEBUG sge:425 - master is marked as non-exec host and will not be included in @allhosts file 2015-06-19 20:47:01,585 DEBUG sge:454 new_allhosts: group_name @allhosts hostlist NONE 2015-06-19 20:47:01,585 DEBUG sge:456 New SGE @allhosts file written successfully to /tmp/ah_remove_20_47_01. 2015-06-19 20:47:01,736 DEBUG sge:507 Successfully updated @allhosts to remove 'i-fd327802' 2015-06-19 20:47:02,051 DEBUG sge:526 Successfully removed instance 'i-fd327802' with IP '172.31.2.194' from SGE execution host list. 2015-06-19 20:47:02,052 INFO master:845 The master instance has been set to *not* execute jobs. To manually change this, use the cloudman admin panel. 2015-06-19 20:47:02,052 INFO ec2:443 Adding 1 on-demand instance(s) 2015-06-19 20:47:02,054 DEBUG ec2:474 Starting instance(s) in VPC with the following command : ec2_conn.run_instances( image_id='ami-a7dbf6ce', min_count='1', max_count='1', key_name='cloudman_key_pair', security_group_ids=['sg-00858065'], user_data(with password/secret_key filtered out)=[s3_port: None cloud_type: ec2 master_public_ip: 54.88.49.203 freenxpass: !galaxySATDEVZGK cluster_type: Galaxy tags: {u'i-b8083047': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-58edd5a7': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-9ff2cf60': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-fd327802': {'clusterName': u'icam_galaxy', 'role': 'master', 'Name': 'master: icam_galaxy'}, u'i-e9251816': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-98f2cf67': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-e21b261d': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-210c0ade': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-2edfe4d1': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-64edd59b': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'i-b9083046': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'vol-86b205c0': {'clusterName': u'icam_galaxy'}, u'i-e31b261c': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, u'vol-d7bd0a91': {'clusterName': u'icam_galaxy', 'Name': 'galaxyIndicesFS', 'bucketName': 'cm-442caa28f69846327455b14f155f75cd', 'roles': 'galaxyIndices', 'filesystem': 'galaxyIndices'}, u'vol-8ebd0ac8': {'clusterName': u'icam_galaxy', 'Name': 'galaxyFS', 'bucketName': 'cm-442caa28f69846327455b14f155f75cd', 'roles': 'galaxyTools,galaxyData', 'filesystem': 'galaxy'}} master_ip: 172.31.2.194 cluster_name: icam_galaxy persistent_data_version: 3 machine_image_id: ami-a7dbf6ce role: worker bucket_cluster: cm-442caa28f69846327455b14f155f75cd boot_script_path: /tmp/cm master_hostname: ip-172-31-2-194.ec2.internal ec2_conn_path: / region_name: us-east-1 region_endpoint: ec2.amazonaws.com ec2_port: None deployment_version: 2 filesystems: [{'kind': 'volume', 'mount_point': '/mnt/galaxy', 'name': 'galaxy', 'roles': ['galaxyTools', 'galaxyData'], 'ids': [u'vol-8ebd0ac8']}, {'kind': 'snapshot', 'mount_point': '/mnt/galaxyIndices', 'name': 'galaxyIndices', 'roles': ['galaxyIndices'], 'ids': ['snap-4b20f451']}] cloud_name: Amazon boot_script_name: cm_boot.py services: [{'name': 'Migration', 'roles': ['Migration']}, {'name': 'SGE', 'roles': ['SGE']}, {'name': 'Postgres', 'roles': ['Postgres']}, {'name': 'ProFTPd', 'roles': ['ProFTPd']}, {'home': '/mnt/galaxy/galaxy-app', 'name': 'Galaxy', 'roles': ['Galaxy']}, {'name': 'GalaxyReports', 'roles': ['GalaxyReports']}] is_secure: True access_key: AKIAJQJYPZIMLKSDWHAQ placement: us-east-1a cloudman_home: /mnt/cm bucket_default: cloudman s3_host: s3.amazonaws.com s3_conn_path: /], instance_type='c3.large', placement='us-east-1a', subnet_id='subnet-e19863b8') 2015-06-19 20:47:05,720 DEBUG ec2:409 Adding tag 'clusterName:icam_galaxy' to resource 'i-69a8a496' 2015-06-19 20:47:05,908 DEBUG ec2:409 Adding tag 'role:worker' to resource 'i-69a8a496' 2015-06-19 20:47:06,056 DEBUG ec2:409 Adding tag 'Name:Worker: icam_galaxy' to resource 'i-69a8a496' 2015-06-19 20:47:06,249 DEBUG ec2:517 Adding Instance Instance:i-69a8a496 2015-06-19 20:47:06,250 DEBUG ec2:531 Started 1 instance(s) 2015-06-19 20:47:08,462 DEBUG sge:552 qstat: [] 2015-06-19 20:47:09,121 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:47:09,380 DEBUG master:3024 Got public IP for instance i-69a8a496: 54.164.165.200 2015-06-19 20:47:09,380 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 3 secs ago); will wait a bit longer before a check... 2015-06-19 20:47:21,618 DEBUG sge:552 qstat: [] 2015-06-19 20:47:22,372 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:47:22,373 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 16 secs ago); will wait a bit longer before a check... 2015-06-19 20:47:34,610 DEBUG sge:552 qstat: [] 2015-06-19 20:47:35,346 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:47:35,347 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 29 secs ago); will wait a bit longer before a check... 2015-06-19 20:47:47,587 DEBUG sge:552 qstat: [] 2015-06-19 20:47:48,149 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:47:48,150 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:47:48,251 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:47:48,251 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:47:48 | last m_state change before 0:00:42 | time_rebooted before 20:47:48 2015-06-19 20:48:00,497 DEBUG sge:552 qstat: [] 2015-06-19 20:48:01,104 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:48:01,104 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait a bit longer before a check... 2015-06-19 20:48:13,341 DEBUG sge:552 qstat: [] 2015-06-19 20:48:13,872 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:48:13,873 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait a bit longer before a check... 2015-06-19 20:48:26,112 DEBUG sge:552 qstat: [] 2015-06-19 20:48:26,654 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:48:26,655 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:48:26,721 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:48:26,721 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:48:26 | last m_state change before 0:01:20 | time_rebooted before 20:48:26 2015-06-19 20:48:38,962 DEBUG sge:552 qstat: [] 2015-06-19 20:48:39,578 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:48:39,578 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait a bit longer before a check... 2015-06-19 20:48:51,819 DEBUG sge:552 qstat: [] 2015-06-19 20:48:52,390 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:48:52,391 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait a bit longer before a check... 2015-06-19 20:49:04,626 DEBUG sge:552 qstat: [] 2015-06-19 20:49:05,181 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:49:05,181 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:49:05,283 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:49:05,283 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:49:05 | last m_state change before 0:01:59 | time_rebooted before 20:49:05 2015-06-19 20:49:17,520 DEBUG sge:552 qstat: [] 2015-06-19 20:49:18,146 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:49:18,147 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait a bit longer before a check... 2015-06-19 20:49:30,384 DEBUG sge:552 qstat: [] 2015-06-19 20:49:31,572 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:49:31,572 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 26 secs ago); will wait a bit longer before a check... 2015-06-19 20:49:43,809 DEBUG sge:552 qstat: [] 2015-06-19 20:49:44,426 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:49:44,427 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:49:44,502 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:49:44,502 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:49:44 | last m_state change before 0:02:38 | time_rebooted before 20:49:44 2015-06-19 20:49:56,743 DEBUG sge:552 qstat: [] 2015-06-19 20:49:57,272 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:49:57,273 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait a bit longer before a check... 2015-06-19 20:50:09,520 DEBUG sge:552 qstat: [] 2015-06-19 20:50:10,102 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:50:10,103 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait a bit longer before a check... 2015-06-19 20:50:22,342 DEBUG sge:552 qstat: [] 2015-06-19 20:50:22,982 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:50:22,983 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:50:23,058 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:50:23,058 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:50:23 | last m_state change before 0:03:16 | time_rebooted before 20:50:23 2015-06-19 20:50:35,300 DEBUG sge:552 qstat: [] 2015-06-19 20:50:35,857 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:50:35,858 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait a bit longer before a check... 2015-06-19 20:50:48,163 DEBUG sge:552 qstat: [] 2015-06-19 20:50:48,751 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:50:48,752 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait a bit longer before a check... 2015-06-19 20:51:00,990 DEBUG sge:552 qstat: [] 2015-06-19 20:51:01,652 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:51:01,653 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:51:01,758 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:51:01,758 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:51:01 | last m_state change before 0:03:55 | time_rebooted before 20:51:01 2015-06-19 20:51:13,997 DEBUG sge:552 qstat: [] 2015-06-19 20:51:14,869 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:51:14,870 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 13 secs ago); will wait a bit longer before a check... 2015-06-19 20:51:27,125 DEBUG sge:552 qstat: [] 2015-06-19 20:51:27,787 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:51:27,788 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 26 secs ago); will wait a bit longer before a check... 2015-06-19 20:51:40,053 DEBUG sge:552 qstat: [] 2015-06-19 20:51:40,577 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:51:40,578 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:51:40,644 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:51:40,645 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:51:40 | last m_state change before 0:04:34 | time_rebooted before 20:51:40 2015-06-19 20:51:52,884 DEBUG sge:552 qstat: [] 2015-06-19 20:51:53,403 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:51:53,404 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait a bit longer before a check... 2015-06-19 20:52:05,641 DEBUG sge:552 qstat: [] 2015-06-19 20:52:06,206 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:52:06,207 DEBUG master:2562 Instance 'i-69a8a496' (IP: 54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait a bit longer before a check... 2015-06-19 20:52:38,452 DEBUG sge:552 qstat: [] 2015-06-19 20:52:38,996 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:52:38,996 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:52:39,093 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:52:39,093 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:52:39 | last m_state change before 0:05:32 | time_rebooted before 20:52:39 2015-06-19 20:53:11,370 DEBUG sge:552 qstat: [] 2015-06-19 20:53:12,115 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:53:12,115 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:53:12,217 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:53:12,217 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:53:12 | last m_state change before 0:06:05 | time_rebooted before 20:53:12 2015-06-19 20:53:44,462 DEBUG sge:552 qstat: [] 2015-06-19 20:53:45,254 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:53:45,254 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:53:45,348 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:53:45,348 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:53:45 | last m_state change before 0:06:39 | time_rebooted before 20:53:45 2015-06-19 20:54:17,595 DEBUG sge:552 qstat: [] 2015-06-19 20:54:18,145 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:54:18,146 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:54:18,223 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:54:18,223 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:54:18 | last m_state change before 0:07:11 | time_rebooted before 20:54:18 2015-06-19 20:54:18,223 INFO master:2810 Rebooting instance 'i-69a8a496' (IP: 54.164.165.200) (reboot #1). 2015-06-19 20:54:18,331 DEBUG master:2823 Incremented instance reboot count to 1 (out of 4) 2015-06-19 20:54:50,573 DEBUG sge:552 qstat: [] 2015-06-19 20:54:51,227 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:54:51,227 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:54:51,303 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:54:51,303 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:54:51 | last m_state change before 0:07:45 | time_rebooted before 0:00:32 2015-06-19 20:55:23,547 DEBUG sge:552 qstat: [] 2015-06-19 20:55:24,143 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:55:24,144 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:55:24,248 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:55:24,249 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:55:24 | last m_state change before 0:08:17 | time_rebooted before 0:01:05 2015-06-19 20:55:56,497 DEBUG sge:552 qstat: [] 2015-06-19 20:55:57,061 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:55:57,061 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:55:57,124 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:55:57,124 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:55:57 | last m_state change before 0:08:50 | time_rebooted before 0:01:38 2015-06-19 20:56:29,374 DEBUG sge:552 qstat: [] 2015-06-19 20:56:30,083 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:56:30,083 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:56:30,168 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:56:30,169 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:56:30 | last m_state change before 0:09:23 | time_rebooted before 0:02:11 2015-06-19 20:57:02,427 DEBUG sge:552 qstat: [] 2015-06-19 20:57:03,010 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; GalaxyReports..OK; 2015-06-19 20:57:03,010 DEBUG master:2556 Have not heard from or checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now. 2015-06-19 20:57:03,073 DEBUG master:2894 Requested instance 'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running 2015-06-19 20:57:03,074 DEBUG master:2659 'Maintaining' instance 'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:57:03 | last m_state change before 0:09:56 | time_rebooted before 0:02:44 2015-06-19 20:57:41,299 DEBUG master:1900 Checking for new version of CloudMan 2015-06-19 20:57:41,300 DEBUG misc:668 Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cm-442caa28f69846327455b14f155f75cd' 2015-06-19 20:57:41,393 DEBUG misc:668 Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cloudman' 2015-06-19 20:57:41,462 DEBUG master:1907 Revision number for user's CloudMan: '9999999'; revision number for default CloudMan: 'None' 2015-06-19 20:57:41,659 DEBUG master:1900 Checking for new version of CloudMan 2015-06-19 20:57:41,659 DEBUG misc:668 Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cm-442caa28f69846327455b14f155f75cd' 2015-06-19 20:57:41,691 DEBUG misc:668 Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cloudman' 2015-06-19 20:57:41,730 DEBUG master:1907 Revision number for user's CloudMan: '9999999'; revision number for default CloudMan: 'None'