New subject: Cloudman - When will the symlinks be resolved ?

1 Aug 2013

      Here it is:

'pd.yaml' not found at paths.py load!
Set paths.P_GALAXY_HOME as '/mnt/galaxyTools/galaxy-central'
Python version:  (2, 6)
Image configuration suports: {'apps': ['cloudman', 'galaxy']}
[DEBUG] app:54 2013-08-01 20:07:40,138: Initializing app
[DEBUG] app:55 2013-08-01 20:07:40,138: Running on 'ec2' type of cloud.
[DEBUG] app:72 2013-08-01 20:07:40,138: Getting pd.yaml
[DEBUG] ec2:275 2013-08-01 20:07:40,138: No S3 Connection, creating a new one.
[DEBUG] ec2:278 2013-08-01 20:07:40,139: Got boto S3 connection.
[DEBUG] misc:113 2013-08-01 20:07:40,229: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not.
[DEBUG] misc:444 2013-08-01 20:07:40,229: Bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' does not exist, did not get remote file 'persistent_data.yaml'
[INFO] app:78 2013-08-01 20:07:40,229: Master starting
[DEBUG] master:70 2013-08-01 20:07:40,229: Initializing console manager - cluster start time: 2013-08-01 20:07:40.229549
[DEBUG] connection:648 2013-08-01 20:07:40,231: Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '1.7.2', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US']
[DEBUG] connection:494 2013-08-01 20:07:40,263: Open OK! known_hosts [ip-10-179-15-41.ec2.internal:5672]
[DEBUG] channel:70 2013-08-01 20:07:40,263: using channel_id: 1
[DEBUG] channel:484 2013-08-01 20:07:40,264: Channel open
[DEBUG] comm:35 2013-08-01 20:07:40,266: Successfully established AMQP connection
[DEBUG] master:510 2013-08-01 20:07:40,266: Trying to discover any worker instances associated with this cluster...
[DEBUG] ec2:246 2013-08-01 20:07:40,267: Establishing boto EC2 connection
[DEBUG] ec2:103 2013-08-01 20:07:40,267: Gathering instance zone, attempt 0
[DEBUG] ec2:108 2013-08-01 20:07:40,270: Instance zone is 'us-east-1d'
[DEBUG] ec2:263 2013-08-01 20:07:40,516: Got boto EC2 connection for region 'us-east-1'
[DEBUG] ec2:66 2013-08-01 20:07:40,574: Gathering instance id, attempt 0
[DEBUG] ec2:71 2013-08-01 20:07:40,575: Instance ID is 'i-7b386714'
[DEBUG] ec2:295 2013-08-01 20:07:40,639: Adding tag 'clusterName:gal_oldami_form2' to resource 'i-7b386714'
[DEBUG] ec2:295 2013-08-01 20:07:41,066: Adding tag 'role:master' to resource 'i-7b386714'
[DEBUG] master:174 2013-08-01 20:07:41,223: ud at manager start: {'access_key': 'AKIAIUSCRANAJESP3GFQ', 'cloudman_home': '/mnt/cm', 'cluster_name': 'gal_oldami_form2', 'bucket_default': 'gxy-workshop', 'boot_script_name': 'cm_boot.py', 'bucket_cluster': 'cm-35f4d40b627583b6949bc5de11bcfecc', 'boot_script_path': '/tmp/cm', 'role': 'master', 'secret_key': 'EV12T4PMGSbM1IZmS7ZUJVi9nCvMncnrxXvytacJ', 'password': 'google'}
[DEBUG] pss:25 2013-08-01 20:07:41,224: Configured PSS as master
[DEBUG] master:204 2013-08-01 20:07:41,225: Cheking for and adding any preconfigured services
[DEBUG] master:529 2013-08-01 20:07:41,225: Trying to discover any volumes attached to this instance...
[DEBUG] base:55 2013-08-01 20:07:41,236: Enabling 'root' controller, class: CM
[DEBUG] buildapp:80 2013-08-01 20:07:41,239: Enabling 'httpexceptions' middleware
[DEBUG] buildapp:86 2013-08-01 20:07:41,469: Enabling 'recursive' middleware
[DEBUG] buildapp:106 2013-08-01 20:07:41,719: Enabling 'print debug' middleware
[DEBUG] buildapp:120 2013-08-01 20:07:41,734: Enabling 'error' middleware
[DEBUG] buildapp:130 2013-08-01 20:07:41,735: Enabling 'config' middleware
[DEBUG] buildapp:134 2013-08-01 20:07:41,736: Enabling 'x-forwarded-host' middleware
[DEBUG] master:535 2013-08-01 20:07:41,737: Attached volumes: [Volume:vol-fe5b7abe]
[INFO] master:198 2013-08-01 20:07:41,758: Completed initial cluster configuration. This seems to be a new cluster; waiting to configure the type
[DEBUG] master:1627 2013-08-01 20:07:41,758: Monitor started; manager started
Starting server in PID 1252.
serving on 0.0.0.0:42284 view at http://127.0.0.1:42284
[DEBUG] master:1247 2013-08-01 20:07:45,657: Checking for new version of CloudMan
[DEBUG] misc:512 2013-08-01 20:07:45,657: Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] misc:113 2013-08-01 20:07:45,740: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not.
[DEBUG] misc:512 2013-08-01 20:07:45,740: Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'gxy-workshop'
[DEBUG] master:1676 2013-08-01 20:07:45,763: Monitor adding service 'SGE'
[DEBUG] __init__:31 2013-08-01 20:07:45,763: Trying to add service 'SGE'
[INFO] __init__:50 2013-08-01 20:07:45,763: SGE prerequisites OK; starting the service
[DEBUG] sge:62 2013-08-01 20:07:45,763: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5'
[DEBUG] __init__:42 2013-08-01 20:07:45,793:    'sge' daemon is NOT running any more (expected pid: '2521').
[DEBUG] sge:77 2013-08-01 20:07:45,793: Cleaning '/opt/sge' directory.
[DEBUG] master:1251 2013-08-01 20:07:45,804: Revision number for user's CloudMan: 'None'; revision number for default CloudMan: 'None'
[DEBUG] ec2:48 2013-08-01 20:07:45,805: Gathering instance type, attempt 0
[DEBUG] sge:83 2013-08-01 20:07:46,195: Unpacking SGE to '/opt/sge'.
[DEBUG] ec2:218 2013-08-01 20:07:46,267: Gathering instance public hostname, attempt 0
[INFO] master:817 2013-08-01 20:07:50,102: Initializing a 'Galaxy' cluster.
[INFO] misc:438 2013-08-01 20:07:51,212: Retrieved file 'snaps.yaml' from bucket 'gxy-workshop' to 'cm_snaps.yaml'.
Error - : 'static_filesystems'
[INFO] sge:97 2013-08-01 20:07:51,551: Configuring SGE...
[DEBUG] ec2:162 2013-08-01 20:07:51,552: Gathering instance private IP, attempt 0
[DEBUG] sge:105 2013-08-01 20:07:51,554: Created SGE install template as file '/opt/sge/galaxyEC2.conf'
[DEBUG] sge:119 2013-08-01 20:07:51,554: Setting up SGE.
[DEBUG] misc:577 2013-08-01 20:07:51,567: Modified /opt/sge/util/arch
[DEBUG] misc:577 2013-08-01 20:07:51,573: Successfully chmod /opt/sge/util/arch
[DEBUG] misc:577 2013-08-01 20:07:51,581: 'sed -i.bak '/^127.0.1./s/^/# (Commented by CloudMan) /' /etc/hosts' command OK
[DEBUG] ec2:142 2013-08-01 20:07:57,465: Gathering instance public keys (i.e., key pairs), attempt 0
[DEBUG] ec2:148 2013-08-01 20:07:57,468: Got key pair: 'rav_kp2'
[DEBUG] misc:577 2013-08-01 20:07:59,992: Successfully set up SGE
[INFO] sge:122 2013-08-01 20:07:59,993: Successfully setup SGE; configuring SGE
[DEBUG] sge:123 2013-08-01 20:07:59,993: Adding parallel environments
[DEBUG] misc:577 2013-08-01 20:08:00,009: 'cd /opt/sge; ./bin/lx24-amd64/qconf -Ap /tmp/SMP_PE' command OK
[DEBUG] misc:577 2013-08-01 20:08:00,026: 'cd /opt/sge; ./bin/lx24-amd64/qconf -Ap /tmp/MPI_PE' command OK
[DEBUG] sge:130 2013-08-01 20:08:00,026: Creating queue 'all.q'
[DEBUG] sge:135 2013-08-01 20:08:00,027: Created SGE all.q template as file '/opt/sge/all.q.conf'
[DEBUG] misc:577 2013-08-01 20:08:00,044: Successfully modified all.q
[DEBUG] sge:137 2013-08-01 20:08:00,045: Configuring users' SGE profiles
[DEBUG] master:1676 2013-08-01 20:08:00,045: Monitor adding service 'post_start_script'
[DEBUG] pss:51 2013-08-01 20:08:00,045: Custom-checking 'post_start_script' service prerequisites
[DEBUG] pss:58 2013-08-01 20:08:00,046: SGE not running (Starting), post_start_script service prerequisites not met afterall,not starting the service yet
[DEBUG] misc:113 2013-08-01 20:08:00,121: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not.
[DEBUG] misc:124 2013-08-01 20:08:00,312: Created bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'.
[ERROR] misc:139 2013-08-01 20:08:00,434: Problem connecting to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc', attempt 1/5
[INFO] misc:455 2013-08-01 20:08:02,976: Saved file 'persistent_data.yaml' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] misc:113 2013-08-01 20:08:03,041: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not.
[DEBUG] master:1601 2013-08-01 20:08:03,042: Saving current instance boot script (/tmp/cm/cm_boot.py) to cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' as 'cm_boot.py'
[INFO] misc:455 2013-08-01 20:08:03,293: Saved file 'cm_boot.py' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] master:1605 2013-08-01 20:08:03,508: Saving CloudMan source (/mnt/cm/cm.tar.gz) to cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' as 'cm.tar.gz'
[INFO] misc:455 2013-08-01 20:08:04,987: Saved file 'cm.tar.gz' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] misc:527 2013-08-01 20:08:04,987: Setting metadata 'revision' for file 'cm.tar.gz' in bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] master:1619 2013-08-01 20:08:05,781: Saving '/mnt/cm/gal_oldami_form2.clusterName' file to cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' as 'gal_oldami_form2.clusterName'
[INFO] misc:455 2013-08-01 20:08:06,027: Saved file 'gal_oldami_form2.clusterName' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] sge:405 2013-08-01 20:08:10,466: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.65     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:08:10,467: S&S: SGE..OK; post_start_script..Unstarted;
[DEBUG] master:1676 2013-08-01 20:08:10,467: Monitor adding service 'post_start_script'
[DEBUG] pss:51 2013-08-01 20:08:10,467: Custom-checking 'post_start_script' service prerequisites
[DEBUG] pss:70 2013-08-01 20:08:10,468: Starting post_start_script service
[DEBUG] pss:74 2013-08-01 20:08:10,468: post_start_script service prerequisites OK (i.e., all other services running), checking if post_start_script was provided...
[DEBUG] pss:91 2013-08-01 20:08:10,535: Cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' found; looking for post start script 'post_start_script'
[DEBUG] misc:440 2013-08-01 20:08:10,856: Failed to get file 'post_start_script' from bucket 'cm-35f4d40b627583b6949bc5de11bcfecc': S3ResponseError: 404 Not Found

NoSuchKeyThe specified key does not exist.post_start_scriptA459857A0CC9D8BFqfo3h0QIZYyRSsubkp1GKCbDWljS5GNMSw0zxCOSrcO/CS5wcMU08G44u5oVdu/G
[DEBUG] pss:102 2013-08-01 20:08:10,857: post_start_script does not exist or could not be downloaded; continuing without running it.
[DEBUG] pss:40 2013-08-01 20:08:10,857: Priming local data variables...
[DEBUG] pss:107 2013-08-01 20:08:10,857: post_start_script service done and marked as 'Shut down'
[DEBUG] ec2:33 2013-08-01 20:08:10,858: Gathering instance ami, attempt 0
[DEBUG] pss:136 2013-08-01 20:08:10,858: Removing post_start_script service from master list of services
[DEBUG] ec2:122 2013-08-01 20:08:10,861: Gathering instance security group, attempt 0
[DEBUG] ec2:181 2013-08-01 20:08:10,862: Gathering instance local hostname, attempt 0
[INFO] misc:455 2013-08-01 20:08:12,333: Saved file 'persistent_data.yaml' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'
[DEBUG] sge:405 2013-08-01 20:08:25,375: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.65     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:08:25,376: S&S: SGE..OK;
[DEBUG] sge:405 2013-08-01 20:08:37,826: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.65     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:08:37,827: S&S: SGE..OK;
[DEBUG] sge:405 2013-08-01 20:08:50,281: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.83     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:08:50,282: S&S: SGE..OK;
[DEBUG] sge:405 2013-08-01 20:09:02,751: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.83     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:09:02,752: S&S: SGE..OK;
[DEBUG] sge:405 2013-08-01 20:09:15,209: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.83     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:09:15,209: S&S: SGE..OK;
[DEBUG] sge:405 2013-08-01 20:09:27,665: qstat: ['all.q@ip-10-179-15-41.ec2.inte BIP   0/0/2          0.91     lx24-amd64    ']
[DEBUG] master:1650 2013-08-01 20:09:27,666: S&S: SGE..OK;

________________________________
From: Ravpreet Setia
Sent: Thursday, August 01, 2013 4:01 PM
To: Dannon Baker
Subject: RE: [galaxy-dev] Cloudman - When will the symlinks be resolved ?

The volumes are not mounting ; I do not see them in /mnt after I ssh in.

Here is the entirety of my log:

  *   19:52:27 - Master starting
  *   19:52:28 - Completed initial cluster configuration. This seems to be a new cluster; waiting to configure the type
  *   19:52:32 - SGE prerequisites OK; starting the service
  *   19:52:39 - Configuring SGE...
  *   19:52:50 - Successfully setup SGE; configuring SGE
  *   19:52:50 - Saved file 'persistent_data.yaml' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075'
  *   19:52:50 - Saved file 'cm_boot.py' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075'
  *   19:52:51 - Saved file 'cm.tar.gz' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075'
  *   19:52:52 - Saved file 'gal_oldami_fromform.clusterName' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075'
  *   19:52:53 - Initializing a 'Galaxy' cluster.
  *   19:52:55 - Retrieved file 'snaps.yaml' from bucket 'gxy-workshop' to 'cm_snaps.yaml'.
  *   19:52:57 - Saved file 'persistent_data.yaml' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075'

________________________________
From: Dannon Baker [dannon.baker@gmail.com]
Sent: Monday, July 29, 2013 10:54 AM
To: Ravpreet Setia
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Cloudman - When will the symlinks be resolved ?

We're working on it but don't have a specific ETA for the release of the new volume.  For now, you can continue to create and run non-migrated clusters using the previous version of Cloudman using this launch form:

https://main.g2.bx.psu.edu/cloudlaunch?ami=ami-da58aab3&bucket_default=gxy-workshop

On Mon, Jul 29, 2013 at 10:49 AM, Ravpreet Setia <Ravpreet.Setia@oicr.on.ca<mailto:Ravpreet.Setia@oicr.on.ca>> wrote:
I am referring to the following bug:

https://bitbucket.org/galaxy/cloudman/issue/6/tool-default-symbolic-links-to...

We are waiting for this bug to resolve before we fix any other issues with our Cloudman scripts. I am wondering when you think this bug will be resolved so I can plan my time. Thanks.

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Cloudman - When will the symlinks be resolved ?

Ravpreet Setia

Ravpreet Setia

Dannon Baker

Ravpreet Setia

Ravpreet Setia

tags

participants (2)