- galaxy-dev - lists.galaxyproject.org

One small problem for experts, one giant question for me..
by Can Holyavkin 09 Aug '13

09 Aug '13

Hello everyone, Firstly, I am beginner. I am a new Galaxy user that installed the Galaxy locally into my ubuntu (13.04) installed computer. Everything is working perfectly right now. Galaxy is really awesome. To learn the basics, I started the basic tutorial (Galaxy101 https://main.g2.bx.psu.edu/u/aun1/p/galaxy101) that helps too much. This tutorial consist a step for explaning "Compare Two Datasets" function. I did exactly everthing the instructor said however that module become red in History with following error: tool error An error occured with dataset: Unable to finish job (screenshot of error: http://i.imgur.com/xHqhK72.png) I wrote the bug report at end of the mail. Interestingly, when I press "View" button, I get the data that I wanted. However, any metadata is seen. So I can not see the final results on the UCSC browser. I replicate the same module at public server, it didn't gave an error. What might cause this kind of problem? Does reinstalling solve this problem? Would you please help me? Thank you everyone. Traceback (most recent call last): File "/home/can/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 94, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File "/home/can/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1009, in finish self.sa_session.flush() File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush self._flush(objects) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1789, in _flush flush_context.execute() File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute rec.execute(self) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute uow File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj mapper, table, update) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements execute(statement, params) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/home/can/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'UPDATE job SET update_time=?, stdout=?, stderr=? WHERE job.id = ?' ('2013-08-09 09:00:48.464260', 'join (GNU coreutils) 8.20\nCopyright (C) 2012 Free Software Foundation, Inc.\nLisans GPLv3+: GNU GPL s\xc3\xbcr\xc3\xbcm 3 ve sonras\xc4\xb1 <http://gnu.org/licenses/gpl.html>.\nBu \xc3\x96zg\xc3\xbcr bir yaz\xc4\xb1l\xc4\xb1md\xc4\xb1r: de\xc4\x9fi\xc5\x9ftirmek ve da\xc4\x9f\xc4\xb1tmakta \xc3\xb6zg\xc3\xbcrs\xc3\xbcn\xc3\xbcz.\nYasalar\xc4\xb1n izin verdi\xc4\x9fi \xc3\xb6l\xc3\xa7\xc3\xbcde, herhangi bir GARANT\xc4\xb0 sunulmaz.\n\nMike Haertel taraf\xc4\xb1ndan yaz\xc4\xb1ld\xc4\xb1.\n', '', 41) Can Holyavkin PhD Student Istanbul Technical University Molecular Biology and Genetics Dept.

2 2

Galaxy cannot start last [server:handlerxx]
by Rodolfo Aramayo 09 Aug '13

09 Aug '13

Hi, I have a 80 core/1TB RAM Dell server. In order improve Galaxy performance I modified the universe_wsgi.ini file so as to improve the Scaling and Load Balancing as follows: # Use a threadpool for the web server instead of creating a thread for each # request. use_threadpool = True # Number of threads in the web server thread pool. threadpool_workers = 200 ################################################################################ [server:web0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web2] use = egg:Paste#http port = 8092 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web3] use = egg:Paste#http port = 8093 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web4] use = egg:Paste#http port = 8094 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web5] use = egg:Paste#http port = 8095 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web6] use = egg:Paste#http port = 8096 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web7] use = egg:Paste#http port = 8097 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web8] use = egg:Paste#http port = 8098 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web9] use = egg:Paste#http port = 8099 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web10] use = egg:Paste#http port = 8100 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 ################################################################################ [server:manager] use = egg:Paste#http port = 8200 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 ################################################################################ [server:handler0] use = egg:Paste#http port = 8300 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8301 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler2] use = egg:Paste#http port = 8302 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler3] use = egg:Paste#http port = 8303 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler4] use = egg:Paste#http port = 8304 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler5] use = egg:Paste#http port = 8305 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler6] use = egg:Paste#http port = 8306 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler7] use = egg:Paste#http port = 8307 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler8] use = egg:Paste#http port = 8308 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler9] use = egg:Paste#http port = 8309 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler10] use = egg:Paste#http port = 8310 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler11] use = egg:Paste#http port = 8311 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler12] use = egg:Paste#http port = 8312 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler13] use = egg:Paste#http port = 8313 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler14] use = egg:Paste#http port = 8314 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler15] use = egg:Paste#http port = 8315 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler16] use = egg:Paste#http port = 8316 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler17] use = egg:Paste#http port = 8317 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler18] use = egg:Paste#http port = 8318 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 # [server:handler19] # use = egg:Paste#http # port = 8319 # host = 127.0.0.1 # use_threadpool = true # threadpool_workers = 5 # [server:handler20] # use = egg:Paste#http # port = 8320 # host = 127.0.0.1 # use_threadpool = true # threadpool_workers = 5 ################################################################################ job_manager = manager job_handlers = handler0,handler1,,handler2,handler3,handler4,handler5,handler6,handler7,handler8,handler9,handler10,handler11,handler12,handler13,handler14,handler15,handler16,handler17,handler18 ################################################################################ Now the problem I have is that when I start galaxy I cannot get the last [server:handler18] to load correclt I get the following error: galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,236 Enabling 'library_contents' API controller, class: LibraryContentsController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,236 Enabling 'group_users' API controller, class: GroupUsersAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,236 Enabling 'configuration' API controller, class: ConfigurationController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,237 Enabling 'requests' API controller, class: RequestsAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,238 Enabling 'folders' API controller, class: FoldersController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,238 Enabling 'quotas' API controller, class: QuotaAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,239 Enabling 'folder_contents' API controller, class: FolderContentsController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,239 Enabling 'roles' API controller, class: RoleAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,240 Enabling 'request_types' API controller, class: RequestTypeAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,240 Enabling 'users' API controller, class: UserAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,241 Enabling 'samples' API controller, class: SamplesAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,241 Enabling 'forms' API controller, class: FormDefinitionAPIController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,241 Enabling 'permissions' API controller, class: PermissionsController galaxy.web.framework.base DEBUG 2013-08-05 17:08:24,242 Enabling 'workflows' API controller, class: WorkflowsAPIController galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,242 app.config.api_folders: False galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,450 Enabling 'httpexceptions' middleware galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,450 Enabling 'recursive' middleware galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,700 Enabling 'error' middleware galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,701 Enabling 'trans logger' middleware galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,701 Enabling 'x-forwarded-host' middleware galaxy.webapps.galaxy.buildapp DEBUG 2013-08-05 17:08:24,701 Enabling 'Request ID' middleware Starting server in PID 3899. Traceback (most recent call last): File "./scripts/paster.py", line 33, in <module> serve.run() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 1049, in run invoke(command, command_name, options, args[1:]) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 1055, in invoke exit_code = runner.run(args) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 220, in run result = self.command() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 670, in command serve() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 654, in serve server(app) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 292, in server_wrapper **context.local_conf) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 100, in fix_call reraise(*exc_info) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 37, in reraise exec('raise t, e, tb', dict(t=t, e=e, tb=tb)) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 97, in fix_call val = callable(*args, **kw) File "/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py", line 1342, in server_runner serve(wsgi_app, **kwargs) TypeError: serve() got an unexpected keyword argument 'job_manager' Removing PID file handler18.pid If I use activate [server:handler20] for example, this one would act up as well. It is always the last one and in all cases the port used is totally incorrect and was never assigned Am I doing something terribly wrong? I tried to use ports not 'assigned' and tried to add web servers and handlers according to my best understanding of the instructions (http://wiki.galaxyproject.org/Admin/Config/Performance/Web%20Application%20…) Please any suggestions would be most welcome Thanks --Rodolfo

2 4

Galaxy on cloud config issues
by Ulyanov_Anatoly 09 Aug '13

09 Aug '13

Hello, Does anybody use the last Galaxy image? I am running Galaxy image on EC2: Instance ID i-1b2d3773 Image ID (AMI) ami-118bfc78 It looks as those three tools have configuration problems: EBOSS Blast+ meme For all three I am getting a message that looks like that: /mnt/galaxy/tmp/job_working_directory/000/2/galaxy_2.sh: blastp: not found Error invoking command: blastp Is there a way to fix configuration files for those tools? Thank you. Best Regards, Anatoly </pre> This e-mail, including any attachments, is meant only for the intended recipient and may be a confidential communication or a communication privileged by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of this e-mail is strictly prohibited. Please notify the sender immediately of the error by return e-mail and please delete this message from your system. Thank you in advance for your cooperation.

3 8

Re: [galaxy-dev] support pbkdf2 in proftpd 1.3.5rc3
by Benjamin Schuster-Boeckler 09 Aug '13

09 Aug '13

I previously tried to get this to work, too. After a lot of trial-and-error, I gave up because proFTPd always calculated a different string after salting and hashing the password with PBKDF2 than what is stored in the database. I'm sure that it extracted the correct salt and password, but something about the salt&hash algorithm in proFTPd is different from what Galaxy does. I ended up reverting to SHA1 for our internal instance which is behind a firewall so pw hashing isn't as critical. Just my 5c, cheers, Ben -- Benjamin Schuster-Böckler Leadership Fellow for Bioinformatics ☎ +44 1865 617481 ✉ benjamin.schuster-boeckler(a)ludwig.ox.ac.uk Ludwig Institute for Cancer Research University of Oxford Old Road Campus Research Building Oxford OX3 7DQ United Kingdom

1 0

support pbkdf2 in proftpd 1.3.5rc3
by Leon Mei 09 Aug '13

09 Aug '13

Dear galaxy developers, We have tried today to upgrade our proftpd configuration to make uploading for our galaxy users possible again, both for users with "old" as well as "new" style hashed passwords. We upgraded proftpd on the server to 1.3.5rc3 and have the following SQL part in our configuration file based on the post of http://dev.list.galaxyproject.org/ProFTPD-integration-with-Galaxy-td4660295… SQLEngine on SQLLogFile /var/log/proftpd-sql.log SQLBackend postgres SQLConnectInfo galaxy@localhost:5840 galaxyftp <[ourpassword]> SQLAuthTypes SHA1 SHA256 PBKDF2 SQLPasswordPBKDF2 SHA256 1000 24 SQLPasswordUserSalt sql:/GetUserSalt SQLAuthenticate users SQLDefaultUID 108 SQLDefaultGID 116 SQLDefaultHomedir /opt/cloudman/pkg/proftpd/var SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 32) ELSE password END) AS password2,'108','116','/mnt/galaxyData/tmp/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" We have executed the LookupGalaxyUser and GetUserSalt commands manually, and the results look good. Now, old users can login via ftp, but for a new user, the authentication still fails: 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: >>> cmd_check 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'sha1' 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: 'sha1' SQLAuthType handler reports failure 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'pbkdf2' 2013-07-26 13:15:06,993 mod_sql/4.3[31761]: 'pbkdf2' SQLAuthType handler reports failure What are we missing? Thanks! Rob and Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl<https://wiki.nbic.nl/index.php/Next_Generation_Sequencing> Skype: leon_mei Mobile: +31 6 41709231

3 5

Re: [galaxy-dev] Cloudman - When will the symlinks be resolved ?
by Ravpreet Setia 08 Aug '13

08 Aug '13

Here it is: 'pd.yaml' not found at paths.py load! Set paths.P_GALAXY_HOME as '/mnt/galaxyTools/galaxy-central' Python version: (2, 6) Image configuration suports: {'apps': ['cloudman', 'galaxy']} [DEBUG] app:54 2013-08-01 20:07:40,138: Initializing app [DEBUG] app:55 2013-08-01 20:07:40,138: Running on 'ec2' type of cloud. [DEBUG] app:72 2013-08-01 20:07:40,138: Getting pd.yaml [DEBUG] ec2:275 2013-08-01 20:07:40,138: No S3 Connection, creating a new one. [DEBUG] ec2:278 2013-08-01 20:07:40,139: Got boto S3 connection. [DEBUG] misc:113 2013-08-01 20:07:40,229: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not. [DEBUG] misc:444 2013-08-01 20:07:40,229: Bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' does not exist, did not get remote file 'persistent_data.yaml' [INFO] app:78 2013-08-01 20:07:40,229: Master starting [DEBUG] master:70 2013-08-01 20:07:40,229: Initializing console manager - cluster start time: 2013-08-01 20:07:40.229549 [DEBUG] connection:648 2013-08-01 20:07:40,231: Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '1.7.2', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL. See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US'] [DEBUG] connection:494 2013-08-01 20:07:40,263: Open OK! known_hosts [ip-10-179-15-41.ec2.internal:5672] [DEBUG] channel:70 2013-08-01 20:07:40,263: using channel_id: 1 [DEBUG] channel:484 2013-08-01 20:07:40,264: Channel open [DEBUG] comm:35 2013-08-01 20:07:40,266: Successfully established AMQP connection [DEBUG] master:510 2013-08-01 20:07:40,266: Trying to discover any worker instances associated with this cluster... [DEBUG] ec2:246 2013-08-01 20:07:40,267: Establishing boto EC2 connection [DEBUG] ec2:103 2013-08-01 20:07:40,267: Gathering instance zone, attempt 0 [DEBUG] ec2:108 2013-08-01 20:07:40,270: Instance zone is 'us-east-1d' [DEBUG] ec2:263 2013-08-01 20:07:40,516: Got boto EC2 connection for region 'us-east-1' [DEBUG] ec2:66 2013-08-01 20:07:40,574: Gathering instance id, attempt 0 [DEBUG] ec2:71 2013-08-01 20:07:40,575: Instance ID is 'i-7b386714' [DEBUG] ec2:295 2013-08-01 20:07:40,639: Adding tag 'clusterName:gal_oldami_form2' to resource 'i-7b386714' [DEBUG] ec2:295 2013-08-01 20:07:41,066: Adding tag 'role:master' to resource 'i-7b386714' [DEBUG] master:174 2013-08-01 20:07:41,223: ud at manager start: {'access_key': 'AKIAIUSCRANAJESP3GFQ', 'cloudman_home': '/mnt/cm', 'cluster_name': 'gal_oldami_form2', 'bucket_default': 'gxy-workshop', 'boot_script_name': 'cm_boot.py', 'bucket_cluster': 'cm-35f4d40b627583b6949bc5de11bcfecc', 'boot_script_path': '/tmp/cm', 'role': 'master', 'secret_key': 'EV12T4PMGSbM1IZmS7ZUJVi9nCvMncnrxXvytacJ', 'password': 'google'} [DEBUG] pss:25 2013-08-01 20:07:41,224: Configured PSS as master [DEBUG] master:204 2013-08-01 20:07:41,225: Cheking for and adding any preconfigured services [DEBUG] master:529 2013-08-01 20:07:41,225: Trying to discover any volumes attached to this instance... [DEBUG] base:55 2013-08-01 20:07:41,236: Enabling 'root' controller, class: CM [DEBUG] buildapp:80 2013-08-01 20:07:41,239: Enabling 'httpexceptions' middleware [DEBUG] buildapp:86 2013-08-01 20:07:41,469: Enabling 'recursive' middleware [DEBUG] buildapp:106 2013-08-01 20:07:41,719: Enabling 'print debug' middleware [DEBUG] buildapp:120 2013-08-01 20:07:41,734: Enabling 'error' middleware [DEBUG] buildapp:130 2013-08-01 20:07:41,735: Enabling 'config' middleware [DEBUG] buildapp:134 2013-08-01 20:07:41,736: Enabling 'x-forwarded-host' middleware [DEBUG] master:535 2013-08-01 20:07:41,737: Attached volumes: [Volume:vol-fe5b7abe] [INFO] master:198 2013-08-01 20:07:41,758: Completed initial cluster configuration. This seems to be a new cluster; waiting to configure the type [DEBUG] master:1627 2013-08-01 20:07:41,758: Monitor started; manager started Starting server in PID 1252. serving on 0.0.0.0:42284 view at http://127.0.0.1:42284 [DEBUG] master:1247 2013-08-01 20:07:45,657: Checking for new version of CloudMan [DEBUG] misc:512 2013-08-01 20:07:45,657: Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] misc:113 2013-08-01 20:07:45,740: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not. [DEBUG] misc:512 2013-08-01 20:07:45,740: Getting metadata 'revision' for file 'cm.tar.gz' from bucket 'gxy-workshop' [DEBUG] master:1676 2013-08-01 20:07:45,763: Monitor adding service 'SGE' [DEBUG] __init__:31 2013-08-01 20:07:45,763: Trying to add service 'SGE' [INFO] __init__:50 2013-08-01 20:07:45,763: SGE prerequisites OK; starting the service [DEBUG] sge:62 2013-08-01 20:07:45,763: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] __init__:42 2013-08-01 20:07:45,793: 'sge' daemon is NOT running any more (expected pid: '2521'). [DEBUG] sge:77 2013-08-01 20:07:45,793: Cleaning '/opt/sge' directory. [DEBUG] master:1251 2013-08-01 20:07:45,804: Revision number for user's CloudMan: 'None'; revision number for default CloudMan: 'None' [DEBUG] ec2:48 2013-08-01 20:07:45,805: Gathering instance type, attempt 0 [DEBUG] sge:83 2013-08-01 20:07:46,195: Unpacking SGE to '/opt/sge'. [DEBUG] ec2:218 2013-08-01 20:07:46,267: Gathering instance public hostname, attempt 0 [INFO] master:817 2013-08-01 20:07:50,102: Initializing a 'Galaxy' cluster. [INFO] misc:438 2013-08-01 20:07:51,212: Retrieved file 'snaps.yaml' from bucket 'gxy-workshop' to 'cm_snaps.yaml'. Error - : 'static_filesystems' [INFO] sge:97 2013-08-01 20:07:51,551: Configuring SGE... [DEBUG] ec2:162 2013-08-01 20:07:51,552: Gathering instance private IP, attempt 0 [DEBUG] sge:105 2013-08-01 20:07:51,554: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:119 2013-08-01 20:07:51,554: Setting up SGE. [DEBUG] misc:577 2013-08-01 20:07:51,567: Modified /opt/sge/util/arch [DEBUG] misc:577 2013-08-01 20:07:51,573: Successfully chmod /opt/sge/util/arch [DEBUG] misc:577 2013-08-01 20:07:51,581: 'sed -i.bak '/^127.0.1./s/^/# (Commented by CloudMan) /' /etc/hosts' command OK [DEBUG] ec2:142 2013-08-01 20:07:57,465: Gathering instance public keys (i.e., key pairs), attempt 0 [DEBUG] ec2:148 2013-08-01 20:07:57,468: Got key pair: 'rav_kp2' [DEBUG] misc:577 2013-08-01 20:07:59,992: Successfully set up SGE [INFO] sge:122 2013-08-01 20:07:59,993: Successfully setup SGE; configuring SGE [DEBUG] sge:123 2013-08-01 20:07:59,993: Adding parallel environments [DEBUG] misc:577 2013-08-01 20:08:00,009: 'cd /opt/sge; ./bin/lx24-amd64/qconf -Ap /tmp/SMP_PE' command OK [DEBUG] misc:577 2013-08-01 20:08:00,026: 'cd /opt/sge; ./bin/lx24-amd64/qconf -Ap /tmp/MPI_PE' command OK [DEBUG] sge:130 2013-08-01 20:08:00,026: Creating queue 'all.q' [DEBUG] sge:135 2013-08-01 20:08:00,027: Created SGE all.q template as file '/opt/sge/all.q.conf' [DEBUG] misc:577 2013-08-01 20:08:00,044: Successfully modified all.q [DEBUG] sge:137 2013-08-01 20:08:00,045: Configuring users' SGE profiles [DEBUG] master:1676 2013-08-01 20:08:00,045: Monitor adding service 'post_start_script' [DEBUG] pss:51 2013-08-01 20:08:00,045: Custom-checking 'post_start_script' service prerequisites [DEBUG] pss:58 2013-08-01 20:08:00,046: SGE not running (Starting), post_start_script service prerequisites not met afterall,not starting the service yet [DEBUG] misc:113 2013-08-01 20:08:00,121: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not. [DEBUG] misc:124 2013-08-01 20:08:00,312: Created bucket 'cm-35f4d40b627583b6949bc5de11bcfecc'. [ERROR] misc:139 2013-08-01 20:08:00,434: Problem connecting to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc', attempt 1/5 [INFO] misc:455 2013-08-01 20:08:02,976: Saved file 'persistent_data.yaml' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] misc:113 2013-08-01 20:08:03,041: Checking if bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' exists... it does not. [DEBUG] master:1601 2013-08-01 20:08:03,042: Saving current instance boot script (/tmp/cm/cm_boot.py) to cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' as 'cm_boot.py' [INFO] misc:455 2013-08-01 20:08:03,293: Saved file 'cm_boot.py' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] master:1605 2013-08-01 20:08:03,508: Saving CloudMan source (/mnt/cm/cm.tar.gz) to cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' as 'cm.tar.gz' [INFO] misc:455 2013-08-01 20:08:04,987: Saved file 'cm.tar.gz' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] misc:527 2013-08-01 20:08:04,987: Setting metadata 'revision' for file 'cm.tar.gz' in bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] master:1619 2013-08-01 20:08:05,781: Saving '/mnt/cm/gal_oldami_form2.clusterName' file to cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' as 'gal_oldami_form2.clusterName' [INFO] misc:455 2013-08-01 20:08:06,027: Saved file 'gal_oldami_form2.clusterName' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] sge:405 2013-08-01 20:08:10,466: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.65 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:08:10,467: S&S: SGE..OK; post_start_script..Unstarted; [DEBUG] master:1676 2013-08-01 20:08:10,467: Monitor adding service 'post_start_script' [DEBUG] pss:51 2013-08-01 20:08:10,467: Custom-checking 'post_start_script' service prerequisites [DEBUG] pss:70 2013-08-01 20:08:10,468: Starting post_start_script service [DEBUG] pss:74 2013-08-01 20:08:10,468: post_start_script service prerequisites OK (i.e., all other services running), checking if post_start_script was provided... [DEBUG] pss:91 2013-08-01 20:08:10,535: Cluster bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' found; looking for post start script 'post_start_script' [DEBUG] misc:440 2013-08-01 20:08:10,856: Failed to get file 'post_start_script' from bucket 'cm-35f4d40b627583b6949bc5de11bcfecc': S3ResponseError: 404 Not Found NoSuchKeyThe specified key does not exist.post_start_scriptA459857A0CC9D8BFqfo3h0QIZYyRSsubkp1GKCbDWljS5GNMSw0zxCOSrcO/CS5wcMU08G44u5oVdu/G [DEBUG] pss:102 2013-08-01 20:08:10,857: post_start_script does not exist or could not be downloaded; continuing without running it. [DEBUG] pss:40 2013-08-01 20:08:10,857: Priming local data variables... [DEBUG] pss:107 2013-08-01 20:08:10,857: post_start_script service done and marked as 'Shut down' [DEBUG] ec2:33 2013-08-01 20:08:10,858: Gathering instance ami, attempt 0 [DEBUG] pss:136 2013-08-01 20:08:10,858: Removing post_start_script service from master list of services [DEBUG] ec2:122 2013-08-01 20:08:10,861: Gathering instance security group, attempt 0 [DEBUG] ec2:181 2013-08-01 20:08:10,862: Gathering instance local hostname, attempt 0 [INFO] misc:455 2013-08-01 20:08:12,333: Saved file 'persistent_data.yaml' to bucket 'cm-35f4d40b627583b6949bc5de11bcfecc' [DEBUG] sge:405 2013-08-01 20:08:25,375: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.65 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:08:25,376: S&S: SGE..OK; [DEBUG] sge:405 2013-08-01 20:08:37,826: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.65 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:08:37,827: S&S: SGE..OK; [DEBUG] sge:405 2013-08-01 20:08:50,281: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.83 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:08:50,282: S&S: SGE..OK; [DEBUG] sge:405 2013-08-01 20:09:02,751: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.83 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:09:02,752: S&S: SGE..OK; [DEBUG] sge:405 2013-08-01 20:09:15,209: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.83 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:09:15,209: S&S: SGE..OK; [DEBUG] sge:405 2013-08-01 20:09:27,665: qstat: ['all.q(a)ip-10-179-15-41.ec2.inte BIP 0/0/2 0.91 lx24-amd64 '] [DEBUG] master:1650 2013-08-01 20:09:27,666: S&S: SGE..OK; ________________________________ From: Ravpreet Setia Sent: Thursday, August 01, 2013 4:01 PM To: Dannon Baker Subject: RE: [galaxy-dev] Cloudman - When will the symlinks be resolved ? The volumes are not mounting ; I do not see them in /mnt after I ssh in. Here is the entirety of my log: * 19:52:27 - Master starting * 19:52:28 - Completed initial cluster configuration. This seems to be a new cluster; waiting to configure the type * 19:52:32 - SGE prerequisites OK; starting the service * 19:52:39 - Configuring SGE... * 19:52:50 - Successfully setup SGE; configuring SGE * 19:52:50 - Saved file 'persistent_data.yaml' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075' * 19:52:50 - Saved file 'cm_boot.py' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075' * 19:52:51 - Saved file 'cm.tar.gz' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075' * 19:52:52 - Saved file 'gal_oldami_fromform.clusterName' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075' * 19:52:53 - Initializing a 'Galaxy' cluster. * 19:52:55 - Retrieved file 'snaps.yaml' from bucket 'gxy-workshop' to 'cm_snaps.yaml'. * 19:52:57 - Saved file 'persistent_data.yaml' to bucket 'cm-e0d3b778388b8c2eebfbc397f8757075' ________________________________ From: Dannon Baker [dannon.baker(a)gmail.com] Sent: Monday, July 29, 2013 10:54 AM To: Ravpreet Setia Cc: galaxy-dev(a)lists.bx.psu.edu Subject: Re: [galaxy-dev] Cloudman - When will the symlinks be resolved ? We're working on it but don't have a specific ETA for the release of the new volume. For now, you can continue to create and run non-migrated clusters using the previous version of Cloudman using this launch form: https://main.g2.bx.psu.edu/cloudlaunch?ami=ami-da58aab3&bucket_default=gxy-… On Mon, Jul 29, 2013 at 10:49 AM, Ravpreet Setia <Ravpreet.Setia(a)oicr.on.ca<mailto:Ravpreet.Setia@oicr.on.ca>> wrote: I am referring to the following bug: https://bitbucket.org/galaxy/cloudman/issue/6/tool-default-symbolic-links-t… We are waiting for this bug to resolve before we fix any other issues with our Cloudman scripts. I am wondering when you think this bug will be resolved so I can plan my time. Thanks. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

2 4

job runner upload file
by Jasper Jan Koehorst 08 Aug '13

08 Aug '13

Hello, I have a very simple jobrunner script that allows only 1 job to be run as this galaxy is used for one thing in particular which consumes almost all resources. To do this I set the following: <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="1"/> </plugins> This works, but is it possible to create a second worker that is only allowed to be used for uploading files? So people can upload their data already and only enter the queue for the big analysis? Otherwise if they turn their computer off in the evening the upload is gone...

2 1

ActionInputError undefined
by Jim Johnson 08 Aug '13

08 Aug '13

I was using the quotas api to update a quota amount and got the error: galaxy.web.framework ERROR 2013-07-30 15:40:59,149 Uncaught exception in exposed API method: Traceback (most recent call last): File "/Users/jj/gxt/gxt/lib/galaxy/web/framework/__init__.py", line 189, in decorator rval = func( self, trans, *args, **kwargs) File "/Users/jj/gxt/gxt/lib/galaxy/web/framework/__init__.py", line 221, in decorator return func( self, trans, *args, **kwargs ) File "/Users/jj/gxt/gxt/lib/galaxy/webapps/galaxy/api/quotas.py", line 108, in update except ActionInputError, e: NameError: global name 'ActionInputError' is not defined Was ActionInputError meant to be a class in lib/galaxy/exceptions/__init__.py ? diff -r 951e853b0bcd lib/galaxy/exceptions/__init__.py --- a/lib/galaxy/exceptions/__init__.py Tue Jul 30 12:54:02 2013 -0400 +++ b/lib/galaxy/exceptions/__init__.py Tue Jul 30 16:38:50 2013 -0500 @@ -21,6 +21,9 @@ class ItemOwnershipException( MessageException ): pass +class ActionInputError( MessageException ): + pass + class ObjectNotFound( Exception ): """ Accessed object was not found """ pass -- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota

2 1

(no subject)
by Vipin TS 08 Aug '13

08 Aug '13

Hello dev-team, I have updated code repository of our production instance <https://galaxy.cbio.mskcc.org> to the recent galaxy-dist (10216:ebe87051fadf), now the instance page it is not rendering in a proper way. The error message I can see from the log file as follows: ------------------------------------------ x.x.x.x - - [08/Aug/2013:15:00:49 -0400] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" x.x.x.x - - [08/Aug/2013:15:00:50 -0400] "GET /root/tool_menu HTTP/1.1" 404 - "https://galaxy.cbio.mskcc.org/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" x.x.x.x - - [08/Aug/2013:15:00:50 -0400] "GET /history HTTP/1.1" 200 - " https://galaxy.cbio.mskcc.org/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" x.x.x.x - - [08/Aug/2013:15:00:50 -0400] "GET /history HTTP/1.1" 500 - " https://galaxy.cbio.mskcc.org/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 42042) Traceback (most recent call last): File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 287, in wsgi_execute self.wsgi_start_response) File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/urlmap.py", line 203, in __call__ return app(environ, start_response) File "/codebase/lib/galaxy/web/framework/middleware/request_id.py", line 12, in __call__ return self.app( environ, start_response ) File "/codebase/lib/galaxy/web/framework/middleware/xforwardedhost.py", line 21, in __call__ return self.app( environ, start_response ) File "/codebase/lib/galaxy/web/framework/middleware/translogger.py", line 68, in __call__ return self.application(environ, replacement_start_response) File "/codebase/lib/galaxy/web/framework/middleware/error.py", line 159, in __call__ exc_info) File "/codebase/lib/galaxy/web/framework/middleware/translogger.py", line 67, in replacement_start_response return start_response( status, headers, exc_info ) File "/codebase/lib/galaxy/web/framework/middleware/error.py", line 149, in __call__ app_iter = self.application(environ, sr_checker) File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py", line 84, in __call__ return self.application(environ, start_response) File "/codebase/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py", line 633, in __call__ return self.application(environ, start_response) File "/codebase/lib/galaxy/web/framework/base.py", line 132, in __call__ return self.handle_request( environ, start_response ) File "/codebase/lib/galaxy/web/framework/base.py", line 199, in handle_request return body( environ, start_response ) File "/codebase/lib/galaxy/web/framework/__init__.py", line 985, in render template.render_context( context ) File "/codebase/eggs/Mako-0.4.1-py2.6.egg/mako/template.py", line 319, in render_context **kwargs) File "/codebase/eggs/Mako-0.4.1-py2.6.egg/mako/runtime.py", line 692, in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File "/codebase/eggs/Mako-0.4.1-py2.6.egg/mako/runtime.py", line 718, in _exec_template callable_(context, *args, **kwargs) File "/codebase/database/compiled_templates/root/history.mako.py", line 73, in render_body if bool( [ data for data in history.active_datasets if data.state in ['running', 'queued', '', None ] ] ): AttributeError: 'Undefined' object has no attribute 'active_datasets' ---------------------------------------- Could you point me where I have to look to fix this issue. thanks in advance, --/Vipin

1 0

qa_url = slyfox?
by David Hoover 08 Aug '13

08 Aug '13

Strangely, I had never noticed this before. On some tools, there is a link at the top of the tool page that attempts to connect to http://slyfox.bx.psu.edu:8080/, which it can't. This URL is set via the qa_url directive in the universe_wsgi.ini file. What is this URL, and what should it be? David Hoover Helix Systems Staff

2 1