- galaxy-dev - lists.galaxyproject.org

Using Mesos to Enable distributed computing under Galaxy?
by Kyle Ellrott 17 Jun '14

17 Jun '14

I think one of the aspects where Galaxy is a bit soft is the ability to do distributed tasks. The current system of split/replicate/merge tasks based on file type is a bit limited and hard for tool developers to expand upon. Distributed computing is a non-trival thing to implement and I think it would be a better use of our time to use an already existing framework. And it would also mean one less API for tool writers to have to develop for. I was wondering if anybody has looked at Mesos ( http://mesos.apache.org/) You can see an overview of the Mesos architecture at https://github.com/apache/mesos/blob/master/docs/Mesos-Architecture.md The important thing about Mesos is that it provides an API for C/C++, Java/Scala and Python to write distributed frameworks. There are already implementations of frameworks for common parallel programming systems such as: - Hadoop (https://github.com/mesos/hadoop) - MPI ( https://github.com/apache/mesos/blob/master/docs/Running-torque-or-mpi-on-m… ) - Spark (http://spark-project.org) And you can find example Python framework at https://github.com/apache/mesos/tree/master/src/examples/python Integration with Galaxy would have three parts: 1) Add a system config variable to Galaxy called 'MESOS_URL' that is then passed to tool wrappers and allows them to contact the local mesos infrastructure (assuming the system has been configured) or pass a null if the system isn't available. 2) Write a tool runner that works as a mesos framework to executes single cpu jobs on the distributed system. 3) For instances where mesos is not available at a system wide level (say they only have access to an SGE based cluster), but the user wants to run distributed jobs, write a wrapper that can create a mesos cluster using the existing queueing system. For example, right now I run a Mesos system under the SGE queue system. I'm curious to see what other people think. Kyle

5 8

Can't startup the galaxy-central instance!
by xlwang 16 Jun '14

16 Jun '14

hi, I downloaded and installed the galaxy-central instance because I want to use toolshed. But I can't startup the instance, The following error occurred： Fetched http://eggs.galaxyproject.org/WebError/WebError-0.8a-py2.7.egg Fetched http://eggs.galaxyproject.org/twill/twill-0.9-py2.7.egg Fetch successful. Traceback (most recent call last): File "/export/galaxy-central/lib/galaxy/webapps/galaxy/buildapp.py", line 38, in app_factory from galaxy.app import UniverseApplication File "/export/galaxy-central/lib/galaxy/app.py", line 21, in <module> from galaxy.queue_worker import GalaxyQueueWorker File "/export/galaxy-central/lib/galaxy/queue_worker.py", line 10, in <module> import galaxy.queues File "/export/galaxy-central/lib/galaxy/queues.py", line 17, in <module> from kombu import Exchange, Queue File "/export/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/__init__.py", line 67, in __getattr__ module = __import__(object_origins[name], None, None, [name]) File "/export/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/entity.py", line 10, in <module> from .abstract import MaybeChannelBound File "/export/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/abstract.py", line 12, in <module> from .connection import maybe_channel File "/export/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py", line 24, in <module> from kombu import exceptions File "/export/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/exceptions.py", line 12, in <module> from amqp import ChannelError, ConnectionError, ResourceError File "/export/galaxy-central/eggs/amqp-1.4.5-py2.7.egg/amqp/__init__.py", line 32, in <module> from .basic_message import Message # noqa File "/export/galaxy-central/eggs/amqp-1.4.5-py2.7.egg/amqp/basic_message.py", line 19, in <module> from .serialization import GenericContent File "/export/galaxy-central/eggs/amqp-1.4.5-py2.7.egg/amqp/serialization.py", line 33, in <module> from .five import int_types, long_t, string, string_t, items File "/export/galaxy-central/eggs/amqp-1.4.5-py2.7.egg/amqp/five.py", line 171, in <module> librt = ctypes.CDLL('librt.so.1', use_errno=True) TypeError: __init__() got an unexpected keyword argument 'use_errno' Why this happened?

1 0

troubleshooting Galaxy with LSF
by I Kozin 16 Jun '14

16 Jun '14

Hello, This is largely a repost from the biostar forum following the suggestion there to post here. I'm doing my first steps in setting up a Galaxy server with an LSF job scheduler. Recently LSF started supporting DRMAA again so I decided to give it a go. I have two setups. The one that works is a stand along server (OpenSuse 12.1, python 2.7.2, LSF 9.1.2). By "works" I mean that when I login into Galaxy using a browser and upload a file, a job gets submitted and run and everything seems fine. The second setup does not work (RH 6.4, python 2.6.6, LSF 9.1.2). It's a server running Galaxy which is meant to submit jobs to an LSF cluster. When I similarly pick and download a file I get Job <72266> is submitted to queue <short>. ./run.sh: line 79: 99087 Segmentation fault python ./scripts/paster.py serve universe_wsgi.ini $@ For the moment, I'm not bothered with the full server setup, I'm just testing whether Galaxy works with LSF and therefore run ./run.sh as a user. The job configuration job_conf.xml is identical in both cases: <?xml version="1.0"?> <job_conf> <plugins> <plugin id="lsf" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner"> <param id="drmaa_library_path">/opt/gridware/lsf/9.1/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so</param> </plugin> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="lsf_default"> <destination id="lsf_default" runner="lsf"> <param id="nativeSpecification">-W 24:00</param> </destination> </destinations> </job_conf> run.sh is only changed to allow remote access. Most recently I tried replacing python with 2.7.5 to no avail. Still the same kind of error. I also updated Galaxy. Any hints would be much appreciated. Thank you

4 11

Missing navigation pane
by Stephen E 13 Jun '14

13 Jun '14

I installed a local instance of galaxy approximately a month ago (build 2014.04.14). I was having an issue with the Data Manager to fetch reference genomes and after checking to see if there was a newer version of galaxy, I decided to update and see if that fixed my problem. I ran "hg pull" and then "hg update latest_2014.06.02". I tried to rerun galaxy but was instructed to run manage_db.sh so I did (sh manage_db.sh upgrade). When I ran galaxy after all this, it started but when opened in a web browser, the navagation pane at the top is missing (i.e. Analyze Data, Workflows, User, etc). There is a blue bar but nothing is on it. How do I get the missing navigation pane back? I can't do a lot of things without it (i.e. check help or change user settings). I can still get to the administrator section but only by appending "/admin" to the url. I need to know how to fix the new version or how to succesfully revert to a previous version without anything else breaking.

4 5

Tools for managing installed toolshed tools?
by Eric Rasche 13 Jun '14

13 Jun '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Asked this on IRC and was directed to send my request here in case anyone in the community knows: I'm looking to find out if anyone has already written tools to manage installed tools from the toolshed. E.g. something along the lines of apt-get for galaxy, where apt-get update would "update toolshed status", apt-get upgrade to upgrade all tools, and apt-get install/remove would be available for removing and adding tools. If no one has this in the works, it has been added to the hackathon project list Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 <tel:4046922048> esr(a)tamu.edu <mailto:esr@tamu.edu> -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJTmyaxAAoJEMqDXdrsMcpVVYcQAIXAtmeWvH0XnIIO0bxEaym1 1EeWTcBvH3UzwpQ9ucHEegee3KN4E4FZ71yvfKQiFXTm5mKShxQ8ArlSxs08Q92K VAAYgySDgRD5QTbhjxmFGDu/18SOu0Br2vSNLCNIaiygRzIldjCMhKve0XAfY6qJ MQrb4/fiAW5skWhI51n2aSFOP2VENjPgJ/3KDfX6yRCYnvxNjcSQcmWo3z6j8AZI vEyqHDJOCWzzY1XE5oqYUC6RK80OPA9JLcwG7YHKRZgQ8uOONThfBBBPjhV++HS4 X7ZPPLd63ja6U/frNNSkCrBNlmN0xhzf1gxZygMbL72rdu/7fq4bqoO0aSpWX3/O Q9NaX891zRDExOAIGx1dRPQ55XgqOJZYDKru/ykwbbN6YZvh3wNnrLia8Y/Ilho5 zvqiBOh5PwK1fj3RPkiWgVY95yk5+MseXA469xi1QU3yUt8PVvSNInPbq0QrLNZo ZFkJ1U7UYsS6psUmnFdNBGM1NWUyf6KCW5EiFxxIcZ2C6pmKDx5jcRVinBR3GtLm jH+fK2vinqxn07x8h7Dq3NeBeqlB++obcsX74xaic9DwZKqiNu6oI9eGNv0cPI0Y OV1ZPMI/8kJZjZMNvpkMIAi+SaGvvM46kraaszqijmYYTa+jhw7Nenb40gkAV8Ud THpnh3a1IfVu76prCa7g =TnpX -----END PGP SIGNATURE-----

1 0

multiple file selection for from_file
by Jun Fan 13 Jun '14

13 Jun '14

Hi all I gather that it is NOT allowed to assign multiple file to from_file attribute for options in select param as the expect value of from_file attribute is the name of A file. But would it be nice to allow this feature? For example, there are two loc file A and B, when I want to get the options from both files, instead of adding an extra file containing all options from two files, it will be more maintainable to use something like from_file = "A.loc, B.loc", isn't it? BTW, could anyone help me on this http://dev.list.galaxyproject.org/jobs-don-t-get-killed-when-histories-dele… Best regards! Jun

2 1

Re: [galaxy-dev] galaxy-dev Digest, Vol 96, Issue 14
by xlwang 13 Jun '14

13 Jun '14

Hello, when i run my galaxy instance type: $ sh run.sh I get some error like this: Traceback (most recent call last): File "/export/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 38, in app_factory from galaxy.app import UniverseApplication File "/export/galaxy-dist/lib/galaxy/app.py", line 21, in <module> from galaxy.queue_worker import GalaxyQueueWorker File "/export/galaxy-dist/lib/galaxy/queue_worker.py", line 10, in <module> import galaxy.queues File "/export/galaxy-dist/lib/galaxy/queues.py", line 17, in <module> from kombu import Exchange, Queue File "/export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/__init__.py", line 67, in __getattr__ module = __import__(object_origins[name], None, None, [name]) File "/export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/entity.py", line 10, in <module> from .abstract import MaybeChannelBound File "/export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/abstract.py", line 12, in <module> from .connection import maybe_channel File "/export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py", line 24, in <module> from kombu import exceptions File "/export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/exceptions.py", line 12, in <module> from amqp import ChannelError, ConnectionError, ResourceError File "/export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/__init__.py", line 32, in <module> from .basic_message import Message # noqa File "/export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/basic_message.py", line 19, in <module> from .serialization import GenericContent File "/export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/serialization.py", line 33, in <module> from .five import int_types, long_t, string, string_t, items File "/export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/five.py", line 171, in <module> librt = ctypes.CDLL('librt.so.1', use_errno=True) TypeError: __init__() got an unexpected keyword argument 'use_errno' I have no idea why this happend. How can I fix it?

2 1

Browsing toolsheds - HTTP(s) and firewalls
by Eric Rasche 12 Jun '14

12 Jun '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Today I tried to access my org's toolshed, only to realise that it wouldn't load as it was behind the firewall. Since I'm not in the mood to connect my VPN, I thought I'd posit the following: Perhaps the galaxy server should make the HTTP requests to the toolshed instead of the user's browser. This would solve the existing, frustrating issue of most major browsers refusing to load the main toolshed if you're running HTTPS (as galaxy would request over HTTP then serve over HTTPS, acting as a proxy), and it would solve the issue of not being able to load toolsheds which are behind firewalls. Thoughts? Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAEBAgAGBQJTmhWZAAoJEMqDXdrsMcpV12wQAKMfVfyFoWjLpGV+102wSCLY 1TgSEBc8YI2mvJ59z1r6k2EobiBKnvWHYfi2rSkkmzAthn0lke56vSGLMmsjJ/k2 /XT844OmL4ybvL5OyseacZRNpThgeT7wz/KT/3FXtaBP6ssqqAgx2S3meqgzm7yQ T7F08JXzDv/0v+wnqRjWzupbQMpZdorMAc3P+A37e0mYFNKru/Nq1zOJEnYko+um tw3K+lgnhOHidM3s3XOFCsfqn5FizkIpduzQIbl7n4dILa5hGSkJm6KLqyykhR+C RNcBljJeULyOB/UDBvFE+hEqv/IqpqUccphwUHwEQoEnVBtl08MOeuEm50Epp6QT GopzYZ5fBgt38IaAlUV3tVHMGr+NjwUVWrIO8nTnlLoSfcDT3NsQuKlSbEN1JJ6z GjS+b6ZSjng7KF+rgTpU3hu/UGGT8tAzAlxK8ipRvZEFN6TkjSkcdSy9jSQrjYi/ MpTl//cpScPc6G0b/OMl1CbD7VJ6N2oPYx8m47b3dbfF5bUppugKYiPcI6Htx62z 4+FU4MrLkvUcj4O1t1hninRp7DfMag0kbLtHcdUmd1DU0h4TQRYQu+QGaBX9B53Q SU2FzCJXQI2skFC+vMPUNSoIYEKkYPRlBvkKQiMlBHMPA2jcHrBLY+VbLUd7AGJU lAAUgMNh5MXvvr6HM56m =cdqI -----END PGP SIGNATURE-----

1 0

Error with fastx reverse compliment
by Umd Bioinformatics 12 Jun '14

12 Jun '14

Hello, I'm getting an error with fastx tools on a local install of galaxy. From this error I can't tell if it's a problem with the file, database, or tool. > An error occurred with this dataset: zcat: /Volumes/Jose-Drobo/galaxy-dist/database/files/002/dataset_2875.dat.Z: No such file or directory fastx_reverse_complement: Premature End-Of-File (filename ='-') Ian

1 0

jobs submitted to a cluster
by Shrum, Donald C 12 Jun '14

12 Jun '14

I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center

3 5