February 2017 - galaxy-dev - lists.galaxyproject.org

Groovy tool for galaxy
by Anshul Tak 23 Aug '17

23 Aug '17

Hi, I have a question regarding galaxy. Can I execute groovy scipts using galaxy? I installed groovy in galaxy docker image and all the dependencies. However, when I try to execute a tool which require groovy engine it says "/export/galaxy-central/database/job_working_directory/000/2/tool_script.sh: line 9: groovy: command not found" So I understood the error that galaxy is not able to find groovy but ideally it should take the groovy engine as it is installed. Can you give me some pointers to solve this issue? -- With Regards, Anshul Tak

4 4

Pulsar - running as real DRMAA user problems
by Poole, Richard 13 May '17

13 May '17

Hey John, So I’ve been happily using Pulsar to send all my Galaxy server jobs to our cluster here at UCL for several months now (I love it!). I am now exploring the ‘run-as-real-user’ option for DRMAA submissions and have run into a problem. The files are correctly staged, correctly chowned, successfully submitted to the queue and the job runs. However, at job end (collection?) fails with the following error message in Pulsar: Exception happened during processing of request from (‘*.*.*.*', 54321) Traceback (most recent call last): File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 1072, in process_request_in_thread self.finish_request(request, client_address) File "/opt/rocks/lib/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/opt/rocks/lib/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 446, in handle BaseHTTPRequestHandler.handle(self) File "/opt/rocks/lib/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 441, in handle_one_request self.wsgi_execute() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 291, in wsgi_execute self.wsgi_start_response) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 39, in __call__ return controller(environ, start_response, **request_args) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 144, in controller_replacement result = self.__execute_request(func, args, req, environ) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 124, in __execute_request result = func(**args) File "/cluster/galaxy/pulsar/pulsar/web/routes.py", line 82, in status return status_dict(manager, job_id) File "/cluster/galaxy/pulsar/pulsar/manager_endpoint_util.py", line 12, in status_dict job_status = manager.get_status(job_id) File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 95, in get_status proxy_status, state_change = self.__proxy_status(job_directory, job_id) File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 115, in __proxy_status proxy_status = self._proxied_manager.get_status(job_id) File "/cluster/galaxy/pulsar/pulsar/managers/queued_external_drmaa_original.py", line 62, in get_status external_status = super(ExternalDrmaaQueueManager, self)._get_status_external(external_id) File "/cluster/galaxy/pulsar/pulsar/managers/base/base_drmaa.py", line 31, in _get_status_external drmaa_state = self.drmaa_session.job_status(external_id) File "/cluster/galaxy/pulsar/pulsar/managers/util/drmaa/__init__.py", line 50, in job_status return self.session.jobStatus(str(external_job_id)) File "build/bdist.linux-x86_64/egg/drmaa/session.py", line 518, in jobStatus c(drmaa_job_ps, jobId, byref(status)) File "build/bdist.linux-x86_64/egg/drmaa/helpers.py", line 299, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "build/bdist.linux-x86_64/egg/drmaa/errors.py", line 151, in error_check raise _ERRORS[code - 1](error_string) InvalidJobException: code 18: The job specified by the 'jobid' does not exist. With this corresponding error from my Galaxy server: galaxy.tools.actions INFO 2016-10-13 18:47:51,851 Handled output (279.421 ms) galaxy.tools.actions INFO 2016-10-13 18:47:52,093 Verified access to datasets (5.271 ms) galaxy.tools.execute DEBUG 2016-10-13 18:47:52,118 Tool [toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4] created job [25008<http://toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4]%20…>] (560.404 ms) galaxy.jobs DEBUG 2016-10-13 18:47:52,579 (25008) Working directory for job is: /Users/galaxy/galaxy-dist/database/job_working_directory/025/25008 galaxy.jobs.handler DEBUG 2016-10-13 18:47:52,591 (25008) Dispatching to pulsar runner galaxy.jobs DEBUG 2016-10-13 18:47:52,677 (25008) Persisting job destination (destination id: hpc_low) galaxy.jobs.runners DEBUG 2016-10-13 18:47:52,681 Job [25008] queued (90.231 ms) galaxy.jobs.handler INFO 2016-10-13 18:47:52,699 (25008) Job dispatched galaxy.tools.deps DEBUG 2016-10-13 18:47:53,138 Building dependency shell command for dependency 'samtools' galaxy.jobs.runners.pulsar INFO 2016-10-13 18:47:53,233 Pulsar job submitted with job_id 25008 galaxy.jobs DEBUG 2016-10-13 18:47:53,257 (25008) Persisting job destination (destination id: hpc_low) galaxy.datatypes.metadata DEBUG 2016-10-13 18:51:03,922 Cleaning up external metadata files galaxy.jobs.runners.pulsar ERROR 2016-10-13 18:51:03,945 failure finishing job 25008 Traceback (most recent call last): File "/Users/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pulsar.py", line 386, in finish_job run_results = client.full_status() File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 132, in full_status return self.raw_check_complete() File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 28, in replacement return func(*args, **kwargs) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 13, in replacement response = func(*args, **kwargs) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 146, in raw_check_complete check_complete_response = self._raw_execute("status", {"job_id": self.job_id}) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 215, in _raw_execute return self.job_manager_interface.execute(command, args, data, input_path, output_path) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/interface.py", line 96, in execute response = self.transport.execute(url, method=method, data=data, input_path=input_path, output_path=output_path) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 34, in execute response = self._url_open(request, data) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 20, in _url_open return urlopen(request, data) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open response = meth(req, response) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response 'http', request, response, code, msg, hdrs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error return self._call_chain(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error I am running 15.10 and Python 2.7.10 on my iMac for the server and the cluster submission node is running Pulsar 0.5.0 and Python 2.7.12 For these tests I run Pulsar in an interactive window so I have not set the sudoers file up, but rather enter sudo password when requested by Pulsar (at the first step of chowning the staging directory). Also have rewrites set up in Galaxy’s pulsar_actions.yml and I am using remote_scp for the file transfers rather than http - although I have also tried switching back to http (as I noticed caching, which I am also testing, does not work with scp transfers) but get an identical set of error messages. As I say, I have no troubles using a regular queued_drmaa manager in pulsar. Any ideas what the problem may be? Cheers, Rich Richard J Poole PhD Wellcome Trust Fellow Department of Cell and Developmental Biology University College London 518 Rockefeller 21 University Street, London WC1E 6DE Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133) https://www.ucl.ac.uk/cdb/academics/poole

2 5

Running jobs as real user and extra_file_path
by Louise-Amélie Schmitt 28 Apr '17

28 Apr '17

Hi everyone, I just wanted to ask how the extra_file_path is handled in case of job running as the real user since the file_path is only writable by the galaxy user. Any clue? Thanks, L-A

3 3

Testing updated NCBI BLAST+ wrappers for version 2.5.0
by Peter Cock 19 Apr '17

19 Apr '17

Hello all, I have updated the NCBI BLAST+ wrappers on the Test Tool Shed, the wrapper is now at v0.2.00: https://testtoolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/ The main changes is this now depends on BLAST+ 2.5.0, and that is available via either BioConda or the Tool Shed: https://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_5_0/ https://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_5_0/ In order for the dependency to work smoothly on both BioConda and the Tool Shed system, we have changed the package name from "blast+" to just "blast". Given the NCBI stopped updated the original "legacy" BLAST some time ago, when combined with the version number this is no longer ambiguous. Jumping from using BLAST+ 2.2.31 to using BLAST+ 2.5.0 required updating lots of the test files for NCBI changes, including dropping the GI numbers in many outputs, expanding the percentage identity field from 2dp to 3dp, and also changing how -parse_deflines works with tabular output. The wrappers (deliberately) do not yet offer any new functionality added in the recent NCBI BLAST+ updates, in particular BLAST XML v2 is not yet available as an output with a datatype in Galaxy. At this point I would welcome feedback from those of you using the BLAST+ wrappers - including if you were able to install this with the dependencies from BioConda or the traditional Tool Shed packages. Once I'm confident that this is all OK, I will update the main Tool Shed (and think about adding new functionality in 2017). Thank you all, Peter

1 1

SLURM configuration problem
by Leonor Palmeira 31 Mar '17

31 Mar '17

Dear all, we have setup a Galaxy instance on a virtual machine, and we want to be able to submit jobs to our HPC system (SLURM). Currently, we do not understand how to define that jobs will be sent to the HPC cluster. We have set : export $DRMAA_LIBRARY_PATH=/var/lib/libdrmaa.so This is our config/job_conf.xml : <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" /> </plugins> <handlers default="handlers"> <handler id="handler0" tags="handlers" /> <handler id="main" /> </handlers> <destinations default="slurm"> <destination id="slurm" runner="drmaa"> <param id="nativeSpecification">-P all_5hrs</param> </destination> </destinations> </job_conf> And the output of "sh run.sh" : galaxy.jobs.manager DEBUG 2017-02-07 15:50:39,962 Starting job handler galaxy.jobs INFO 2017-02-07 15:50:39,962 Handler 'main' will load all configured runner plugins galaxy.jobs.runners.state_handler_factory DEBUG 2017-02-07 15:50:39,971 Loaded 'failure' state handler from module galaxy.jobs.runners.state_handlers.resubmit pulsar.managers.util.drmaa DEBUG 2017-02-07 15:50:39,975 Initializing DRMAA session from thread MainThread Traceback (most recent call last): File "/home/mass/GAL/APP/galaxy/lib/galaxy/webapps/galaxy/buildapp.py", line 55, in paste_app_factory app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs ) File "/home/mass/GAL/APP/galaxy/lib/galaxy/app.py", line 170, in __init__ self.job_manager = manager.JobManager( self ) File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/manager.py", line 23, in __init__ self.job_handler = handler.JobHandler( app ) File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/handler.py", line 32, in __init__ self.dispatcher = DefaultJobDispatcher( app ) File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/handler.py", line 723, in __init__ self.job_runners = self.app.job_config.get_job_runner_plugins( self.app.config.server_name ) File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/__init__.py", line 687, in get_job_runner_plugins rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get( 'kwds', {} ) ) File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/runners/drmaa.py", line 88, in __init__ self.ds = DrmaaSessionFactory().get() File "/usr/local/lib/python2.7/dist-packages/pulsar/managers/util/drmaa/__init__.py", line 31, in get return DrmaaSession(session_constructor, **kwds) File "/usr/local/lib/python2.7/dist-packages/pulsar/managers/util/drmaa/__init__.py", line 49, in __init__ DrmaaSession.session.initialize() File "/usr/local/lib/python2.7/dist-packages/drmaa/session.py", line 257, in initialize py_drmaa_init(contactString) File "/usr/local/lib/python2.7/dist-packages/drmaa/wrappers.py", line 73, in py_drmaa_init return _lib.drmaa_init(contact, error_buffer, sizeof(error_buffer)) File "/usr/local/lib/python2.7/dist-packages/drmaa/errors.py", line 151, in error_check raise _ERRORS[code - 1](error_string) InternalException: code 1: cell directory "/usr/lib/gridengine-drmaa/default" doesn't exist Could anyone point us in the right direction? This would be greatly appreciated. Best regards Leonor -- Leonor Palmeira | PhD Associate Scientist Department of Human Genetics CHU de Liège | Domaine Universitaire du Sart-Tilman 4000 Liège | BELGIQUE Tél: +32-4-366.91.41 Fax: +32-4-366.72.61 e-mail: lpalmeira(a)chu.ulg.ac.be

2 8

BioBlend: getting a list of datasets in a folder
by D K 15 Mar '17

15 Mar '17

I'm trying to get a list of all the datasets in a folder using BioBlend. I don't see a function for doing this in the API documentation. I can use "show_folder" to get some information regarding the contents of a folder, but not the dataset IDs themselves This is as far as I've gotten: s = gi.libraries.show_folder(LibID, folder_id = subfolderID) > print(s) {u'parent_library_id': u'd413a19dec13d11e', u'update_time': > u'2017-02-24T18:10:02.789955', u'description': > u'170124_M02550_0174_000000000-AV8FJ', u'name': > u'170124_M02550_0174_000000000-AV8FJ', u'deleted': False, u'item_count': 2, > u'parent_id': u'Ff1b9846ab84237e7', u'genome_build': u'?', u'model_class': > u'LibraryFolder', u'id': u'F84ffe6fca6c4fbda', u'library_path': [u'2017', > u'170124_M02550_0174_000000000-AV8FJ']} There are two items (item_count) as expected, how do I get the actual dataset IDs? Thanks!

1 2

Newline characters for txt files
by Lawrence, Cris W 14 Mar '17

14 Mar '17

I have observed that when I upload a file with CRLF line endings into Galaxy and then later download it, the endings have changed to UNIX (LF only). I also observed that an uploaded file containing CR only line endings in Galaxy downloads to a file again with LF only line endings. These file types are txt. Just want to confirm - does Galaxy scrub line endings? - Cris

2 1

installation error on venv dependencies.xml
by Linda Bakker 14 Mar '17

14 Mar '17

Hi All, I am having some problems with my tool dependencies to setup a python venv. Dependencies.xml looks like this: When i install my tool from the toolshed, it fails to install the venv dependency with this error message: I am not sure what the problem exactly seems to be. os.py should be a default python package right or do i need to specify it in the dependencies file? Can someone point me in the right direction of what the problem might be. Thanks! with regards , Linda Bakker -- Linda Bakker Bioinformatics Research Assistant Plant Research International, Wageningen UR Building 107 (Radix), room W1.Fa.054 Droevendaalsesteeg 1, 6708 PB, Wageningen, The Netherlands tel: +31-317-480756 email: linda.bakker(a)wur.nl

2 1

SAML Authentication
by Keith Suderman 04 Mar '17

04 Mar '17

Is anyone using SAML for authentication? There is a feature request for SAML support on GitHub ( https://github.com/galaxyproject/galaxy/issues/3146) but it doesn't look like any work has been done. We need to add the ability for our users to authenticate with a SAML identity provider (IdP), in particular with InCommon/EduRoam/EduCause et al. Looking at the documentation there appears to be "the old way" (configuring Apache/Nginx to do the authentication) and "the new way" where Galaxy handles the authentication. Is it correct to assume that to use the new way I should implement an external authenticator, something like galaxy/auth/providers/saml.py and then some sort of magic to get a config/auth_conf.xml working? I just want to make sure I am heading down the correct path before investing too much time. - Keith Keith Suderman Research Associate Department of Computer Science Vassar College, Poughkeepsie NY suderman(a)cs.vassar.edu

3 3

ansible-galaxy-tools and venv
by Matthias Bernt 01 Mar '17

01 Mar '17

Dear galaxy developers, Is it expected that the virtualenv in .venv/ does include the python binary but does not include the python shared libraries? Some more details: I have installed galaxy on CentOS 6.8 which includes the required python 2.7 only via software collections. For the installation of galaxy I have enabled the software collection by calling "source /opt/rh/python27/enable" (which sets PATH, LD_LIBRARY_PATH, MANPATH, XDG_DATA_DIRS, and PKG_CONFIG_PATH). Currently I was trying to run the ansible role ansible-galaxy-tools from https://github.com/galaxyproject/ansible-galaxy-tools which runs "/home/galaxy/galaxy/.venv/bin/python manage_bootstrap_user.py ..." and gives me /home/galaxy/galaxy/.venv/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory. This python call would work if I again "source /opt/rh/python27/enable". But then the venv python binary is used with the library of the system. Intuitively I would expect that the library and binary belong together. Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 2