Groovy tool for galaxy
by Anshul Tak
Hi,
I have a question regarding galaxy. Can I execute groovy scipts using
galaxy? I installed groovy in galaxy docker image and all the dependencies.
However, when I try to execute a tool which require groovy engine it
says "/export/galaxy-central/database/job_working_directory/000/2/tool_script.sh:
line 9: groovy: command not found"
So I understood the error that galaxy is not able to find groovy but
ideally it should take the groovy engine as it is installed.
Can you give me some pointers to solve this issue?
--
With Regards,
Anshul Tak
2 years, 5 months
Pulsar - running as real DRMAA user problems
by Poole, Richard
Hey John,
So I’ve been happily using Pulsar to send all my Galaxy server jobs to our cluster here at UCL for several months now (I love it!). I am now exploring the ‘run-as-real-user’ option for DRMAA submissions and have run into a problem. The files are correctly staged, correctly chowned, successfully submitted to the queue and the job runs. However, at job end (collection?) fails with the following error message in Pulsar:
Exception happened during processing of request from (‘*.*.*.*', 54321)
Traceback (most recent call last):
File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 1072, in process_request_in_thread
self.finish_request(request, client_address)
File "/opt/rocks/lib/python2.6/SocketServer.py", line 322, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/opt/rocks/lib/python2.6/SocketServer.py", line 617, in __init__
self.handle()
File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 446, in handle
BaseHTTPRequestHandler.handle(self)
File "/opt/rocks/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 441, in handle_one_request
self.wsgi_execute()
File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 291, in wsgi_execute
self.wsgi_start_response)
File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 39, in __call__
return controller(environ, start_response, **request_args)
File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 144, in controller_replacement
result = self.__execute_request(func, args, req, environ)
File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 124, in __execute_request
result = func(**args)
File "/cluster/galaxy/pulsar/pulsar/web/routes.py", line 82, in status
return status_dict(manager, job_id)
File "/cluster/galaxy/pulsar/pulsar/manager_endpoint_util.py", line 12, in status_dict
job_status = manager.get_status(job_id)
File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 95, in get_status
proxy_status, state_change = self.__proxy_status(job_directory, job_id)
File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 115, in __proxy_status
proxy_status = self._proxied_manager.get_status(job_id)
File "/cluster/galaxy/pulsar/pulsar/managers/queued_external_drmaa_original.py", line 62, in get_status
external_status = super(ExternalDrmaaQueueManager, self)._get_status_external(external_id)
File "/cluster/galaxy/pulsar/pulsar/managers/base/base_drmaa.py", line 31, in _get_status_external
drmaa_state = self.drmaa_session.job_status(external_id)
File "/cluster/galaxy/pulsar/pulsar/managers/util/drmaa/__init__.py", line 50, in job_status
return self.session.jobStatus(str(external_job_id))
File "build/bdist.linux-x86_64/egg/drmaa/session.py", line 518, in jobStatus
c(drmaa_job_ps, jobId, byref(status))
File "build/bdist.linux-x86_64/egg/drmaa/helpers.py", line 299, in c
return f(*(args + (error_buffer, sizeof(error_buffer))))
File "build/bdist.linux-x86_64/egg/drmaa/errors.py", line 151, in error_check
raise _ERRORS[code - 1](error_string)
InvalidJobException: code 18: The job specified by the 'jobid' does not exist.
With this corresponding error from my Galaxy server:
galaxy.tools.actions INFO 2016-10-13 18:47:51,851 Handled output (279.421 ms)
galaxy.tools.actions INFO 2016-10-13 18:47:52,093 Verified access to datasets (5.271 ms)
galaxy.tools.execute DEBUG 2016-10-13 18:47:52,118 Tool [toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4] created job [25008<http://toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4]%...>] (560.404 ms)
galaxy.jobs DEBUG 2016-10-13 18:47:52,579 (25008) Working directory for job is: /Users/galaxy/galaxy-dist/database/job_working_directory/025/25008
galaxy.jobs.handler DEBUG 2016-10-13 18:47:52,591 (25008) Dispatching to pulsar runner
galaxy.jobs DEBUG 2016-10-13 18:47:52,677 (25008) Persisting job destination (destination id: hpc_low)
galaxy.jobs.runners DEBUG 2016-10-13 18:47:52,681 Job [25008] queued (90.231 ms)
galaxy.jobs.handler INFO 2016-10-13 18:47:52,699 (25008) Job dispatched
galaxy.tools.deps DEBUG 2016-10-13 18:47:53,138 Building dependency shell command for dependency 'samtools'
galaxy.jobs.runners.pulsar INFO 2016-10-13 18:47:53,233 Pulsar job submitted with job_id 25008
galaxy.jobs DEBUG 2016-10-13 18:47:53,257 (25008) Persisting job destination (destination id: hpc_low)
galaxy.datatypes.metadata DEBUG 2016-10-13 18:51:03,922 Cleaning up external metadata files
galaxy.jobs.runners.pulsar ERROR 2016-10-13 18:51:03,945 failure finishing job 25008
Traceback (most recent call last):
File "/Users/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pulsar.py", line 386, in finish_job
run_results = client.full_status()
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 132, in full_status
return self.raw_check_complete()
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 28, in replacement
return func(*args, **kwargs)
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 13, in replacement
response = func(*args, **kwargs)
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 146, in raw_check_complete
check_complete_response = self._raw_execute("status", {"job_id": self.job_id})
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 215, in _raw_execute
return self.job_manager_interface.execute(command, args, data, input_path, output_path)
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/interface.py", line 96, in execute
response = self.transport.execute(url, method=method, data=data, input_path=input_path, output_path=output_path)
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 34, in execute
response = self._url_open(request, data)
File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 20, in _url_open
return urlopen(request, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
I am running 15.10 and Python 2.7.10 on my iMac for the server and the cluster submission node is running Pulsar 0.5.0 and Python 2.7.12
For these tests I run Pulsar in an interactive window so I have not set the sudoers file up, but rather enter sudo password when requested by Pulsar (at the first step of chowning the staging directory). Also have rewrites set up in Galaxy’s pulsar_actions.yml and I am using remote_scp for the file transfers rather than http - although I have also tried switching back to http (as I noticed caching, which I am also testing, does not work with scp transfers) but get an identical set of error messages.
As I say, I have no troubles using a regular queued_drmaa manager in pulsar. Any ideas what the problem may be?
Cheers,
Rich
Richard J Poole PhD
Wellcome Trust Fellow
Department of Cell and Developmental Biology
University College London
518 Rockefeller
21 University Street, London WC1E 6DE
Office (518 Rockefeller): +44 20 7679 6577 (int. 46577)
Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133)
https://www.ucl.ac.uk/cdb/academics/poole
2 years, 8 months
Running jobs as real user and extra_file_path
by Louise-Amélie Schmitt
Hi everyone,
I just wanted to ask how the extra_file_path is handled in case of job
running as the real user since the file_path is only writable by the
galaxy user. Any clue?
Thanks,
L-A
2 years, 9 months
Testing updated NCBI BLAST+ wrappers for version 2.5.0
by Peter Cock
Hello all,
I have updated the NCBI BLAST+ wrappers on the Test Tool Shed,
the wrapper is now at v0.2.00:
https://testtoolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/
The main changes is this now depends on BLAST+ 2.5.0, and that is
available via either BioConda or the Tool Shed:
https://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_5_0/
https://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_5_0/
In order for the dependency to work smoothly on both BioConda
and the Tool Shed system, we have changed the package name
from "blast+" to just "blast". Given the NCBI stopped updated the
original "legacy" BLAST some time ago, when combined with the
version number this is no longer ambiguous.
Jumping from using BLAST+ 2.2.31 to using BLAST+ 2.5.0
required updating lots of the test files for NCBI changes, including
dropping the GI numbers in many outputs, expanding the percentage
identity field from 2dp to 3dp, and also changing how -parse_deflines
works with tabular output.
The wrappers (deliberately) do not yet offer any new functionality
added in the recent NCBI BLAST+ updates, in particular BLAST
XML v2 is not yet available as an output with a datatype in Galaxy.
At this point I would welcome feedback from those of you using the
BLAST+ wrappers - including if you were able to install this with the
dependencies from BioConda or the traditional Tool Shed packages.
Once I'm confident that this is all OK, I will update the main Tool Shed
(and think about adding new functionality in 2017).
Thank you all,
Peter
2 years, 9 months
installation error on venv dependencies.xml
by Linda Bakker
Hi All,
I am having some problems with my tool dependencies to setup a python venv.
Dependencies.xml looks like this:
When i install my tool from the toolshed, it fails to install the venv
dependency with this error message:
I am not sure what the problem exactly seems to be. os.py should be a
default python package right or do i need to specify it in the
dependencies file?
Can someone point me in the right direction of what the problem might be.
Thanks!
with regards ,
Linda Bakker
--
Linda Bakker
Bioinformatics Research Assistant
Plant Research International, Wageningen UR
Building 107 (Radix), room W1.Fa.054
Droevendaalsesteeg 1, 6708 PB, Wageningen, The Netherlands
tel: +31-317-480756
email: linda.bakker(a)wur.nl
2 years, 10 months
Issue when using xsendfile with galaxy release_16.07
by SAPET, Frederic
Hello
I'm trying the latest Galaxy (git status gives me : # On branch release_16.07)
I've set the xsend option in galaxy.ini (in order to run Galaxy behind a web server proxy, and apply all methods described here : https://wiki.galaxyproject.org/Admin/Config/ApacheProxy)
apache_xsendfile = True
HTML, gff3,tabular files are just fine but the preview of some other files (txt, csv) is broken on the interface.
When I want to preview a txt file :
Not Found
The requested URL /datasets/fd1df6a9052810d5/display/ was not found on this server.
And apache says :
[Thu Aug 18 15:33:48 2016] [error] [client 192.168.29.12] (13)Permission denied: xsendfile: cannot open file: /softs/bioinfo/galaxy-prod/database/files/000/dataset_19.dat, referer: http:
But it looks like this is not really a problem related to access.
Indeed, from the same file, I switch it from txt to tabular (edit attributes), the preview works.
Are you able to reproduce the behavior ?
The same problem occurs on release_16.01.
It's fine with the release v15.05 which is my production server right now.
Fred
2 years, 11 months
Blast db permission
by Mohamed Kassam
Dear all,
I would like to know if it is possible to put permission on the macro.xml
for ncbi-blast + database.
For example if I want to show the db to one user and not to all.
Best reagards,
Mohamed
2 years, 11 months
Tools using Galaxy's Python library?
by Peter Cock
Hi all,
A few of my tools have for a long time used Galaxy's own parsing
functionality in order to avoid an external dependency. Lately
this has stopped working on my TravisCI testing with planemo
using the Galaxy dev branch (the stable master branch is fine):
e.g.
https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
The tools fail with things like:
| from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
| ImportError: No module named galaxy_utils.sequence.fasta
or:
| from galaxy_utils.sequence.fastq import fastqReader
| ImportError: No module named galaxy_utils.sequence.fastq
Is this a temporary regression in Galaxy, or a deliberate change?
Do the tools need to do something to explicit have access to the
Galaxy Python library, or are they now considered private?
If so, I can update these tools to use an explicit dependency
for parsing FASTA and FASTQ (e.g. Biopython).
Thanks,
Peter
2 years, 11 months
Vote for the Training you would like to see at GCC2017
by Dave Clements
Hello potential GCC2017 Attendees!
The 2017 Galaxy Community Conference (GCC2017)
<https://gcc2017.sciencesconf.org/> will be held 26-30 June, in
Montpellier, France. GCC2017 training <https://gcc2017.sciencesconf.org/> will
be held 27-28 June. *We need your vote to determine what to offer at
GCC2017 training.*
If there is even a remote chance that you will attend GCC2017 (it's in the
South of France; it will be summer; don't let that influence your
decision), then *please vote on which of the **34 nominated topics you
would like to see offered at GCC2017.*
*Vote Now!
<https://docs.google.com/forms/d/e/1FAIpQLSdpkLoRuCGZOO3_T6dgRwFeDxVjb_zW-...>*
27 June will be single track and focus on how to use Galaxy for research.
28 June will have multiple tracks, and offer the full spectrum of Galaxy
training.
Vote for as many topics as you like, but please note that the more topics
you vote for, the less your vote for each single topic will count.
*Voting closes on 8 February.* Once that happens the organisers will
scramble to confirm instructors and set a schedule. The training schedule
will be posted before registration opens in February.
The nominated topics this year are:
- *Using Galaxy Topic Nominations
<https://gcc2017.sciencesconf.org/resource/page/id/8#using>*
1. Galaxy 101 - A gentle introduction to Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#galaxy_101>
2. Galaxy at scale
<https://gcc2017.sciencesconf.org/resource/page/id/8#galaxy_at_scale>
3. Visualisation of BIG DATA in Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#visualisation_of_big_...>
4. GSuite HyperBrowser: integrative analysis of dataset collections
across the genome and epigenome
<https://gcc2017.sciencesconf.org/resource/page/id/8#gsuite_hyperbrowser>
5. ChIPseq analysis using DeepTools and MACS
<https://gcc2017.sciencesconf.org/resource/page/id/8#chipseq>
6. RADseq Data Analysis Through STACKS on Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#radseq>
7. RNAseq analysis in Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#rnaseq>
8. Flow Cytometry Analysis with ImmPort Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#flow_cytometry>
9. Metabolomics Analysis with Galaxy on W4M infrastructure
<https://gcc2017.sciencesconf.org/resource/page/id/8#metabolomics>
10. How to analyze metagenomic and amplicon data in Galaxy?
<https://gcc2017.sciencesconf.org/resource/page/id/8#metagenomics>
11. Galaxy for Proteomics
<https://gcc2017.sciencesconf.org/resource/page/id/8#proteomics>
12. Processing and Analysis of Hi-C data using HiCExplorer
<https://gcc2017.sciencesconf.org/resource/page/id/8#hic>
13. SNP analysis and GWAS using SNiPlay
<https://gcc2017.sciencesconf.org/resource/page/id/8#snp>
14. Small genome de novo assembly using Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#assembly>
15. Scripting Galaxy using the API and BioBlend
<https://gcc2017.sciencesconf.org/resource/page/id/8#scripting>
- *Deploying, Administering, and Extending Galaxy Topics
<https://gcc2017.sciencesconf.org/resource/page/id/8#admin>*
1. Introduction to Galaxy admin: Setting up a Galaxy instance as a
service
<https://gcc2017.sciencesconf.org/resource/page/id/8#intro_admin>
2. Advanced customisation of a Galaxy instance
<https://gcc2017.sciencesconf.org/resource/page/id/8#custom>
3. Advanced accelerated Galaxy admin
<https://gcc2017.sciencesconf.org/resource/page/id/8#aaga>
4. How to use Galaxy Ansible Playbooks
<https://gcc2017.sciencesconf.org/resource/page/id/8#ansible>
5. Testing Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#testing>
6. Using and deploying your application to the cloud
<https://gcc2017.sciencesconf.org/resource/page/id/8#cloud>
7. Galaxy Architecture
<https://gcc2017.sciencesconf.org/resource/page/id/8#arch>
8. Writing & Publishing Galaxy Tools
<https://gcc2017.sciencesconf.org/resource/page/id/8#tools_basic>
9. Advanced Topics in Galaxy Tool Development
<https://gcc2017.sciencesconf.org/resource/page/id/8#tools_advanced>
10. Visualisation Development in Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#viz_dep>
11. Adding support for new external sources
<https://gcc2017.sciencesconf.org/resource/page/id/8#data_sources>
12. Conda and Containers for Tool Dependencies - A Developers
Perspective
<https://gcc2017.sciencesconf.org/resource/page/id/8#conda_dev>
13. Conda and Containers for Tool Dependencies - A Deployers
Perspective
<https://gcc2017.sciencesconf.org/resource/page/id/8#conda_dep>
14. Introduction to Galaxy Interactive Environments
<https://gcc2017.sciencesconf.org/resource/page/id/8#ie_intro>
15. Advanced topics in Galaxy Interactive Environments
<https://gcc2017.sciencesconf.org/resource/page/id/8#ie_adv>
16. Galaxy and Tripal - coupling Galaxy to a community database
<https://gcc2017.sciencesconf.org/resource/page/id/8#tripal>
17. The Galaxy Genome Annotation project: Dockerized genome
annotation environment based on Galaxy and GMOD tools
<https://gcc2017.sciencesconf.org/resource/page/id/8#gap>
18. Using Hadoop with Galaxy
<https://gcc2017.sciencesconf.org/resource/page/id/8#hadoop>
- *Trainer Training Topics
<https://gcc2017.sciencesconf.org/resource/page/id/8#trainer_training>*
1. Galaxy for Training and Education
<https://gcc2017.sciencesconf.org/resource/page/id/8#galaxy_for_training>
Please vote
<https://docs.google.com/forms/d/e/1FAIpQLSdpkLoRuCGZOO3_T6dgRwFeDxVjb_zW-...>!
Your participation helps make Galaxy Community Conferences the successful
and community focused events that they are. And, thanks to everyone who
nominated topics and/or contributed topic descriptions.
Hope to see you in Montpellier!
Dave Clements, on behalf of the GCC2017 Organising Committee
--
http://galaxyproject.org/
http://getgalaxy.org/
http://usegalaxy.org/
https://wiki.galaxyproject.org/
2 years, 11 months
Request more details from Nikolaos Tur about pyBamTools & pyBamParser dependency packages message
by Jennifer Hsieh
Hi,
I've installed a local instance of Galaxy (on Ubuntu 16.04, python 2.7.12)
and have run in to this exact problem
(posted by Nikolaos Tur
https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/msg03955....
)
while trying to add naive_variant_caller tool.
Naive_variant_caller can't install because these two dependency packages
can't be installed.
I basically get exactly the same error message that ends with "IOError:
Could not build the egg."
Unfortunately I'm very new to both Galaxy and Python (Taking Coursera
Galaxy course at the moment) and I don't really understand the solution
Nikolaos described about modifying the setup.py to use "setuptools" instead
of "distribute". I don't even know where to find setup.py...
Can anyone help me with step-by-step details as to how to solve this
dependency problem?
Much Thanks!
--
Jennifer Hsieh
PhD Neuroscience (student)
Department of Psychiatry and Mental Health
University of Cape Town
Tell: +27 21 447 3157
Cell: +27 74 841 4761
2 years, 11 months
