November 2016 - galaxy-dev - lists.galaxyproject.org

Pulsar - running as real DRMAA user problems
by Poole, Richard 13 May '17

13 May '17

Hey John, So I’ve been happily using Pulsar to send all my Galaxy server jobs to our cluster here at UCL for several months now (I love it!). I am now exploring the ‘run-as-real-user’ option for DRMAA submissions and have run into a problem. The files are correctly staged, correctly chowned, successfully submitted to the queue and the job runs. However, at job end (collection?) fails with the following error message in Pulsar: Exception happened during processing of request from (‘*.*.*.*', 54321) Traceback (most recent call last): File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 1072, in process_request_in_thread self.finish_request(request, client_address) File "/opt/rocks/lib/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/opt/rocks/lib/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 446, in handle BaseHTTPRequestHandler.handle(self) File "/opt/rocks/lib/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 441, in handle_one_request self.wsgi_execute() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 291, in wsgi_execute self.wsgi_start_response) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 39, in __call__ return controller(environ, start_response, **request_args) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 144, in controller_replacement result = self.__execute_request(func, args, req, environ) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 124, in __execute_request result = func(**args) File "/cluster/galaxy/pulsar/pulsar/web/routes.py", line 82, in status return status_dict(manager, job_id) File "/cluster/galaxy/pulsar/pulsar/manager_endpoint_util.py", line 12, in status_dict job_status = manager.get_status(job_id) File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 95, in get_status proxy_status, state_change = self.__proxy_status(job_directory, job_id) File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 115, in __proxy_status proxy_status = self._proxied_manager.get_status(job_id) File "/cluster/galaxy/pulsar/pulsar/managers/queued_external_drmaa_original.py", line 62, in get_status external_status = super(ExternalDrmaaQueueManager, self)._get_status_external(external_id) File "/cluster/galaxy/pulsar/pulsar/managers/base/base_drmaa.py", line 31, in _get_status_external drmaa_state = self.drmaa_session.job_status(external_id) File "/cluster/galaxy/pulsar/pulsar/managers/util/drmaa/__init__.py", line 50, in job_status return self.session.jobStatus(str(external_job_id)) File "build/bdist.linux-x86_64/egg/drmaa/session.py", line 518, in jobStatus c(drmaa_job_ps, jobId, byref(status)) File "build/bdist.linux-x86_64/egg/drmaa/helpers.py", line 299, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "build/bdist.linux-x86_64/egg/drmaa/errors.py", line 151, in error_check raise _ERRORS[code - 1](error_string) InvalidJobException: code 18: The job specified by the 'jobid' does not exist. With this corresponding error from my Galaxy server: galaxy.tools.actions INFO 2016-10-13 18:47:51,851 Handled output (279.421 ms) galaxy.tools.actions INFO 2016-10-13 18:47:52,093 Verified access to datasets (5.271 ms) galaxy.tools.execute DEBUG 2016-10-13 18:47:52,118 Tool [toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4] created job [25008<http://toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4]%20…>] (560.404 ms) galaxy.jobs DEBUG 2016-10-13 18:47:52,579 (25008) Working directory for job is: /Users/galaxy/galaxy-dist/database/job_working_directory/025/25008 galaxy.jobs.handler DEBUG 2016-10-13 18:47:52,591 (25008) Dispatching to pulsar runner galaxy.jobs DEBUG 2016-10-13 18:47:52,677 (25008) Persisting job destination (destination id: hpc_low) galaxy.jobs.runners DEBUG 2016-10-13 18:47:52,681 Job [25008] queued (90.231 ms) galaxy.jobs.handler INFO 2016-10-13 18:47:52,699 (25008) Job dispatched galaxy.tools.deps DEBUG 2016-10-13 18:47:53,138 Building dependency shell command for dependency 'samtools' galaxy.jobs.runners.pulsar INFO 2016-10-13 18:47:53,233 Pulsar job submitted with job_id 25008 galaxy.jobs DEBUG 2016-10-13 18:47:53,257 (25008) Persisting job destination (destination id: hpc_low) galaxy.datatypes.metadata DEBUG 2016-10-13 18:51:03,922 Cleaning up external metadata files galaxy.jobs.runners.pulsar ERROR 2016-10-13 18:51:03,945 failure finishing job 25008 Traceback (most recent call last): File "/Users/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pulsar.py", line 386, in finish_job run_results = client.full_status() File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 132, in full_status return self.raw_check_complete() File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 28, in replacement return func(*args, **kwargs) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 13, in replacement response = func(*args, **kwargs) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 146, in raw_check_complete check_complete_response = self._raw_execute("status", {"job_id": self.job_id}) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 215, in _raw_execute return self.job_manager_interface.execute(command, args, data, input_path, output_path) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/interface.py", line 96, in execute response = self.transport.execute(url, method=method, data=data, input_path=input_path, output_path=output_path) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 34, in execute response = self._url_open(request, data) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 20, in _url_open return urlopen(request, data) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open response = meth(req, response) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response 'http', request, response, code, msg, hdrs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error return self._call_chain(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error I am running 15.10 and Python 2.7.10 on my iMac for the server and the cluster submission node is running Pulsar 0.5.0 and Python 2.7.12 For these tests I run Pulsar in an interactive window so I have not set the sudoers file up, but rather enter sudo password when requested by Pulsar (at the first step of chowning the staging directory). Also have rewrites set up in Galaxy’s pulsar_actions.yml and I am using remote_scp for the file transfers rather than http - although I have also tried switching back to http (as I noticed caching, which I am also testing, does not work with scp transfers) but get an identical set of error messages. As I say, I have no troubles using a regular queued_drmaa manager in pulsar. Any ideas what the problem may be? Cheers, Rich Richard J Poole PhD Wellcome Trust Fellow Department of Cell and Developmental Biology University College London 518 Rockefeller 21 University Street, London WC1E 6DE Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133) https://www.ucl.ac.uk/cdb/academics/poole

2 5

Running jobs as real user and extra_file_path
by Louise-Amélie Schmitt 28 Apr '17

28 Apr '17

Hi everyone, I just wanted to ask how the extra_file_path is handled in case of job running as the real user since the file_path is only writable by the galaxy user. Any clue? Thanks, L-A

3 3

Issue when using xsendfile with galaxy release_16.07
by SAPET, Frederic 28 Feb '17

28 Feb '17

Hello I'm trying the latest Galaxy (git status gives me : # On branch release_16.07) I've set the xsend option in galaxy.ini (in order to run Galaxy behind a web server proxy, and apply all methods described here : https://wiki.galaxyproject.org/Admin/Config/ApacheProxy) apache_xsendfile = True HTML, gff3,tabular files are just fine but the preview of some other files (txt, csv) is broken on the interface. When I want to preview a txt file : Not Found The requested URL /datasets/fd1df6a9052810d5/display/ was not found on this server. And apache says : [Thu Aug 18 15:33:48 2016] [error] [client 192.168.29.12] (13)Permission denied: xsendfile: cannot open file: /softs/bioinfo/galaxy-prod/database/files/000/dataset_19.dat, referer: http: But it looks like this is not really a problem related to access. Indeed, from the same file, I switch it from txt to tabular (edit attributes), the preview works. Are you able to reproduce the behavior ? The same problem occurs on release_16.01. It's fine with the release v15.05 which is my production server right now. Fred

2 4

display at UCSC does not work
by Hans-Rudolf Hotz 24 Jan '17

24 Jan '17

Hi I am trying to set up the possibility to display data sets with the UCSC genome browser for our new galaxy installation. After clicking on "display at UCSC main", I get the UCSC genome browser, opening in the right assembly, and at the correct location, but I get the following Error: Warning/Error(s): redirected to non-http(s): /login?redirect=%2Froot%2Fdisplay_as In galaxy.ini, I did uncomment the line: display_servers = hgw1.cse.ucsc.edu,hgw2.cse.ucsc.edu,hgw3.cse. ... We use the 'new' authentication method to connect to our LDAP server, with the following changes in galaxy.ini: require_login = True auth_config_file = config/auth_conf.xml Connection from UCSC to our galaxy server is working, since I can display the data, when I switch to require_login = False it works, independent of whether I am logged in or not. Is there a way to get it to work, even if I have "require_login = True"? Thank you very much for your help Regards, Hans-Rudolf -- Hans-Rudolf Hotz, PhD Bioinformatics Support Friedrich Miescher Institute for Biomedical Research Maulbeerstrasse 66 4058 Basel/Switzerland

1 1

Python script can't can't import BioPython modules in Galaxy 16.01
by Makis Ladoukakis 17 Jan '17

17 Jan '17

Hello everyone, I've recently upgraded to Galaxy 16.01 and i have some issues with a python wrapper from a tool. When I run it via Galaxy I get the following error: Traceback (most recent call last): File "/home/galaxy_user/galaxy/tools/myTools/know_parser/know_parser.py", line 12, in <module> from Bio import SeqIO ImportError: No module named Bio However when I click on the "view details" icon to copy the command from the tool and run it on my command line the tool works perfectly. In addition this issue occurs while I already have installed BioPython. When I type Python and write "from Bio import SeqIO" I get no error. Can this be an issue related to the virtual environment that is initialized each time Galaxy starts? Thank you in advance, Makis Ladoukakis

5 8

tmp file under database/tmp get regenerated after restarting Galaxy
by Mark Xiao 20 Dec '16

20 Dec '16

Dear Galaxy Developers, I am using Galaxy version 15.10 on my server, load-balancing is enabled. I found a problem when running the jobs. When there are running jobs and if I restart the Galaxy process at this time, the job will fail at the end of the job with the error message such as: IOError: [Errno 2] No such file or directory: '/galaxy/database/tmp/tmpleXN4b' That tmp file doesn't exist, I guess it is because when Galaxy process get restarted, the tmp file get regenerated, so the job cannot find the correct tmp file, then it was failed. Do you think this is a known issue, and is there a fix for this? Thank you in advance! Mark

1 1

Datasets incorrectly flagged as deleted
by Lance Parsons 05 Dec '16

05 Dec '16

I've run into issues over the past year where some jobs would occasionally fail to start (stuck in a `new` state). I tracked them down to a situataion where `dataset.deleted` is set to `t` yet the `history_dataset_association.deleted` is `f`. Simply setting `dataset.deleted` to `f` in those instances resolved the issue and the jobs ran. The datasets have all still been on disk. Since this is a pretty annoying situation, I thought I'd check to see if there are other datasets with this problem. Shockingly, I found many thousands of such datasets: ``` select count(d.id) from dataset d join history_dataset_association hda on d.id = hda.dataset_id where d.deleted = 't' and hda.deleted = 'f'; count ------- 76977 (1 row) ``` I'm hesitant to update so many rows in my database so I thought I'd put this out there for comment. What do others see when running the above query? Has anyone run into this or a similar issue? Thanks. -- Lance Parsons - Scientific Programmer Carl C. Icahn Laboratory - Room 136 Lewis-Sigler Institute for Integrative Genomics Princeton University

3 5

Jobs don't run for one user
by Langhorst, Brad 02 Dec '16

02 Dec '16

Hi: I have one user who’s jobs won’t execute on my local instance. The user was mysteriously listed as inactive. I reactivated and restored the handlers, still jobs are not executing. The user has plenty of quota (82% used) Anybody have an idea about what might be wrong with this user? I have impersonated the user, jobs can be added galaxy.util.object_wrapper WARNING 2016-11-30 00:40:08,118 Unable to create dynamic subclass for <type 'instance'>, None: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases galaxy.tools.actions INFO 2016-11-30 00:40:08,271 Handled output (153.865 ms) galaxy.tools.actions INFO 2016-11-30 00:40:08,330 Verified access to datasets (5.648 ms) galaxy.tools.execute DEBUG 2016-11-30 00:40:08,490 Tool [toolshed.g2.bx.psu.edu/repos/jjohnson/fastqc/FastQC/1.0.0<http://toolshed.g2.bx.psu.edu/repos/jjohnson/fastqc/FastQC/1.0.0>] created job [537201] (384.040 ms) galaxy.tools.execute DEBUG 2016-11-30 00:40:08,498 Executed all jobs for tool request: (413.473 ms) 10.254.254.95 - - [30/Nov/2016:00:40:07 -0400] "POST /api/tools HTTP/1.0" 200 - "http://galaxy.neb.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 OPR/41.0.2353.69” but they never get scheduled or run. Nothing else in the logs after submitting the job. They just remain in this state: This is a new dataset and not all of its data are available yet Any ideas? Brad

2 5

History is not accessible by user
by Maximilian Friedersdorff 01 Dec '16

01 Dec '16

-------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk

2 3

Errno 13 when trying to visualize Bowtie2 data
by Federico Miozzo 30 Nov '16

30 Nov '16

Hi, since few days, when trying to visualize my Bowtie2 data on IGV or UCSC, I get this error message : There was a conflict when trying to complete your request. Error generating display_link: [Errno 13] Permission denied: '/galaxy-repl/main/files/_metadata_files/000' Any idea how to solve this? Thank you for your help. Best, Federico

2 1