December 2016 - galaxy-dev - lists.galaxyproject.org

Datasets incorrectly flagged as deleted
by Lance Parsons 05 Dec '16

05 Dec '16

I've run into issues over the past year where some jobs would occasionally fail to start (stuck in a `new` state). I tracked them down to a situataion where `dataset.deleted` is set to `t` yet the `history_dataset_association.deleted` is `f`. Simply setting `dataset.deleted` to `f` in those instances resolved the issue and the jobs ran. The datasets have all still been on disk. Since this is a pretty annoying situation, I thought I'd check to see if there are other datasets with this problem. Shockingly, I found many thousands of such datasets: ``` select count(d.id) from dataset d join history_dataset_association hda on d.id = hda.dataset_id where d.deleted = 't' and hda.deleted = 'f'; count ------- 76977 (1 row) ``` I'm hesitant to update so many rows in my database so I thought I'd put this out there for comment. What do others see when running the above query? Has anyone run into this or a similar issue? Thanks. -- Lance Parsons - Scientific Programmer Carl C. Icahn Laboratory - Room 136 Lewis-Sigler Institute for Integrative Genomics Princeton University

3 5

Unable to upload bam files as links to my local galaxy instance - please help!
by Konstantinos Voutetakis 05 Dec '16

05 Dec '16

Dear Sir(s), I have consumed a few days in order to resolve this issue but nothing! I have installed all the SAM tools of public Galaxy in my local galaxy via the Toolshed (installing automatically all the dependencies were needed). I have also export the path of the executable file SAMtools from bin folder to my Galaxy path and I take the following error during uploading: Traceback (most recent call last): File "/home/user/galaxy/tools/data_source/upload.py", line 434, in <module> __main__() File "/home/user/galaxy/tools/data_source/upload.py", line 423, in __main__ add_file( dataset, registry, json_file, output_path ) File "/home/user/galaxy/tools/data_source/upload.py", line 317, in add_file if datatype.dataset_content_needs_grooming( dataset.path ): File "/home/user/galaxy/lib/galaxy/datatypes/binary.py", line 251, in dataset_content_needs_grooming version = self._get_samtools_version() File "/home/user/galaxy/lib/galaxy/datatypes/binary.py", line 212, in _get_samtools_version raise Exception(message) Exception: Attempting to use functionality requiring samtools, but it cannot be located on Galaxy's PATH. When I am in my galaxy folder (from where I connect to my galaxy server) and I type samtools in the terminal I take: Program: samtools (Tools for alignments in the SAM format) Version: 0.1.19-44428cd Usage: samtools <command> [options] Command: view SAM<->BAM conversion sort sort alignment file mpileup multi-way pileup depth compute the depth faidx index/extract FASTA tview text alignment viewer index index alignment idxstats BAM index stats (r595 or later) fixmate fix mate information flagstat simple stats calmd recalculate MD/NM tags and '=' bases merge merge sorted alignments rmdup remove PCR duplicates reheader replace BAM header cat concatenate BAMs bedcov read depth per BED region targetcut cut fosmid regions (for fosmid pool only) phase phase heterozygotes bamshuf shuffle and group alignments by name What can I do? I installed galaxy to my pc (ubuntu 16.04) before two weeks and it is up to date. I would be grateful for your help. Yours sincerely, Kostas -- Konstantinos G. Voutetakis, MSc - Research Officer National Hellenic Research Foundation (N.H.R.F.) Institute of Biology, Medicinal Chemistry and Biotechnology 48 Vas. Constantinou Ave., Athens 11635 Greece ------------------------------ ---------------------------------------------------- tel.: +30-210-7273894 <%2B30-210-7273752> e-mail: kvoutet(a)eie.gr

2 1

Jobs don't run for one user
by Langhorst, Brad 02 Dec '16

02 Dec '16

Hi: I have one user who’s jobs won’t execute on my local instance. The user was mysteriously listed as inactive. I reactivated and restored the handlers, still jobs are not executing. The user has plenty of quota (82% used) Anybody have an idea about what might be wrong with this user? I have impersonated the user, jobs can be added galaxy.util.object_wrapper WARNING 2016-11-30 00:40:08,118 Unable to create dynamic subclass for <type 'instance'>, None: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases galaxy.tools.actions INFO 2016-11-30 00:40:08,271 Handled output (153.865 ms) galaxy.tools.actions INFO 2016-11-30 00:40:08,330 Verified access to datasets (5.648 ms) galaxy.tools.execute DEBUG 2016-11-30 00:40:08,490 Tool [toolshed.g2.bx.psu.edu/repos/jjohnson/fastqc/FastQC/1.0.0<http://toolshed.g2.bx.psu.edu/repos/jjohnson/fastqc/FastQC/1.0.0>] created job [537201] (384.040 ms) galaxy.tools.execute DEBUG 2016-11-30 00:40:08,498 Executed all jobs for tool request: (413.473 ms) 10.254.254.95 - - [30/Nov/2016:00:40:07 -0400] "POST /api/tools HTTP/1.0" 200 - "http://galaxy.neb.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 OPR/41.0.2353.69” but they never get scheduled or run. Nothing else in the logs after submitting the job. They just remain in this state: This is a new dataset and not all of its data are available yet Any ideas? Brad

2 5

Pulsar: issue configuring to use shared filesystem rather than staging data
by Philip Blood 02 Dec '16

02 Dec '16

Hi Folks, I'm working on a time-sensitive project (just a day or two left to sort it out) that requires I be able to submit jobs from a remote resource (TACC) to my resources (Pittsburgh Supercomputing Center) using a shared filesystem for data rather than staging via Pulsar. I've tried to set it up so that Pulsar does not do any staging, but is just handling the remote job submission and tool dependencies. However, Pulsar continues to use a local staging directory for the data and then copies the data back to the Galaxy directory. What I'm hoping to do is have all the data stay in the shared filesystem for the entire course of the job. The quick test below does not have input data, but just issues commands on the remote compute node and generates output. I'm using latest stable releases of Galaxy (16.07, installed on a TACC VM) and Pulsar (installed via pip at PSC). If anyone can provide some quick pointers I'd appreciate it. Here are relevant parts of my configuration and some of the log output: *Shared filesystem:* /hopper *Local staging dir created by Pulsar:* /usr/local/packages/pulsar/etc/files/staging *galaxy.ini* file_path = /hopper/sy3l67p/blood/galaxy_home/database/files new_file_path = /hopper/sy3l67p/blood/galaxy_home/database/tmp job_working_directory = /hopper/sy3l67p/blood/galaxy_home/database/job_working_directory tool_dependency_dir = /hopper/sy3l67p/blood/galaxy_home/database/dependencies dependency_resolvers_config_file = /hopper/sy3l67p/blood/galaxy_home/config/dependency_resolvers_conf.xml *job_conf.xml* <destination id="pulsar" runner="pulsar_rest"> <param id="url">https://128.182.99.126:8913/</param> <param id="default_file_action">none</param> <param id="file_action_config">/home/tg455546/galaxy/config/file_actions.yaml</param> <param id="dependency_resolution">remote</param> <param id="submit_native_specification">-q batch</param> </destination> *file_actions.yml* paths: # If Galaxy, the Pulsar, and the compute nodes all mount the same directory # staging can be disabled altogether for given paths. - path: /hopper/sy3l67p/blood/galaxy_home/database/files action: none - path: /hopper/sy3l67p/blood/galaxy_home/database/job_working_directory action: none *app.yml* --- manager: type: queued_drmaa dependency_resolvers_config_file: /usr/local/packages/pulsar/etc/dependency_resolvers_conf.xml tool_dependency_dir: /usr/local/packages/pulsar/dependencies *Galaxy paster.log* galaxy.jobs DEBUG 2016-12-01 08:07:35,770 (50) Working directory for job is: /hopper/sy3l67p/blood/galaxy_home/database/job_working_directory\ /000/50 pulsar.client.staging.down INFO 2016-12-01 08:08:08,305 collecting output output with action FileAction[action_type=copy] pulsar.client.client DEBUG 2016-12-01 08:08:08,814 Copying path [/usr/local/packages/pulsar/etc/files/staging/50/working/output] to [/hopper/\ sy3l67p/blood/galaxy_home/database/files/000/dataset_50.dat] *Pulsar uwsgi.log* 2016-12-01 09:07:38,174 INFO [pulsar.managers.base.base_drmaa][[manager=_default_]-[action=preprocess]-[job=50]] Submitting DRMA\ A job with nativeSpecification [-q batch] t #78bc [ 539.97] -> drmaa_allocate_job_template t #78bc [ 539.97] <- drmaa_allocate_job_template =0: jt=0x7f69cc002f80 t #78bc [ 539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, name='drmaa_remote_command', value='/usr/local/packages/pulsar/etc/\ files/staging/50/command.sh') t #78bc [ 539.97] -> fsd_template_set_attr(drmaa_remote_command=/usr/local/packages/pulsar/etc/files/staging/50/command.sh) t #78bc [ 539.97] <- drmaa_set_attribute =0 t #78bc [ 539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, name='drmaa_output_path', value=':/usr/local/packages/pulsar/etc/fi\ les/staging/50/stdout') t #78bc [ 539.97] -> fsd_template_set_attr(drmaa_output_path=:/usr/local/packages/pulsar/etc/files/staging/50/stdout) t #78bc [ 539.97] <- drmaa_set_attribute =0 t #78bc [ 539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, name='drmaa_job_name', value='pulsar_50') t #78bc [ 539.97] -> fsd_template_set_attr(drmaa_job_name=pulsar_50) t #78bc [ 539.97] <- drmaa_set_attribute =0 t #78bc [ 539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, name='drmaa_error_path', value=':/usr/local/packages/pulsar/etc/fil\ es/staging/50/stderr') t #78bc [ 539.97] -> fsd_template_set_attr(drmaa_error_path=:/usr/local/packages/pulsar/etc/files/staging/50/stderr) t #78bc [ 539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, name='drmaa_native_specification', value='-q batch') t #78bc [ 539.97] -> fsd_template_set_attr(drmaa_native_specification=-q batch) t #78bc [ 539.97] <- drmaa_set_attribute =0 t #78bc [ 539.97] -> drmaa_run_job(jt=0x7f69cc002f80) t #78bc [ 539.97] -> pbsdrmaa_session_run_impl(jt=0x7f69cc002f80, bulk_idx=-1) t #78bc [ 539.97] -> fsd_template_set_attr(Checkpoint=u) t #78bc [ 539.97] -> fsd_template_set_attr(Keep_Files=n) t #78bc [ 539.97] -> fsd_template_set_attr(Priority=0) t #78bc [ 539.97] -> pbsdrmaa_write_tmpfile t #78bc [ 539.97] <- pbsdrmaa_write_tmpfile=/tmp/pbs_drmaa.n6VpuR t #78bc [ 539.97] -> fsd_template_set_attr(Job_Name=pulsar_50) t #78bc [ 539.97] -> fsd_template_set_attr(Output_Path=/usr/local/packages/pulsar/etc/files/staging/50/stdout) t #78bc [ 539.97] -> fsd_template_set_attr(Error_Path=/usr/local/packages/pulsar/etc/files/staging/50/stderr) t #78bc [ 539.97] -> fsd_template_set_attr(Variable_List=PBS_O_WORKDIR=/usr/local/packages/pulsar/etc) t #78bc [ 539.97] -> pbsdrmaa_submit_apply_native_specification({native_specification=(null)}) d #78bc [ 539.97] * self->destination_queue = batch t #78bc [ 539.97] -> fsd_template_set_attr(submit_args=-q batch) d #78bc [ 539.97] * set attr: submit_args = -q batch d #78bc [ 539.97] * set attr: Job_Name = pulsar_50 d #78bc [ 539.97] * set attr: Variable_List = PBS_O_WORKDIR=/usr/local/packages/pulsar/etc d #78bc [ 539.97] * set attr: Priority = 0 d #78bc [ 539.97] * set attr: Output_Path = /usr/local/packages/pulsar/etc/files/staging/50/stdout d #78bc [ 539.97] * set attr: Keep_Files = n d #78bc [ 539.97] * set attr: Error_Path = /usr/local/packages/pulsar/etc/files/staging/50/stderr d #78bc [ 539.97] * set attr: Checkpoint = u 2016-12-01 09:08:06,707 INFO [pulsar.client.staging.down][[manager=_default_]-[action=postprocess]-[job=50]] collecting output o\ utput with action FileAction[action_type=copy] 2016-12-01 09:08:06,708 INFO [pulsar.managers.stateful][[manager=_default_]-[action=postprocess]-[job=50]] Status of job [50] ch\ anged to [complete]. No callbacks enabled. -- Philip D. Blood, Ph.D. Senior Computational Scientist Voice: (412) 268-9329 Pittsburgh Supercomputing Center Fax: (412) 268-5832 Carnegie Mellon University Email: blood(a)psc.edu

2 2

Datasets no longer automatically update to yellow "running" state using novel job runner
by Peter Briggs 02 Dec '16

02 Dec '16

Hello all I've recently updated our local Galaxy instances to use a novel in-house job runner as part of a migration to a new VM platform. As far as I can determine the runner is able to run jobs without any issues, on all versions of Galaxy that I've tried (v15.10/v16.01/v16.04/release_16.10). However: when moving to release_16.10 I've noticed a minor issue with the UI, where the job state in the history no longer automatically switches from the grey "queued" state to the yellow "running" state - even though the logs indicate that the job status has changed internally. Refreshing the history correctly updates the job to the correct yellow "running" state (so it would seem that internally Galaxy knows that the job status has changed), and transitions between other states seem to happen automatically. Comparing my novel runner code with that for e.g. the drmaa runner , I can't see what might be missing from mine to cause this issue. I had wondered if it could be a problem with out of date Javascript in the UI somehow not detecting the job status change from "queued" to "running", but using the "local" job runner doesn't exhibit this behaviour. (The source code for the runner can be found here: https://github.com/pjbriggs/ansible-palfinder-galaxy/blob/jse-drop-add-pref… It submits jobs to the cluster by "dropping" the scripts into a shared directory, which are then collected and executed by a separate process running on the cluster - with all communication done via files.) Can anyone suggest what might be happening to cause this issue? Many thanks Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

1 0

htseq_count version 0.6.1p1 dated (2016-09-21) and (2015-07-28)
by Raphenya, Amogelang 01 Dec '16

01 Dec '16

1 0

History is not accessible by user
by Maximilian Friedersdorff 01 Dec '16

01 Dec '16

-------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk

2 3