- galaxy-dev - lists.galaxyproject.org

Which ID ('id', 'workflow_id', and 'dataset_id') should be used?
by Aarthi Mohan 22 May '17

22 May '17

Hi all, I will appreciate your help in understanding the 'id' key returned from the API. I am using Galaxy Version 15.03 & bioblend Version 0.8.0. Example: I have highlighted the id and related fields with bold and red. >>> workflowClient.get_invocations('f7bb1edd6b95db62') > [{u'inputs': {u'1': {u'src': u'hda', u'id': u'06d9fe130fbe098e'}}, > u'update_time': u'2017-05-17T03:09:10', u'uuid': > u'fd066a98-3aad-11e7-90e9-1cc1de6d5ef4', u'history_id': > u'b8a0d6158b9961df', u'state': u'scheduled', *u'workflow_id': > u'915ae9a80309f157'*, u'steps': > ... > u'model_class': u'WorkflowInvocation', *u'id': u'8c49be448cfe29bc'*}] Why is the '*workflow_id*' different from the one I passed to the fucntion? And why is that '*workflow_id' *is not found anywhere in the return value? >>> historyClient.show_dataset(hid,'468b2dfe96a5a9a1') > {u'accessible': True, u'resubmitted': False, u'create_time': > u'2017-05-17T03:04:02', u'download_url': > u'/api/histories/b8a0d6158b9961df/contents/468b2dfe96a5a9a1/display', > u'file_size': 545, *u'dataset_id': u'56c890cbef28295c', u'id': > u'468b2dfe96a5a9a1'*, u'misc_info': u'uploaded fastqsanger file', > u'hda_ldda': u'hda', u'metadata_sequences': 5, u'state': u'ok', > u'display_types': [], u'display_apps': [], u'type': u'file', u'file_path': > None, u'misc_blurb': u'5 sequences', u'peek': u'<table cellspacing="0" > cellpadding="3"><tr><td>@1</td></tr><tr><td>tccacaagccattgtgtgtaattaaccactaattgtgtataagtttaaact</td></tr><tr><td>+</td></tr><tr><td>IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII</td></tr><tr><td>@2</td></tr><tr><td>tccacaagccattgtgtgtaattaaccactaattgtgtataagtttaaact</td></tr></table>', > u'update_time': u'2017-05-17T03:04:06', u'data_type': > u'galaxy.datatypes.sequence.FastqSanger', u'tags': [], u'deleted': False, > u'history_id': u'b8a0d6158b9961df', u'meta_files': [], u'genome_build': > u'?', u'hid': 1, u'model_class': u'HistoryDatasetAssociation', > u'metadata_data_lines': 20, u'file_ext': u'fastqsanger', u'annotation': > None, u'metadata_dbkey': u'?', u'history_content_type': u'dataset', *u'name': > u'a_1.fastq'*, u'extension': u'fastqsanger', u'visible': True, u'url': > u'/api/histories/b8a0d6158b9961df/contents/468b2dfe96a5a9a1', u'uuid': > u'aa6dcf49-6fe9-49e0-8064-c8bc275a37d5', u'visualizations': [], u'purged': > False, u'api_type': u'file'} > >>> historyClient.show_dataset(hid,'56c890cbef28295c') > {u'accessible': True, u'resubmitted': False, u'create_time': > u'2017-05-17T02:59:27', u'file_size': 64, *u'dataset_id': > u'9ccf9e6f1cf4d1fa', u'id': u'56c890cbef28295c'*, u'misc_info': > u'##fileformat=VCFv4.1\n##FILTER=<ID=PASS,Description="All filters > passed">\n##fileDate=20170517\n##source=freeBayes > v0.9.20\n##reference=localref.fa\n##phasing=none\n##commandline="freebayes > --bam localbam_0.bam --fasta-reference localref.fa --vcf /home/sphadmi', > u'hda_ldda': u'hda', u'download_url': > u'/api/histories/06ec17aefa2d49dd/contents/56c890cbef28295c/display', > u'state': u'ok', u'display_types': [], u'display_apps': [], u'type': > u'file', u'file_path': None, u'misc_blurb': u'0 lines', u'peek': u'<table > cellspacing="0" cellpadding="3"><tr><td>#Calculation and writing of high > density regions has completed.</td></tr></table>', u'update_time': > u'2017-05-17T02:59:36', u'data_type': u'galaxy.datatypes.data.Text', > u'tags': [], u'deleted': False, u'history_id': u'06ec17aefa2d49dd', > u'meta_files': [], u'genome_build': u'?', u'hid': 44, u'model_class': > u'HistoryDatasetAssociation', u'metadata_data_lines': None, u'file_ext': > u'txt', u'annotation': None, u'metadata_dbkey': u'?', > u'history_content_type': u'dataset', *u'name': u'High density regions',* > u'extension': u'txt', u'visible': False, u'url': > u'/api/histories/06ec17aefa2d49dd/contents/56c890cbef28295c', u'uuid': > u'8b8c70a4-cd2e-43d3-bc77-b06511557c96', u'visualizations': [], u'purged': > False, u'api_type': u'file'} Similarly, here the '*dataset_id' *is different from the one I passed to *show_dataset* method. If I check the '*dataset_id*' from first call, it points to another different file! Please let me know which of these 'id' should be used and what would be the purpose of the other id? Thanks for your help and time! Best, Aarthi

2 2

Problem with dataset not being cleaned.
by C. Ch. 18 May '17

18 May '17

Hi all, I have the problem that there are a lot of files that are not deleted in the database/files dir. Among these files I have failed uploads and duplicated library files. The latter do not correspond to new versions of existing datasets. I have run the cleanup_datasets.py script but I still have files dating back a year that are in this condition. I have no clue as why/how this could have happened. The galaxy instance is installed on a cluster with the filesystem on a NAS. I have been running 16.01 on this instance (will update as soon as I can stop the service). Could anyone suggest a way to identify those files that have to be in database/files so that I can delete all the rest?? Thanks in advance! Cristian

1 0

Project Publication Reference Tracking Tool -- are you interested in?
by Krzysztof Kutt 16 May '17

16 May '17

Hi! We do not know each other yet, so I will briefly introduce myself. I'm Chris, a Polish PhD candidate in Computer Science field at AGH University of Science and Technology in Krakow. In my work I deal with the broad concept of artificial intelligence. In particular the semantic web technologies (RDF, SPARQL, Linked Data) and the ability to support a group of knowledge engineers who create knowledge inside (semantic) wiki systems - in my PhD I propose some methods and prototypical tools for quality management, change management and motivation. I have also finished the Master's degree in Psychology and as a result I also deal with Affective Computing field. I know, I'm quite far away from bioinformatics :) so, what I am doing here? As a part of Google Summer of Code <https://summerofcode.withgoogle.com/> program I will prepare a tool to keep track of publications that reference a specific project (project's papers, tools, etc). I think that at the end it will be a quite good tool that any researcher can use for dealing with the task, regardless of the research field. Specifically, it will be a set of Python scripts + GUI that will do four tasks: 1) performs database search for new projects (search terms), 2) handles alerts for updating the results later, 3) gives the user the possibility to manually annotate results (e.g. as irrelevant), 4) generates citation reports (e.g. list of papers that cite our project during last year). If you are interested in such a tool, do not hesitate to: * read the more detailed description placed in the project's github wiki: https://github.com/kkutt/references-tracking/wiki/Tool-Description * discuss it on the Gitter channel: https://gitter.im/references- tracking/Lobby or simply write me an e-mail We already have interest from members of the Galaxy, Reactome, InterMine, and BioPython communities. I am waiting for your comments :) If you are not interested in a project, maybe your colleagues are? :) Best regards, Chris Kutt

3 2

location of history id
by Aarthi Mohan 16 May '17

16 May '17

Hello all, I have a local instance of galaxy version 15.03.1 (this is needed by IRIDA tool). I have retrieved datasets associated with a history using Bioblend. I want to confirm, if the history id used in get_histories(hid) or show_history(hid) is stored in the database? I have tried searching the entire database by mysqldump, and couldn't find such values in it. Where else is this id stored and will this be deleted periodically? Appreciate your help! Thanks, Aarthi ________________________________ Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you.

2 1

Galaxy Start Up Error - Double Requirements, Tempita
by Yip, Miu ki 16 May '17

16 May '17

Hi all, When attempting to start up Galaxy, via the run.sh script, we’re getting the following error: galaxyd@genomics:/galaxy$ sh run.sh Activating virtualenv at .venv Requirement already satisfied: pip>=8.1 in ./.venv/lib/python2.7/site-packages Double requirement given: Tempita==0.5.3.dev0 (from -r requirements.txt (line 155)) (already in Tempita==0.5.3dev (from -r requirements.txt (line 57)), name='Tempita') This occurs when the start up script is run with the —no-create-venv and without the flag. It looks like there are two “Tempita” items in the requirements.txt, if one of them is removed, it leads to another error about “ethtools” galaxyd@genomics:/galaxy$ sh run.sh --daemon --no-create-venv Activating virtualenv at .venv Requirement already satisfied: pip>=8.1 in ./.venv/lib/python2.7/site-packages ... Collecting crypto==1.4.1 (from -r requirements.txt (line 81)) Using cached crypto-1.4.1-py2.py3-none-any.whl Requirement already satisfied: cycler==0.10.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 82)) Collecting Cython==0.25.2 (from -r requirements.txt (line 83)) Using cached Cython-0.25.2-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: docopt==0.6.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 86)) Requirement already satisfied: drmaa==0.7.6 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 88)) Collecting ethtool==0.8 (from -r requirements.txt (line 90)) Could not find a version that satisfies the requirement ethtool==0.8 (from -r requirements.txt (line 90)) (from versions: ) No matching distribution found for ethtool==0.8 (from -r requirements.txt (line 90)) We’re not sure if both versions of “Tempita” are required and if they are, how can the application be started up? Running both the run.sh and the run_reports.sh scripts lead to this error about “Tempita”. Has anyone encountered this error before? Thanks!

2 1

Pulsar - metadata staging rewrite problem
by Poole, Richard 13 May '17

13 May '17

Hi John and all, We are running an old version of Galaxy v15.10 and use Pulsar to stage certain jobs to our hpc cluster. This works great and we love it. However, we have noticed that for certain tools which require metadata to be staged, although the metadata file is actually staged, the Galaxy pulsar runner fails to rewrite the file path for the metadata correctly. Here is an example command from the latest Freebayes wrappers: ln -s -f '/cluster/galaxy/pulsar/files/staging/32728/inputs/dataset_54494.dat' 'b_0.bam' && ln -s -f '/Volumes/ngs/database/files/_metadata_files/002/metadata_2754.dat' 'b_0.bam.bai' && ln -s -f '/cluster/galaxy/pulsar/files/staging/32728/inputs/dataset_54496.dat' 'b_1.bam' && ln -s -f '/Volumes/ngs/database/files/_metadata_files/002/metadata_2755.dat' 'b_1.bam.bai' && samtools view -H b_0.bam | grep "^@SQ" | cut -f 2- | awk '{ gsub("^SN:","",$1); gsub("^LN:","",$2); print $1"\t0\t"$2; }' >> regions_all.bed && samtools view -H b_1.bam | grep "^@SQ" | cut -f 2- | awk '{ gsub("^SN:","",$1); gsub("^LN:","",$2); print $1"\t0\t"$2; }' >> regions_all.bed && sort -u regions_all.bed > regions_uniq.bed && mkdir vcf_output && mkdir failed_alleles && mkdir trace && for i in `cat regions_uniq.bed | awk '{print $1":"$2".."$3}'`; do echo " freebayes --region '$i' --bam 'b_0.bam' --bam 'b_1.bam' --fasta-reference '/cluster/galaxy/indexes/danRer7/sam_index/danRer7.fa' --vcf './vcf_output/part_$i.vcf' --standard-filters --min-coverage '3' "; done > freebayes_commands.sh && cat freebayes_commands.sh | parallel --no-notice -j ${GALAXY_SLOTS:-1} && grep "^#" "./vcf_output/part_$i.vcf" > header.txt && for i in `cat regions_uniq.bed | awk '{print $1":"$2".."$3}'`; do cat "./vcf_output/part_$i.vcf" | grep -v "^#" || true ; done | sort -k1,1 -k2,2n -k5,5 -u | cat header.txt - > '/cluster/galaxy/pulsar/files/staging/32728/outputs/dataset_54897.dat’ As you can see, the file path for the BAM file is correctly re-written but not the file path for the bam.bai index, which still has a file path local to Galaxy. Has anyone come across this problem before and is there a fix? We are very hesitant to upgrade our Galaxy server as everything else is working perfectly for us right now. Thanks, Richard

1 0

Pulsar - running as real DRMAA user problems
by Poole, Richard 13 May '17

13 May '17

Hey John, So I’ve been happily using Pulsar to send all my Galaxy server jobs to our cluster here at UCL for several months now (I love it!). I am now exploring the ‘run-as-real-user’ option for DRMAA submissions and have run into a problem. The files are correctly staged, correctly chowned, successfully submitted to the queue and the job runs. However, at job end (collection?) fails with the following error message in Pulsar: Exception happened during processing of request from (‘*.*.*.*', 54321) Traceback (most recent call last): File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 1072, in process_request_in_thread self.finish_request(request, client_address) File "/opt/rocks/lib/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/opt/rocks/lib/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 446, in handle BaseHTTPRequestHandler.handle(self) File "/opt/rocks/lib/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 441, in handle_one_request self.wsgi_execute() File "/opt/rocks/lib/python2.6/site-packages/Paste-2.0.1-py2.6.egg/paste/httpserver.py", line 291, in wsgi_execute self.wsgi_start_response) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 39, in __call__ return controller(environ, start_response, **request_args) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 144, in controller_replacement result = self.__execute_request(func, args, req, environ) File "/cluster/galaxy/pulsar/pulsar/web/framework.py", line 124, in __execute_request result = func(**args) File "/cluster/galaxy/pulsar/pulsar/web/routes.py", line 82, in status return status_dict(manager, job_id) File "/cluster/galaxy/pulsar/pulsar/manager_endpoint_util.py", line 12, in status_dict job_status = manager.get_status(job_id) File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 95, in get_status proxy_status, state_change = self.__proxy_status(job_directory, job_id) File "/cluster/galaxy/pulsar/pulsar/managers/stateful.py", line 115, in __proxy_status proxy_status = self._proxied_manager.get_status(job_id) File "/cluster/galaxy/pulsar/pulsar/managers/queued_external_drmaa_original.py", line 62, in get_status external_status = super(ExternalDrmaaQueueManager, self)._get_status_external(external_id) File "/cluster/galaxy/pulsar/pulsar/managers/base/base_drmaa.py", line 31, in _get_status_external drmaa_state = self.drmaa_session.job_status(external_id) File "/cluster/galaxy/pulsar/pulsar/managers/util/drmaa/__init__.py", line 50, in job_status return self.session.jobStatus(str(external_job_id)) File "build/bdist.linux-x86_64/egg/drmaa/session.py", line 518, in jobStatus c(drmaa_job_ps, jobId, byref(status)) File "build/bdist.linux-x86_64/egg/drmaa/helpers.py", line 299, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "build/bdist.linux-x86_64/egg/drmaa/errors.py", line 151, in error_check raise _ERRORS[code - 1](error_string) InvalidJobException: code 18: The job specified by the 'jobid' does not exist. With this corresponding error from my Galaxy server: galaxy.tools.actions INFO 2016-10-13 18:47:51,851 Handled output (279.421 ms) galaxy.tools.actions INFO 2016-10-13 18:47:52,093 Verified access to datasets (5.271 ms) galaxy.tools.execute DEBUG 2016-10-13 18:47:52,118 Tool [toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4] created job [25008<http://toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.4]%20…>] (560.404 ms) galaxy.jobs DEBUG 2016-10-13 18:47:52,579 (25008) Working directory for job is: /Users/galaxy/galaxy-dist/database/job_working_directory/025/25008 galaxy.jobs.handler DEBUG 2016-10-13 18:47:52,591 (25008) Dispatching to pulsar runner galaxy.jobs DEBUG 2016-10-13 18:47:52,677 (25008) Persisting job destination (destination id: hpc_low) galaxy.jobs.runners DEBUG 2016-10-13 18:47:52,681 Job [25008] queued (90.231 ms) galaxy.jobs.handler INFO 2016-10-13 18:47:52,699 (25008) Job dispatched galaxy.tools.deps DEBUG 2016-10-13 18:47:53,138 Building dependency shell command for dependency 'samtools' galaxy.jobs.runners.pulsar INFO 2016-10-13 18:47:53,233 Pulsar job submitted with job_id 25008 galaxy.jobs DEBUG 2016-10-13 18:47:53,257 (25008) Persisting job destination (destination id: hpc_low) galaxy.datatypes.metadata DEBUG 2016-10-13 18:51:03,922 Cleaning up external metadata files galaxy.jobs.runners.pulsar ERROR 2016-10-13 18:51:03,945 failure finishing job 25008 Traceback (most recent call last): File "/Users/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pulsar.py", line 386, in finish_job run_results = client.full_status() File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 132, in full_status return self.raw_check_complete() File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 28, in replacement return func(*args, **kwargs) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/decorators.py", line 13, in replacement response = func(*args, **kwargs) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 146, in raw_check_complete check_complete_response = self._raw_execute("status", {"job_id": self.job_id}) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/client.py", line 215, in _raw_execute return self.job_manager_interface.execute(command, args, data, input_path, output_path) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/interface.py", line 96, in execute response = self.transport.execute(url, method=method, data=data, input_path=input_path, output_path=output_path) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 34, in execute response = self._url_open(request, data) File "/Users/galaxy/galaxy-dist/lib/pulsar/client/transport/standard.py", line 20, in _url_open return urlopen(request, data) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open response = meth(req, response) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response 'http', request, response, code, msg, hdrs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error return self._call_chain(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error I am running 15.10 and Python 2.7.10 on my iMac for the server and the cluster submission node is running Pulsar 0.5.0 and Python 2.7.12 For these tests I run Pulsar in an interactive window so I have not set the sudoers file up, but rather enter sudo password when requested by Pulsar (at the first step of chowning the staging directory). Also have rewrites set up in Galaxy’s pulsar_actions.yml and I am using remote_scp for the file transfers rather than http - although I have also tried switching back to http (as I noticed caching, which I am also testing, does not work with scp transfers) but get an identical set of error messages. As I say, I have no troubles using a regular queued_drmaa manager in pulsar. Any ideas what the problem may be? Cheers, Rich Richard J Poole PhD Wellcome Trust Fellow Department of Cell and Developmental Biology University College London 518 Rockefeller 21 University Street, London WC1E 6DE Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133) https://www.ucl.ac.uk/cdb/academics/poole

2 5

Fwd: Galaxy Error while Using PBS
by Jennifer Hillman-Jackson 13 May '17

13 May '17

---------- Forwarded message ---------- From: Wenkai Wang (Kevin) <wenkai_wang(a)shbiochip.com> Date: Thu, May 11, 2017 at 8:01 PM Subject: Galaxy Error while Using PBS I am an engineer at a National Research and Engineering Center in Shanghai. Recently I am trying the outstanding Galaxy system developed by your team. The Galaxy system is great; however, there are some errors when Galaxy is connected to Torque PBS, about which errors I have to ask for help and advice from you. When I tried to execute a tool for output like "Hello Galaxy" from within the webpage, it was shown in webpage that "Unable to run this job due to a cluster error, please retry it later" (ref.: line 321 of file galaxy/lib/galaxy/jobs/runners/pbs.py). Meanwhile, the STDOUT of Linux shell displays information including, galaxy.jobs.runners.pbs DEBUG 2017-05-12 11:41:47,459 (25) submitting file /data3/wangwk/app/galaxy/database/pbs/25.sh galaxy.jobs.runners.pbs WARNING 2017-05-12 11:41:47,460 (25) pbs_submit failed (try 1/5), PBS error 15033: No free connections ...... galaxy.jobs.runners.pbs WARNING 2017-05-12 11:41:55,470 (25) pbs_submit failed (try 5/5), PBS error 15033: No free connections galaxy.jobs.runners.pbs ERROR 2017-05-12 11:41:57,473 (25) All attempts to submit job failed galaxy.model.metadata DEBUG 2017-05-12 11:41:57,574 Cleaning up external metadata files galaxy.model.metadata DEBUG 2017-05-12 11:41:57,604 Failed to cleanup MetadataTempFile temp files from /data3/wangwk/app/galaxy/ database/jobs_directory/000/25/metadata_out_HistoryDatasetAssociation_26_wrVwaf: No JSON object could be decoded Further information: (1) Linux version: RHEL 6.2, 2.6.32-220.el6.x86_64; (2) Torque PBS 4.2; (3) pbs-python: 4.4.1.2 (pbs-python 4.4.2.1 requires CentOS 7 with PBS Torque 5, and thus was not chosen); (4) glibc 2.12; (5) munge service has been installed, configured, and started on the master node and compute nodes of the cluster; (6) job_conf.xml file is attached with this email. Also, I tried sara_nodes.py from webpage https://oss.trac.surfsara.nl/ pbs_python/wiki/TorqueExamples . The error information is, Traceback (most recent call last): File "./sara_nodes.py", line 644, in <module> print_overview_normal(args.nodes) File "./sara_nodes.py", line 298, in print_overview_normal matched, rest = print_get_nodes(hosts) File "./sara_nodes.py", line 199, in print_get_nodes pbsq = PBSQuery.PBSQuery() File "/home/wangwk/app/anaconda2/lib/python2.7/site-packages/pbs/PBSQuery.py", line 137, in __init__ self.job_server_id = list(self.get_serverinfo())[0] IndexError: list index out of range The error above means function self.get_serverinfo() returns nothing and thus the size of list created based on it is zero. This might be useful for analysis. It would be deeply appreciated if you could offer some guide regarding how to further track or address this problem. Thanks a lot! Best Wishes, Wenkai (Kevin) ------------------------------ Wenkai Wang, National Research and Engineering Center for Biochip Email: wenkai_wang(a)shbiochip.com Email: ww288(a)cantab.net

1 0

Unable to run workflow from command-line
by Aarthi Mohan 12 May '17

12 May '17

Hi all, Aplogies for the long message! I am trying to execute a galaxy workflow from command-line using bioblend. The workflow invloves bwa, gatk, samtools mpileup, bcftools and works with dataset paired-end collections. The following is the script used to achieve it and I am unable to get the workflow to run > from bioblend.galaxy import GalaxyInstance > from bioblend.galaxy.histories import HistoryClient > from bioblend.galaxy.tools import ToolClient > from bioblend.galaxy.workflows import WorkflowClient > from bioblend.galaxy.datasets import DatasetClient > from bioblend.galaxy import dataset_collections as collections > import os,re > > url , key = "http://#.#.#.#:9090", "c5fe47950c99c52a95c6de2542599ddb" > galaxyInstance = GalaxyInstance(url=url, key=key) > historyClient = HistoryClient(galaxyInstance) > toolClient = ToolClient(galaxyInstance) > workflowClient = WorkflowClient(galaxyInstance) > datasetClient = DatasetClient(galaxyInstance) > workflow_list = workflowClient.get_workflows() > > bwa_workflow_id = [ i['id'] for i in workflow_list if > i['name'].startswith('bwa') ][0] #desired workflow id > bwa_workflow_details = workflowClient.show_workflow(bwa_workflow_id) > #workflow details like input, steps, etc., > bwa_workflow_input_id = bwa_workflow_details['inputs'].keys()[0] #workflow > input ids > created_history = historyClient.create_history(name='testing_bwa2bctools2') > print 'created a new history with name ' , created_history['name'] path = "/home/sphadmin/data/fastq_files/" > files = {i:os.path.join(path,i) for i in os.listdir(path) if > os.path.splitext(i)[1] == '.fastq'} > hda = {i:toolClient.upload_file(files[i], created_history['id'], > type='fastq',dbkey='H37Rv') for i in files} > files_dataset_id = {i:hda[i]['outputs'][0]['id'] for i in files} > print 'uploading fastq files to history' > > fwd = re.compile(r'R1') > paired_list = {} > for i in files: > if fwd.search(i): > prefix, file = i.split('_R1')[0] , i > if not prefix in paired_list: > paired_list[prefix] = {} > paired_list[prefix]['forward'] = file > else: > prefix, file = i.split('_R2')[0], i > if not prefix in paired_list: > paired_list[prefix] = {} > paired_list[prefix]['reverse'] = file > > paired_collection = > [collections.CollectionElement(name=i,type='paired',elements=[collections.HistoryDatasetElement(name=j,id=files_dataset_id[paired_list[i][j]]) > for j in paired_list[i] ]) for i in paired_list] > collectionDesc = collections.CollectionDescription(name='My collection > list', type='list:paired', elements=paired_collection) > dataset_collection = > historyClient.create_dataset_collection(history_id=created_history['id'], > collection_description=collectionDesc) > print 'creating dataset collection in history' > data_wstep_id = {bwa_workflow_input_id: {'id': dataset_collection['id'], > 'src':'hda'}} > data_wstep_index = {"0": {'id': dataset_collection['id'], 'src':'hda'}} > exec_wf = workflowClient.invoke_workflow(bwa_workflow_id, > inputs=data_wstep_index,history_id=created_history['id']) > print 'submitting workflow' I have tried using the step_id with run_workflow() and get the following error, workflowClient.run_workflow(bwa_workflow_id, data_wstep_id, > history_id=created_history['id']) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File > "/usr/local/lib/python2.7/dist-packages/bioblend/galaxy/workflows/__init__.py", > line 271, in run_workflow > return Client._post(self, payload) > File "/usr/local/lib/python2.7/dist-packages/bioblend/galaxy/client.py", > line 157, in _post > files_attached=files_attached) > File "/usr/local/lib/python2.7/dist-packages/bioblend/galaxyclient.py", > line 134, in make_post_request > body=r.text, status_code=r.status_code) > bioblend.ConnectionError: Unexpected HTTP status code: 500: {"err_msg": > "Uncaught exception in exposed API method:", "err_code": 0} step_order_index with invoke_workflow() does not produce any error, but workflow is executed. So 'exec_wf' returns with failed 'status'. {u'inputs': {}, u'update_time': u'2017-05-11T09:13:31', u'uuid': > u'199adb04-362a-11e7-90e9-1cc1de6d5ef4', u'history_id': > u'fd767e8b7355e21a', u'state': u'failed', u'workflow_id': > u'1fad1eaf5f4f1766', u'steps': [{u'update_time': u'2017-05-11T09:13:31', > u'job_id': None, u'state': None, u'order_index': 0, u'action': None, > u'model_class': u'WorkflowInvocationStep', u'workflow_step_id': > u'a73186a6d4577bbf', u'id': u'd882ad2044776478'}], u'model_class': > u'WorkflowInvocation', u'id': u'964b37715ec9bd22'} However, If I run the workflow from UI, everything goes fine. Upon checking the invocation details of the workflow run from UI, the 'inputs' is still empty (there is no associated dataset collection id) {u'inputs': {}, u'update_time': u'2017-05-09T03:44:31', u'uuid': > u'c495fef0-3466-11e7-90e9-1cc1de6d5ef4', u'history_id': > u'f158df71bb77bb86', u'state': u'scheduled', u'workflow_id': > u'1fad1eaf5f4f1766', u'steps': [{u'update_time': u'2017-05-09T03:22:47', > u'job_id': u'48885bbd1215db53', u'state': u'ok', u'order_index': 1, > u'action': None, u'model_class': u'WorkflowInvocationStep', > u'workflow_step_id': u'4df0bfb0de34905a', u'id': u'50807609e10e339d'}, > {u'update_time': u'2017-05-09T03:22:47', u'job_id': None, u'state': None, > u'order_index': 0, u'action': None, u'model_class': > u'WorkflowInvocationStep', u'workflow_step_id': u'a73186a6d4577bbf', u'id': > u'f5e1495a2ebc6080'}, {u'update_time': u'2017-05-09T03:22:47', u'job_id': > u'f1cb3a6ff6a93c01', u'state': u'ok', u'order_index': 1, u'action': None, > u'model_class': u'WorkflowInvocationStep', u'workflow_step_id': > u'4df0bfb0de34905a', u'id': u'1d972c3873de3f79'}, {u'update_time': > u'2017-05-09T03:22:50', u'job_id': u'822aeedf25d30c76', u'state': u'ok', > u'order_index': 2, u'action': None, u'model_class': > u'WorkflowInvocationStep', u'workflow_step_id': u'89e52cb393afc192', u'id': > u'89788b3f6c8e75f4'}, {u'update_time': u'2017-05-09T03:22:50', u'job_id': > u'4fb563a6a1060ab1', u'state': u'ok', u'order_index': 2, u'action': None, > u'model_class': u'WorkflowInvocationStep', u'workflow_step_id': > u'89e52cb393afc192', u'id': u'b4666ccc5542d171'}, {u'update_time': > u'2017-05-09T03:22:53', u'job_id': u'cab68a1ccca17e68', u'state': u'ok', > u'order_index': 3, u'action': None, u'model_class': > u'WorkflowInvocationStep', u'workflow_step_id': u'b6b7934af4ebec63', u'id': > u'3dfe330b26f78ced'}, {u'update_time': u'2017-05-09T03:22:53', u'job_id': > u'bb80841fd9d212dd', u'state': u'ok', u'order_index': 3, u'action': None, > u'model_class': u'WorkflowInvocationStep', u'workflow_step_id': > u'b6b7934af4ebec63', u'id': u'f89ec36d748e8f90'}, {u'update_time': > u'2017-05-09T03:22:57', u'job_id': u'02b1cfb59f716d61', u'state': u'ok', > u'order_index': 4, u'action': None, u'model_class': > u'WorkflowInvocationStep', u'workflow_step_id': u'dba3bb6dfae451b8', u'id': > u'dbc267d11443f9a4'}, {u'update_time': u'2017-05-09T03:22:57', u'job_id': > u'705cc5c1fdaa7d3e', u'state': u'ok', u'order_index': 4, u'action': None, > u'model_class': u'WorkflowInvocationStep', u'workflow_step_id': > u'dba3bb6dfae451b8', u'id': u'67ce804af6ec796b'}, {u'update_time': > u'2017-05-09T03:22:59', u'job_id': u'665a42f7453c0065', u'state': u'ok', > u'order_index': 5, u'action': None, u'model_class': > u'WorkflowInvocationStep', u'workflow_step_id': u'5935f39e477d88b2', u'id': > u'09d8568e61ebd1d9'}, {u'update_time': u'2017-05-09T03:22:59', u'job_id': > u'4488b53dbfdabfab', u'state': u'ok', u'order_index': 5, u'action': None, > u'model_class': u'WorkflowInvocationStep', u'workflow_step_id': > u'5935f39e477d88b2', u'id': u'b3e4e0c76fe49b64'}], u'model_class': > u'WorkflowInvocation', u'id': u'e89067bb68bee7a0'} Galaxy Version 15.03 & bioblend Version 0.8.0. Appreciate your inputs. Thanks, Aarthi

1 0

python environment on SGE cluster nodes
by David Simpson 12 May '17

12 May '17

Hi, We're close to having a working Galaxy job_conf.xml to submit galaxy jobs to an SGE cluster. At the moment a submitted job is failing with: python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory We'd like to solve this by sourcing the dev tools for all our jobs: source /opt/rh/python27 enable Please could you advise on how to proceed? regards David -- David Simpson Systems Engineer MRC Centre for Neuropsychiatric Genetics and Genomics Cardiff University School of Medicine Hadyn Ellis Building Maindy Road Cardiff CF24 4HQ ------------------ phone 44 29206 88426 mail SimpsonD4 at Cardiff.ac.uk

1 3