- galaxy-dev - lists.galaxyproject.org

Sharing galaxy on CentOS7/Slurm/HPC/Proftp/Postgres/Apache/LDAP setup configurations
by Sven E. Templer 07 Oct '15

07 Oct '15

Hello together, not being an easy task to set up galaxy on a HPC cluster, satisfying user and the IT department needs, I would like to share our configuration (based on 15.07) to make it easier for others to dive into this. All configuration files and an installation manual is available at: https://github.com/mpg-age-bioinformatics/galaxy-admin Feedback is welcome, Sven

2 2

Some issues with job submission to pulsar
by Raj Ayyampalayam 07 Oct '15

07 Oct '15

Hello, I am setting up a new instance of Galaxy (based on the latest dev branch on github repo) configured to submit the jobs to a remote cluster via pulsar. I used the usegalaxy playbook to get the configuration parameters for the galaxy job_conf.xml and pulsar app.yml files. I was partially successful in running the jobs via pulsar, the regular tools (analysis tools, etc..) are running OK on the remote cluster. I am having trouble getting the upload tool to run on the the remote cluster. Here is the error I see in the galaxy error report for the job: It looks like the upload tool running on the remote machine is trying to access a file in the database/tmp area of the galaxy server machine. Traceback (most recent call last): File "/escratch4/apps/galaxy_scratch/pulsar/files/staging/46/tool_files/upload.py", line 430, in <module> __main__() File "/escratch4/apps/galaxy_scratch/pulsar/files/staging/46/tool_files/upload.py", line 405, in __main__ registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy/lib/galaxy/datatypes/registry.py", line 94, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy/lib/galaxy/util/__init__.py", line 178, in parse_xml root = tree.parse( fname, parser=ElementTree.XMLParser( target=DoctypeSafeCallbackTarget() ) ) File "/usr/local/python/2.7.8/lib/python2.7/xml/etree/ElementTree.py", line 647, in parse source = open(source, "rb") IOError: [Errno 2] No such file or directory: '/home/galaxy/wkdir/galaxy/database/tmp/tmpp6j83l' I've created a Gist with the contents of job_conf.xml, app.yml, file action file, error content at https://gist.github.com/raj76/7be86f6a56714deef050 . Any suggestions on how to debug this issue? Thanks, -Raj

2 2

Problems with Galaxy cluster using grid engine
by Richard Polich 07 Oct '15

07 Oct '15

Hi All, Trying to setup a Galaxy cluster using Rocks Gridengine OS is Centos 6.5. psql (9.1.18) shell bash Getting error messages in paster.log below. I can submit jobs to Gridengine using qsub so this is not an issue. But when trying to "Upload File from you computer”, history indicates jobs does not complete. Any help would be appreciated. galaxy.tools.actions.upload_common DEBUG 2015-10-02 09:35:02,272 Changing ownership of /share/apps/galaxy/database/tmp/upload_file_data_xvpaYs with: /usr/bin/sudo -E /share/apps/galaxy/scripts/external_chown_script.py /share/apps/galaxy/database/tmp/upload_file_data_xvpaYs rpolich 507 galaxy.tools.actions.upload_common WARNING 2015-10-02 09:35:02,297 Changing ownership of uploaded file /share/apps/galaxy/database/tmp/upload_file_data_xvpaYs failed: sudo: no tty present and no askpass program specified galaxy.tools.actions.upload_common DEBUG 2015-10-02 09:35:02,297 Changing ownership of /share/apps/galaxy/database/tmp/tmplIgC3n with: /usr/bin/sudo -E /share/apps/galaxy/scripts/external_chown_script.py /share/apps/galaxy/database/tmp/tmplIgC3n rpolich 507 galaxy.tools.actions.upload_common WARNING 2015-10-02 09:35:02,323 Changing ownership of uploaded file /share/apps/galaxy/database/tmp/tmplIgC3n failed: sudo: no tty present and no askpass program specified galaxy.tools.actions.upload_common INFO 2015-10-02 09:35:02,357 tool upload1 created job id 101 galaxy.tools.execute DEBUG 2015-10-02 09:35:02,423 Tool [upload1] created job [101] (332.351 ms) 206.124.61.6 - - [02/Oct/2015:09:34:59 -0500] "POST /api/tools HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" 206.124.61.6 - - [02/Oct/2015:09:35:02 -0500] "GET /api/histories/1fad1eaf5f4f1766/contents HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" galaxy.jobs DEBUG 2015-10-02 09:35:02,676 (101) Working directory for job is: /share/apps/galaxy/database/job_working_directory/000/101 galaxy.jobs.handler DEBUG 2015-10-02 09:35:02,682 (101) Dispatching to drmaa runner galaxy.jobs DEBUG 2015-10-02 09:35:02,894 (101) Persisting job destination (destination id: sge_default) galaxy.jobs.runners DEBUG 2015-10-02 09:35:02,903 Job [101] queued (220.456 ms) galaxy.jobs.handler INFO 2015-10-02 09:35:02,958 (101) Job dispatched galaxy.jobs.command_factory INFO 2015-10-02 09:35:03,821 Built script [/share/apps/galaxy/database/job_working_directory/000/101/tool_script.sh] for tool command[/share/apps/galaxy/database/job_working_directory/000/101/tool_script.sh] galaxy.jobs.runners DEBUG 2015-10-02 09:35:04,010 (101) command is: /share/apps/galaxy/database/job_working_directory/000/101/tool_script.sh; return_code=$?; python "/share/apps/galaxy/database/job_working_directory/000/101/set_metadata_QaaegG.py" "/share/apps/galaxy/database/tmp/tmpglze54" "/share/apps/galaxy/database/job_working_directory/000/101/galaxy.json" "/share/apps/galaxy/database/job_working_directory/000/101/metadata_in_HistoryDatasetAssociation_71_7DAllZ,/share/apps/galaxy/database/job_working_directory/000/101/metadata_kwds_HistoryDatasetAssociation_71_YiPkTL,/share/apps/galaxy/database/job_working_directory/000/101/metadata_out_HistoryDatasetAssociation_71_JbkolS,/share/apps/galaxy/database/job_working_directory/000/101/metadata_results_HistoryDatasetAssociation_71_d93tKG,/share/apps/galaxy/database/job_working_directory/000/101/galaxy_dataset_71.dat,/share/apps/galaxy/database/job_working_directory/000/101/metadata_override_HistoryDatasetAssociation_71_ih81Fj" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2015-10-02 09:35:04,074 (101) submitting file /share/apps/galaxy/database/job_working_directory/000/101/galaxy_101.sh galaxy.jobs.runners.drmaa DEBUG 2015-10-02 09:35:04,075 (101) native specification is: -q galaxy.q -V galaxy.jobs DEBUG 2015-10-02 09:35:04,075 (101) Changing ownership of working directory with: /usr/bin/sudo -E /share/apps/galaxy/scripts/external_chown_script.py /share/apps/galaxy/database/job_working_directory/000/101 rpolich 507 galaxy.jobs ERROR 2015-10-02 09:35:04,102 (101) Failed to change ownership of /share/apps/galaxy/database/job_working_directory/000/101, making world-writable instead Traceback (most recent call last): File "/share/apps/galaxy/lib/galaxy/jobs/__init__.py", line 1649, in change_ownership_for_run self._change_ownership( self.user_system_pwent[0], str( self.user_system_pwent[3] ) ) File "/share/apps/galaxy/lib/galaxy/jobs/__init__.py", line 1643, in _change_ownership assert p.returncode == 0 AssertionError galaxy.jobs.runners.drmaa DEBUG 2015-10-02 09:35:04,102 (101) submitting with credentials: rpolich [uid: 1006] galaxy.jobs.runners.drmaa DEBUG 2015-10-02 09:35:04,104 (101) Job script for external submission is: /share/apps/galaxy/database/gridengine/101.jt_json galaxy.jobs.runners.drmaa INFO 2015-10-02 09:35:04,104 Running command ['/usr/bin/sudo', '-E', '/share/apps/galaxy/scripts/drmaa_external_runner.py', '1006', '/share/apps/galaxy/database/gridengine/101.jt_json'] galaxy.jobs.runners.drmaa INFO 2015-10-02 09:35:04,308 (101) queued as 239 galaxy.jobs DEBUG 2015-10-02 09:35:04,375 (101) Persisting job destination (destination id: sge_default) galaxy.jobs.runners.drmaa DEBUG 2015-10-02 09:35:05,462 (101/239) state change: job is queued and active 206.124.61.6 - - [02/Oct/2015:09:35:06 -0500] "GET /api/histories/1fad1eaf5f4f1766/contents HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" 206.124.61.6 - - [02/Oct/2015:09:35:10 -0500] "GET /api/histories/1fad1eaf5f4f1766/contents HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" 206.124.61.6 - - [02/Oct/2015:09:35:14 -0500] "GET /api/histories/1fad1eaf5f4f1766/contents HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" galaxy.jobs.runners.drmaa DEBUG 2015-10-02 09:35:17,626 (101/239) state change: job is running 206.124.61.6 - - [02/Oct/2015:09:35:18 -0500] "GET /api/histories/1fad1eaf5f4f1766/contents HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" galaxy.jobs.runners.drmaa INFO 2015-10-02 09:35:22,084 (101/239) job left DRM queue with following message: code 18: The job specified by the 'jobid' does not exist. galaxy.jobs DEBUG 2015-10-02 09:35:22,212 (101) Changing ownership of working directory with: /usr/bin/sudo -E /share/apps/galaxy/scripts/external_chown_script.py /share/apps/galaxy/database/job_working_directory/000/101 galaxy 507 galaxy.jobs.runners ERROR 2015-10-02 09:35:22,240 (unknown) Unhandled exception calling finish_job Traceback (most recent call last): File "/share/apps/galaxy/lib/galaxy/jobs/runners/__init__.py", line 100, in run_next method(arg) File "/share/apps/galaxy/lib/galaxy/jobs/runners/__init__.py", line 554, in finish_job job_state.job_wrapper.reclaim_ownership() File "/share/apps/galaxy/lib/galaxy/jobs/__init__.py", line 1657, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/share/apps/galaxy/lib/galaxy/jobs/__init__.py", line 1643, in _change_ownership assert p.returncode == 0 AssertionError 206.124.61.6 - - [02/Oct/2015:09:35:22 -0500] "GET /api/histories/1fad1eaf5f4f1766/contents HTTP/1.1" 200 - "http://galaxy.txbiomedgenetics.org:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:40.0) Gecko/20100101 Firefox/40.0" My job_conf.xml below…. <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner"/> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="sge_default">  <destination id="sge_default" runner="drmaa"> <param id="nativeSpecification">-q galaxy.q -V</param> </destination> <destination id="local" runner="local"/> </destinations> </job_conf> Output from qacct -j qname galaxy.q hostname compute-1-1703.local group galaxy owner rpolich project NONE department defaultdepartment jobname g101_upload1_rpolich_txbiomed_org jobnumber 239 taskid undefined account sge priority 0 qsub_time Fri Oct 2 09:35:04 2015 start_time Fri Oct 2 09:35:17 2015 end_time Fri Oct 2 09:35:21 2015 granted_pe NONE slots 1 failed 0 exit_status 0 ru_wallclock 4 ru_utime 1.975 ru_stime 0.494 ru_maxrss 37792 ru_ixrss 0 ru_ismrss 0 ru_idrss 0 ru_isrss 0 ru_minflt 63746 ru_majflt 7 ru_nswap 0 ru_inblock 26720 ru_oublock 152 ru_msgsnd 0 ru_msgrcv 0 ru_nsignals 0 ru_nvcsw 6188 ru_nivcsw 522 cpu 2.469 mem 0.372 io 0.225 iow 0.000 maxvmem 463.258M arid undefined Thank you, Richard Polich Systems Administrator Department of Genetics Texas Biomedical Research Institute 7620 NW Loop 410, San Antonio, TX 78227-5301 Phone:(210)258-9727 Email: rpolich(a)txbiomed.org<mailto:rpolich@txbiomed.org>

2 2

Connection to PBS server XXX for state check failed - is it possible to get any help?
by Zuzanna K. Filutowska 06 Oct '15

06 Oct '15

Dear All, is it possible to get any help in a subject of pbs_python problems? Especially Galaxy crash related to this issue. Let me remind: I encountered exactly the same problem: http://osdir.lowified.com/galaxy-development-source-control/2012-07/msg0013… However my Galaxy instance completely crashes until restart, I am getting Proxy Error on HTTPd. Any help would be much appreciated. In logs I have: galaxy.jobs.runners.pbs DEBUG 2015-09-09 15:04:21,306 connection to PBS server ce.reef.man.poznan.pl for state check failed galaxy.jobs.runners.pbs DEBUG 2015-09-09 15:04:21,380 (441/13320109.ce.reef.man.poznan.pl) Skipping state check because PBS server connection failed galaxy.jobs.runners.pbs DEBUG 2015-09-09 15:04:21,430 (444/13320114.ce.reef.man.poznan.pl) Skipping state check because PBS server connection failed +2+12+19+6galaxy+0+32+12+9job_state+1+0+0+02+172+14resources_used+1+0+0 +02+142+11exit_status+1+0+0+0+0 But the PBS server works fine. -- Pozdrawiam, -- Zuzanna K. Filutowska, HPC Systems Administrator Poznan Supercomputing and Networking Center Institute of Bioorganic Chemistry Polish Academy of Sciences Seize the day boys! Make your lifes extraordinary! --John Keating

1 0

QIIME tools for Galaxy (WIP, call for collaborators)
by Lance Parsons 05 Oct '15

05 Oct '15

I was recently asked if I could provide a QIIME analysis pipeline for 16S data in Galaxy using tools in the QIIME pipeline (http://qiime.org/) I did a bit of looking around for existing Galaxy wrappers and found an application that generates the wrappers for QIIME scripts for Galaxy (https://github.com/qiime/qiime-galaxy) This is a very well written application that does a great job of wrapping the QIIME scripts for Galaxy. However, there are a few things about it that don't quite fit my needs. 1. The tools output tgz files of all of the output files. This means that to execute a pipeline, the user would have to download the tgz files, untar, and then upload whichever file(s) are needed for the next step. 2. There is no toolshed repository to install the dependencies for these tools, making it a tricky for administrators to automate and also maintain various versions of QIIME going forward. 3. There are no toolshed versions of the tools themselves, which also makes installation and integration a bit tricky and makes it hard to me to create and manage updates, fixes, tweaks, etc. There are also no tests, etc. For these reasons I decided to investigate the feasibility of using the generated wrappers as a basis for a "toolshed" version of QIIME. If anyone is interested in helping, or has suggestions, or is working on something related, I'd be very happy to collaborate. The repository for the WIP is at https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0. There is also a package on the testtoolshed as well as a first pass at package_qiime_1_9_1 (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_…) -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

1 0

External Pipeline Cleanup
by Ryan G 05 Oct '15

05 Oct '15

Hi all - For various reasons, we've had to implement our pipelines external to Galaxy, but still give Galaxy the ability to run them. I've accomplished this by having the Galaxy tool wrapper call the external app to launch the pipelines. My problem is that when a user aborts the tool, the tool needs to be able to communicate to the pipeline that its need aborted. How can I do this? Is there some way to get a notification that the job has been cancelled before its actually cancelled to perform clean up?

1 0

httpd-ldap vs. ftpd-sql (or: ldap password hash incorrect?)
by Sven E. Templer 05 Oct '15

05 Oct '15

Dear galaxy admins, when I install a galaxy server with httpd/ldap proxy, login, let galaxy put the user credentials into the postgresql database, and try to connect with my email as user to the ftpd (which queries the sql db), I cannot connect (credentials refused). When I do reset the user password in the galaxy interface (to the LDAP matching one), then the pbkdf2 hash salt and key changes in the sql table (galaxy_user), and the user can connect to ftpd. What do I do wrong, I guess it might derive from the httpd/ldap request which galaxy uses to register the user in the sql db? I put the following into httpd.conf: AuthType Basic AuthBasicProvider ldap AuthLDAPURL "ldap:// baz.foo.bar.de/dc=foo,dc=bar,dc=de?uid?sub?(objectClass=person)" AuthLDAPBindDN uid=galaxy,ou=specialusers,dc=foo,dc=bar,dc=de AuthLDAPBindPassword [...] RequestHeader set REMOTE_USER %{AUTHENTICATE_uid}e Do I need to change the RequestHeader? Any hints are appreciated! Sven

1 0

Creating R galaxy wrapper where a function checks file extension
by Martin Vickers 02 Oct '15

02 Oct '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, I've been tasked with getting someone's R script working in our galaxy installation and I'm struggling to resolve an issue with an bioconductor function that appears to not like the galaxy naming convention (e.g. dataset_2.dat) for the bam index file. The R script can run from the command line like this; Rscript script.R input1.xls alignment.bam alignment.bai p so I have created a wrapper using planemo that does this, and that's fine. The problem is, when I run the script in galaxy, I get the following error from R; Error in value[[3L]](cond) : failed to open BamFile: failed to load BAM index file: /tmp/tmpdP2eBC/files/000/dataset_3.dat Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous> In addition: Warning messages: In doTryCatch(return(expr), name, parentenv, handler) : [bam_index_load] fail to load BAM index. Execution halted The function that reads the bam + bai file in the script is; bam = readGAlignments(file=bam.file, index=bai.file, param=ScanBamParam(which=gr.signal)) http://www.rdocumentation.org/packages/GenomicRanges/html/GAlignments.html In the documentation for this function it states the following; "file, index, The path to the BAM file to read, and to the index file of the BAM file to read, respectively. The latter is given /without/ the '.bai' extension. See |scanBam <http://www.rdocumentation.org/packages/Rsamtools/functions/scanBam.html>| for more information." I've played around with this a little and from the command line I've been successfully able to run the script to completion when the input bam file and bai are completely different names; e.g. Rscript script.R input1.xls something.bam meh.bai p but I can't run it if I change the extension, e.g. Rscript script.R input1.xls something.dat meh.dat p Error in value[[3L]](cond) : failed to open BamFile: failed to load BAM index file: /tmp/tmpdP2eBC/files/000/dataset_3.dat Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous> In addition: Warning messages: In doTryCatch(return(expr), name, parentenv, handler) : [bam_index_load] fail to load BAM index. Execution halted Does anyone have any experience of resolving this kind of issue? - -- - -- Dr. Martin Vickers Data Manager/HPC Systems Administrator Institute of Biological, Environmental and Rural Sciences IBERS New Building Aberystwyth University SY23 3FG w: http://www.martin-vickers.co.uk/ e: mjv08(a)aber.ac.uk t: 01970 62 2807 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iQEcBAEBAgAGBQJWDlUgAAoJEHa0a8GkKQgIpUcIAK9OZ9402FRaROF89b6bO1CY d+KP/X0NCxY+AXEddU3lhXi8O9haaKRA/pmxsGyydP0VQfaCrSSlS9nzhncG6Zav gpXe5NjFkwe8a/qaZTVf0YjlYq0sI81S7QAJnzlKXvYSjcz1xNDE9O4qSHqdhefX oI6GWmR+vHi7CTqNuHwpyY9Qoafd5nls0oQoCYDh9YWrn9GZ8fe1qQ9W7iUli/oT dPMYwWs0OL8NfSCuIcz2Dbr1XMt+o5HpQY/afoGWaq3zz5Ie0NhCLKz34bFps3IW 8K0DEROmCTxsXBa54gnnkhVgCI3zkfFCfa47xp13vT2dJALdzYoT6vDj8aC/mes= =ijgr -----END PGP SIGNATURE-----

2 4

Documentation of Galaxy API metadata objects
by Clare Sloggett 01 Oct '15

01 Oct '15

Hi all, I'm wondering if there is any documentation of the JSON structures returned by particular Galaxy REST API calls. Some aspects of the Galaxy API are effectively documented via bioblend's structure, and others are documented at http://galaxy.readthedocs.org/en/master/api_doc.html . However I don't know where to look to find out the actual JSON structure that should be returned by an API call. Does this exist? For instance, if I call (using bioblend) gi.histories.show_history(..., contents=True, details="all"), I will get back a list of dicts where each dict contains dataset metadata, e.g.: { u'api_type': u'file', u'create_time': u'2015-09-28T06:00:19.850421', u'data_type': u'galaxy.datatypes.sequence.FastqSanger', u'file_name': u'/mnt/galaxy/files/000/dataset_2.dat', .... I'd like to know how to find out what dictionary keys to expect for a particular API call in a particular Galaxy version. In this example, IIRC at some point the key 'file_name' changed to 'file_path', but I don't know how to determine which version of Galaxy uses which convention. I don't think this change would necessarily be reflected in bioblend, because bioblend just returns the entire JSON structure as a Python object without caring much about its contents. I also don't know how to derive this information from the Galaxy source code itself (even though I know where the API code is under webapp), so alternatively any guidance on that would be helpful! Thanks, Clare

3 3

October 2015 News of the Galaxy
by Dave Clements 01 Oct '15

01 Oct '15

1 0