- galaxy-dev - lists.galaxyproject.org

PBS Torque 15007: No permission
by Ben Archuleta 02 Sep '15

02 Sep '15

Hello All, I have Galaxy running on the latest version of Biolinux configured to work with PBS/Torque version Version: 4.2.8, all jobs are being submitted as Debian-galaxy (This is a global user in the cluster and a local user on the Biolinux box). Whenever we submit a job we get the error below: galaxy.jobs.runners.pbs ERROR 2015-08-28 11:52:42,608 Connection to PBS server for submit failed: 15007: No permission We are using: /usr/lib/galaxy-server/eggs/pbs_python-4.4.0-py2.7-linux-x86_64-ucs4.egg to interact with the PBS. Below is how the two identical accounts are configured: Debian-galaxy:x:2136:100::/var/lib/galaxy-server/database:/bin/fasle Debian-galaxy:XXXXXXX:2136:100:Debian-galaxy:/var/lib/galaxy-server/database:/bin/bash Is this a PBS Python error or could this be directly related to the PBS? I can submit jobs on that node by hand through Bash but not through Biolinux. Thank You! Ben

1 0

Renaming pages on the Galaxy wiki
by Peter Cock 02 Sep '15

02 Sep '15

Hi all, I just renamed a page on the wiki to fix a typo: https://wiki.galaxyproject.org/ToolsWitDependenciesInSameRepository now: https://wiki.galaxyproject.org/ToolsWithDependenciesInSameRepository I didn't see any option about setting up a redirection from the old URL (for pages linked to it, or external lines bookmarks etc). I've searched the wiki and fixed the only internal link to this page, but I can't see how to add a redirect. Is there an easy way to move a page & setup a redirection in one go? Peter

2 1

September 2015 Galactic News
by Dave Clements 01 Sep '15

01 Sep '15

1 0

IUC Tools & Collections Hackathon
by Björn Grüning 01 Sep '15

01 Sep '15

We are planning a remote hackathon on September 17th and 18th for Galaxy tool developers to hack on dataset collections. Dataset collections enable MapReduce style workflows in Galaxy and have come a long way over the last year+. Several groups are using dataset collections and newer tools to express workflows of various degrees of complexity that were not possible in Galaxy before. The remote nature of this gives people who don't have the opportunity to come to GCC hackathons (which have always been productive and a lot of fun) a chance to participate in a Galaxy hackathon. Having a well defined topic will allow us to accomplish a lot and let people who don't have particular tasks in mind find something to work on very quickly. We are collecting ideas to work on, but we expect to attract the most participation by simply getting tool developers interested in getting help adding collection support to their existing tools and workflows to show up and participate. If you are interested in participating in the hackathon but not interested in actual tool development - we will assemble a list of smaller, manageable Python and JavaScript tasks to work on and certainly documentation is a chronically lacking for collections so we could use help there and no actual coding would be required. This is the first remote hackathon we have organized and we encourage ideas or advice about how to organize it please let us know. A core group of will be available on IRC all day and we will have 4 google hangouts across those days to organize, answer questions, and report progress. For more information please see our github issue: https://github.com/galaxyproject/tools-iuc/issues/239 Thanks, Bjoern, on behalf of the Galaxy IUC

1 0

Unable to run simple Docker tool
by Mikel Egaña Aranguren 29 Aug '15

29 Aug '15

Hi; I'm trying to develop a Docker based tool, as suggested by a peer-reviewer who might be reading this :P However, I'm having trouble with even the most basic setting, and I don't know what might be wrong, so any help will be much appreciated. I have developed a very simple docker image and corresponding Galaxy tool, so that I get it working before starting with the actual tool, but when I execute it through Galaxy it simply stays executing forever, instead of failing or terminating. My image simply executes a shell script that reads the content of a file and concatenates a string to it. The image: FROM busybox:ubuntu-14.04 MAINTAINER Mikel Egaña Aranguren <mikel.egana.aranguren(a)gmail.com> RUN mkdir /sadi COPY test-io.sh /sadi/ RUN chmod a+x /sadi/test-io.sh ENV PATH $PATH:/sadi The test-io.sh script within the image: #!/bin/sh cat $1 echo "AAA" Invoking the container and executing the script through a normal shell works fine: REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE mikeleganaaranguren/busybox-galaxy-test-io v1 9c2b8bdade1d 54 minutes ago 5.609 MB docker run -i -t mikeleganaaranguren/busybox-galaxy-test-io:v1 BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. ~ # ~ # echo "BBB" > test ~ # test-io.sh test BBB AAA This is the tool file I'm using in Galaxy: <tool id="SADIBUSYBOX" name="SADIBUSYBOX"> <description>IO</description> <requirements> <container type="docker">mikeleganaaranguren/busybox-galaxy-test-io:v1</container> </requirements> <command> test-io.sh $input > $output </command> <inputs> <param name="input" type="data" label="Dataset"/> </inputs> <outputs> <data format="txt" name="output" /> </outputs> <help> </help> </tool> And my job_conf.xml: <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="docker_local"> <destination id="local" runner="local"/> <destination id="docker_local" runner="local"> <param id="docker_enabled">true</param> </destination> </destinations> </job_conf> As I said, when I execute the tool in Galaxy, it simply executes forever, it stays in a "yellow state", till I kill the Galaxy server. The log says: 127.0.0.1 - - [18/Aug/2015:19:08:00 +0200] "GET /tool_runner?tool_id=SADIBUSYBOX HTTP/1.1" 200 - "http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" galaxy.tools.actions INFO 2015-08-18 19:08:07,178 Handled output (77.825 ms) galaxy.tools.actions INFO 2015-08-18 19:08:07,280 Verified access to datasets (12.387 ms) galaxy.tools.execute DEBUG 2015-08-18 19:08:07,307 Tool [SADIBUSYBOX] created job [2] (217.508 ms) 127.0.0.1 - - [18/Aug/2015:19:08:07 +0200] "POST /api/tools HTTP/1.1" 200 - "http://127.0.0.1:8080/tool_runner?tool_id=SADIBUSYBOX" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:07 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" galaxy.jobs DEBUG 2015-08-18 19:08:07,714 (2) Working directory for job is: /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2 galaxy.jobs.handler DEBUG 2015-08-18 19:08:07,721 (2) Dispatching to local runner galaxy.jobs DEBUG 2015-08-18 19:08:07,839 (2) Persisting job destination (destination id: docker_local) galaxy.jobs.runners DEBUG 2015-08-18 19:08:07,849 Job [2] queued (128.176 ms) galaxy.jobs.handler INFO 2015-08-18 19:08:07,897 (2) Job dispatched galaxy.jobs.command_factory INFO 2015-08-18 19:08:08,163 Built script [/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/tool_script.sh] for tool command[/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/tool_script.sh] galaxy.jobs.runners DEBUG 2015-08-18 19:08:08,322 (2) command is: sudo docker inspect mikeleganaaranguren/busybox-galaxy-test-io:v1 > /dev/null 2>&1 [ $? -ne 0 ] && sudo docker pull mikeleganaaranguren/busybox-galaxy-test-io:v1 > /dev/null 2>&1 sudo docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy:ro -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/tools/catDocker:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/tools/catDocker:ro -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2:rw -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/files:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/files:rw -w /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2 --net none --rm -u 1001 mikeleganaaranguren/busybox-galaxy-test-io:v1 /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/tool_script.sh; return_code=$?; python "/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/set_metadata_3aWBf9.py" "/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/tmp/tmpH2gy3X" "/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/galaxy.json" "/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/metadata_in_HistoryDatasetAssociation_2_WxbKNP,/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/metadata_kwds_HistoryDatasetAssociation_2_p8UK91,/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/metadata_out_HistoryDatasetAssociation_2_wPm09U,/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/metadata_results_HistoryDatasetAssociation_2_irSo7t,/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/files/000/dataset_2.dat,/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/metadata_override_HistoryDatasetAssociation_2_1glVfb" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-08-18 19:08:08,323 (2) executing job script: /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/galaxy_2.sh galaxy.jobs DEBUG 2015-08-18 19:08:08,375 (2) Persisting job destination (destination id: docker_local) 127.0.0.1 - - [18/Aug/2015:19:08:11 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:21 +0200] "GET /history/current_history_json HTTP/1.1" 200 - "http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:21 +0200] "GET /api/histories/f2db41e1fa331b3e/contents?dataset_details=ebfb8f50c6abde6d%2C33b43b4e7093c91f%2Ca799d38679e985db%2C5969b1f7201f12ae%2Cdf7a1f0c02a5b08e%2C0a248a1f62a0cc04%2C03501d7626bd192f%2C3f5830403180d620%2Ce85a3be143d5905b%2Cc9468fdb6dc5c5f1%2C2a56795cad3c7db3 HTTP/1.1" 200 - "http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:25 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:29 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:33 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:38 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:42 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" 127.0.0.1 - - [18/Aug/2015:19:08:46 +0200] "GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1" 200 - " http://127.0.0.1:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" Thanks! -- Mikel Egaña Aranguren, Ph.D. http://mikeleganaaranguren.com

2 5

Get dataset/API ids for a dataset
by Asma Riyaz 28 Aug '15

28 Aug '15

Hello Galaxy-dev, I thank you so much for all the help you have given me. I have a question about data set ids in galaxy. As a background, I am running my own galaxy instance on a server. A pipeline implemented in the lab produces the following files in the history: 1) 2 BAM files 2) A JSON file My goal is to use this JSON file to pass the path/URL of bam files into a custom JS we wrote for visualization purpose. This JSON file contains among many other details the paths/URLs to the above bam files. I am using JSON filetypes to send data to the JS visualization within Galaxy. To do this, I have my own JS which loads a BAM file from URL provided into an IGV.js track. IGV.js, which is responsible for making the tracks, expects a valid URL which is updated in the JSON file in this manner: 1) Extract the API_key and history id from a loaded BAM file 2) Edit the JSON file to reflect the BAM file's dataset id to be something like this: { "CLL-HL_pilot.r1.fastq": { "DNA": "/datasets/36ddb788a0f14eb3/display?to_ext=bam", ... This works fine if I know the API Key for bam files. When a pipeline executes dataset ids are generated for each output. I want to access and include these ids in the JSON file and load the updated JSON file into the history with the bams. Is there a way to get the ids from the history in this manner? Thank you, Asma

2 12

login problems when using a proxy and serving at a subdirectory
by Bernau, Christoph 28 Aug '15

28 Aug '15

Dear all, I have a problem with configuring my galaxy instance with a proxy server when I try to serve at a subdirectory. I followed the description at https://wiki.galaxyproject.org/Admin/Config/nginxProxy The basic configuration works just fine. When I use the procedure described in the section "serving galaxy at a subdirectory", the proxying principally also works. However, when registered users try to log in to galaxy, galaxy will simply directs them to the starting page (which I think is what normally happens after logging in) without displaying an error message. Nonetheless, users do not have access to the functionalities which only logged in users can use. E.g. each time they try to use the "workflows"-button they are told to log in. Could you please provide any help on this issue? Cheers, Christoph

3 3

Rename output from a repeat
by Ulf Schaefer 27 Aug '15

27 Aug '15

Hi all We frequently use the syntax below to rename outputs of workflows that we run in batch. It is convenient to have sample names from fastqs carried over to sams, bams, vcfs, etc. #{input1 | basename}.bam This does not seem to be working for inputs that are in repeats, e.g. the VelvetOptimiser. Does anybody know if there is a syntax to make this work, maybe #{repeatname[0].input1 | basename}.bam ? Thanks a lot for your help Ulf ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************

3 2

samtools dependency changes ?
by Wolfgang Maier 27 Aug '15

27 Aug '15

Dear all, with older Galaxies (prior to latest_15.03 I think), you could satisfy Galaxy's samtools dependence for indexing bam files by having a samtools executable in tool-dependencies/samtools/default/bin (with the tool-dependencies directory declared as tool_dependency_dir in galaxy.ini of course). Now (checked with latest_15.03 and .05), this is not working any more! The executable will still be used during bam uploads, but not when a bam file gets created by a tool. The reason is that before the job runner (tested this with the local job runner only) used to build the dependency shell command for dependency 'samtools' before finishing a job, but now the job wrapper finish method fails because it naively expects to find samtools on $PATH. Best, Wolfgang

3 3

Galaxy and PBS/Torque Cluster
by Ryan Johnson 27 Aug '15

27 Aug '15

Hello All, We are using a Bio-Linux 8 Galaxy instance and are currently trying to integrate in into a PBS/Torque cluster. Currently we are able to run jobs locally but not via PBS. Can anyone provide us with a step by step guide to configure Galaxy and the PBS. I have included some of the errors we are receiving below: galaxy.datatypes.metadata DEBUG 2015-08-19 14:56:42,920 Failed to cleanup MetadataTempFile temp files from /usr/lib/galaxy-server/database/job_working_directory/000/96 /metadata_out_HistoryDatasetAssociation_100_Kk1ajw: No JSON object could be decoded galaxy.jobs.runners.pbs ERROR 2015-08-19 14:56:42,960 Connection to PBS server for submit failed: 0: no error We can not figure out what is stopping our jobs from entering the PBS queue. Below is our job_conf.xml. <?xml version="1.0"?> <job_conf> <plugins workers="2">   <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> </plugins> <handlers> <handler id="main"/>  </handlers> <destinations default="defaultq">  <destination id="defaultq" runner="pbs" tags="cluster"> <param id="-q">defaultq</param> <param id="-l">walltime=24:00:00</param> </destination> </destinations> </job_conf> Thank you very much for all the assistance! Regards, Ryan -- Ryan S. Johnson, PhD Applications Scientist Center for Advanced Research Computing P: (650) 430-6194 E:rjohns03@carc.unm.edu

2 1