October 2013 - galaxy-dev - lists.galaxyproject.org

toolshed contact not working (varscan)
by Geert Vandeweyer 26 Oct '13

26 Oct '13

Hi, I tried to contact the owner of the varscan repository through the main toolshed ( http://toolshed.g2.bx.psu.edu/) "repository actions, contact owner function. Upon submission I got the following error : An error occurred sending your message by email: 'Configuration' object has no attribute 'smtp_ssl' Is this a known issue? The option is not available if I browse the repository from my local galaxy admin pages. In case fcaramia reads this, please contact me, I have some updates to your varscan somatics script, as the default scripts failed to run for me. Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer

2 1

Toolshed reset all repository metadata
by Simon Guest 24 Oct '13

24 Oct '13

We've noticed a funny about resetting repository metadata from the toolshed web interface. My understanding is that this is used to get the toolshed back in sync with the underlying mercurial repo, for example after pushing updates from hg on the command line. It seems that only the person who created the repository can reset metadata. I granted authority to make changes to a colleague, which enables him to push updates, but he doesn't see the reset metadata entry in the menu. This seems like a bug to me. Or is it by design? Also, is there a reason why this function is necessary at all? Wouldn't it be better if the toolshed just noticed updates to the underlying mercurial repo, and reset its own metadata automatically? cheers, Simon

3 3

Defining $GALAXY_CPUS for use in tool wrappers
by Peter Cock 24 Oct '13

24 Oct '13

Hello all, Re: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-October/011557.html Something I raised during the GCC2013, and we talked about via Twitter as well was a Galaxy environment variable for use within Tool Wrappers setting the number of threads/CPUs to use. The idea is that you can configure a default value, and then override this per runner or per tool etc. James Taylor had suggested calling this environment variable $GALAXY_CPUS which seem fine to me (personally I'd say threads not CPUs but I don't really mind). e.g. <command>my_tool --threads "\$GALAXY_CPUS" --input "$input" --output "$output"</command> Everyone I spoke to about this seemed positive about the idea. This would/should be integrated into the various cluster back ends, for example for SGE/OGE the number of threads is already configurable via the DRMAA settings and available as the environment variable $NSLOTS for non-MPI jobs, so my guess is all Galaxy needs to do is something like this: $ hg diff diff -r ce0d758bb995 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py Tue Jul 30 12:30:30 2013 +0100 +++ b/lib/galaxy/jobs/runners/drmaa.py Tue Jul 30 16:10:40 2013 +0100 @@ -43,6 +43,7 @@ # - execute the command # - take the command's exit code ($?) and write it to a file. drm_template = """#!/bin/sh +export GALAXY_CPUS="$NSLOTS" GALAXY_LIB="%s" if [ "$GALAXY_LIB" != "None" ]; then if [ -n "$PYTHONPATH" ]; then Is there an open Trello card for this? Thanks, Peter

5 11

Re: [galaxy-dev] Tool Shed Tests failing with sniffer/ftype problem
by Peter Cock 24 Oct '13

24 Oct '13

Retitling thread to focus on a regression in the (Tool Shed) test framework to do with input file types and sniffers. On Thu, Aug 1, 2013 at 11:47 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > On Thu, Aug 1, 2013 at 11:43 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> On Mon, Jul 29, 2013 at 5:03 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >>> Peter, >>> >>> The issue with seq_rename incorrectly failing functional tests has been >>> resolved in 10266:fe04978dadac, and the test results will be corrected the >>> next time the automated testing framework runs. I'm still looking into the >>> other issues you reported. >>> >>> --Dave B. >> >> Hi Dave, >> >> The seq_rename test results (apparently) from last night are still failing, >> http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/e440681dd17c >> >> AssertionError: Attempting to set field 'new_column' to value '['2']' >> in form 'tool_form' threw exception: cannot find value/label "2" in >> list control control: <SelectControl(new_column=[1])> If the above >> control is a DataToolparameter whose data type class does not include >> a sniff() method, make sure to include a proper 'ftype' attribute to >> the tag for the control within the <test> tag set. > > Same issue here?, > > http://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus/2be36fa7565e > > AssertionError: Attempting to set field 'fasta_file' to value > '['empty.fasta']' in form 'tool_form' threw exception: cannot find > value/label "empty.fasta" in list control control: > <SelectControl(fasta_file=[])> If the above control is a > DataToolparameter whose data type class does not include a sniff() > method, make sure to include a proper 'ftype' attribute to the tag for > the control within the <test> tag set. And another case which looks the same, http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/392279f2e120 AssertionError: Attempting to set field 'fasta_file' to value '['empty.fasta']' in form 'tool_form' threw exception: cannot find value/label "empty.fasta" in list control control: <SelectControl(fasta_file=[])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. Note that for the nlstradamus and effectivet3 tests this is using an empty FASTA file with no sequences in it, so the sniffer will not be able guess this is a FASTA file - which is why the tool text gives the ftype explicitly. I originally used a literally empty fasta file (0 bytes) but as I recall the (non-Tool Shed) framework at the time didn't like me doing that in the test framework, so this empty.fasta file just has a couple of blank lines in it. I am testing with empty FASTA files deliberately - they can often occur in gene/protein selection pipelines where an early part of the pipeline is stringent enough to reject all the sequences. However, the seq_rename sniffer problem is not to do with an 'empty' fasta file - it uses a non-empty tabular file. My guess is somewhere in the test framework the ftype attribute is being lost. This is still working locally, $ ./run_functional_tests.sh -id seq_rename $ ./run_functional_tests.sh -id nlstradamus $ ./run_functional_tests.sh -id effectiveT3 Regards, Peter

2 7

Re: [galaxy-dev] SeqPrep wrapper?
by Lionel Guy 23 Oct '13

23 Oct '13

Hi Brad, (cc-ing the list) I'm more interested in the merging part, and after a quick search on the toolshed, there are no tools to merge reads. Seems that there is a need for a read merger wrapper then... Lionel On 14 Oct 2013, at 13:46 , "Langhorst, Brad" <Langhorst(a)neb.com> wrote: > I would also like to see a seqprep wrapper… > if you need something sooner, cutdadapt is already wrapped and i'm pretty sure there is an existing tool to merge reads as well. > > Brad > -- > Brad Langhorst > Applications and Product Development Scientist > > On Oct 14, 2013, at 6:59 AM, Lionel Guy <guy.lionel(a)gmail.com> > wrote: > >> Hi, >> >> I'm interested in a wrapper for SeqPrep: >> https://github.com/jstjohn/SeqPrep >> Anyone preparing one/aware of someone preparing one? Otherwise I'll put one together. >> >> From its description: "SeqPrep is a program to merge paired end Illumina reads that are overlapping into a single longer read. It may also just be used for its adapter trimming feature without doing any paired end overlap." >> >> Cheers, >> >> Lionel >> >> Lionel Guy >> Uppsala University >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > -- Lionel Guy Fålhagsleden 13, SE-75324 Uppsala | email: guy.lionel(a)gmail.com | mobile: +46 (0)73 9760618 | phone: +46 (0)18 410 7398

1 1

jobid exception
by Rémy Dernat 23 Oct '13

23 Oct '13

Hi, After a big upgrade, I have an error when I submit job to SGE through drmaa Here is the error I get: galaxy.jobs.runners.drmaa WARNING 2013-10-14 11:26:08,075 (15615/68154) job check resulted in InvalidJobException: code 18: The job specified by the 'jobid' does not exist. The job is launch to sge because I see it with "qstat" SGE command. On the galaxy Web interface, the job is stuck in yellow (running state), although it is finished. When I submit job by using "local" destination it works fine. Here is my job_conf.xml file : <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner"/> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="sge_default">  <destination id="sge_default" runner="drmaa"> <param id="nativeSpecification">-q all.q -V</param> </destination> <destination id="local" runner="local"/> </destinations> </job_conf> Any help would be usefull. Regards, Rem

2 6

Empty or 'no peek' output.
by Luobin Yang 22 Oct '13

22 Oct '13

This is strange: when I submit a job through the computers on campus to my locally installed Galaxy, the job finishes without no problem, but when I submit the same job through a computer off-campus, the job generates empty or 'no peek' output. What's causing the problem? Luobin

1 0

Associate jobs to a Project ID
by Jean-Francois Payotte 22 Oct '13

22 Oct '13

Dear Galaxy Gurus, I have been requested to find a way to tag each job (created when running a workflow) with a "Project ID". The user would have the possibility to enter a "Project ID" (maybe as a workflow parameter) and then, some mechanism would link all of these jobs with the "Project ID". Ultimately, the goal would be to create reports on our computer resources utilisation (total CPU time each project consumed). This would be really helpful in understanding the costs associated with the projects analyses. I know I can use the "job_runner_external_id" to retrieve the actual CPU usages but I still need to find a way to associate the jobs with the "Project ID". As of now, I don't see any *clean* way to do this. Anyone would have an idea on how I could implement this feature? Or tell me if it's not possible at all? Many thanks for your help. Galaxy is just incredible! Jean-François Payotte

2 1

No "Upload files to repository" button
by Stewart, Andrew 22 Oct '13

22 Oct '13

No "Upload files to repository" button after creating repository in self-hosted toolshed. screenshot: http://imgur.com/AlQB1jL Any ideas what could be wrong?

1 1

Test Tool Shed - tests failing in subprocess / sniff
by Peter Cock 22 Oct '13

22 Oct '13

Hi Dave, Greg, Based on my tool's overnight test results from the Test Tool Shed, something seems to have been broken to do with sniffing datatypes on upload. e.g. using blastxml format, http://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr Tool test results Automated test environment Tests that failed Tool id: blastxml_to_top_descr Tool version: blastxml_to_top_descr Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 78, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 1327, in run_tool self.submit_form( **kwd ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 1276, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'blastxml_file' to value '['blastp_four_human_vs_rhodopsin.xml']' in form 'tool_form' threw exception: cannot find value/label "blastp_four_human..odopsin.xml" in list control control: <SelectControl(blastxml_file=[])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/runners/local.py", line 83, in queue_job preexec_fn = os.setpgrp ) File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1205, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: e.g. just fasta and tabular, Tool test results Automated test environment Tests that failed Tool id: seq_primer_clip Tool version: seq_primer_clip Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 78, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 1327, in run_tool self.submit_form( **kwd ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 1276, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'primer_fasta' to value '['dop_primers.fasta']' in form 'tool_form' threw exception: cannot find value/label "dop_primers.fasta" in list control control: <SelectControl(primer_fasta=[])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/runners/local.py", line 83, in queue_job preexec_fn = os.setpgrp ) File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1205, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory galaxy.tools.actions.upload_common: INFO: tool upload1 created job id 381 galaxy.jobs: DEBUG: (381) Working directory for job is: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/database/job_working_directory/000/381 galaxy.jobs.handler: DEBUG: (381) Dispatching to local runner galaxy.jobs: DEBUG: (381) Persisting job destination (destination id: local) galaxy.jobs.handler: INFO: (381) Job dispatched galaxy.tools.deps: DEBUG: Building dependency shell command for dependency 'samtools' galaxy.tools.deps: WARNING: Failed to resolve dependency on 'samtools', ignoring galaxy.jobs.runners.local: DEBUG: (381) executing: export GALAXY_SLOTS="1"; python /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/tools/data_source/upload.py /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpDEZCBr/tmpG8dIQ8 /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpDEZCBr/tmpMD8cHQ 404:/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/database/job_working_directory/000/381/dataset_404_files:/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpALDdhE/database/files/000/dataset_404.dat galaxy.datatypes.metadata: DEBUG: Cleaning up external metadata files galaxy.jobs.runners.local: ERROR: failure running job 381 Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/runners/local.py", line 83, in queue_job preexec_fn = os.setpgrp ) File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1205, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: base.twilltestcase: DEBUG: (snip - first test only, rest do the same). Peter

2 1