- galaxy-dev - lists.galaxyproject.org

ACTION needed: fix error in Cloud Cluster Galaxy Trackster DATE needed: needed for workshop on Wednesday 20 November
by Young, Lynn (NIH/OD/ORS) [E] 18 Nov '13

18 Nov '13

Hi, We're using Galaxy Cloud -> New Cloud Cluster to create instances of Galaxy for training on Wednesday 20 November. When we try to view a bam file using Trackster, we get the following error: /opt/sge/default/spool/execd/ip-10-32-111-66/job_scripts/14: line 13: bedtools: command not found /opt/sge/default/spool/execd/ip-10-32-111-66/job_scripts/14: line 13: bedGraphToBigWig: command not found Please let us know how to proceed. Sincerely, Lynn Lynn Young, PhD | Informationist/Bioinformatics Scientist | National Institutes of Health Library | Office of Research Services | Building 10, Room B1L-306D | 10 Center Drive | Bethesda, Maryland 20892-1150 | T. 301-594-6426 | F. 301-402-0254 | E. lynny(a)mail.nih.gov<mailto:lynny@mail.nih.gov> | http://twitter.com/bioinformatics1 Stay connected with the NIH Library NIH Library: http://nihlibrary.nih.gov<http://nihlibrary.nih.gov/> | Facebook: http://www.facebook.com/nihlibrary | Twitter: http://www.twitter.com/nihlib<http://www.twitter.com/nihlib> _____________________________ Amazing Research. Amazing Help.

2 3

BAM/BAI index file test problem on (Test) Tool Shed
by Peter Cock 18 Nov '13

18 Nov '13

Hi guys, I have a new wrapper for samtools idxstats with a working unit test via run_functional_tests.sh run locally or on TravisCI: https://github.com/peterjc/pico_galaxy/tree/master/tools/samtools_idxstats http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-November/017406.html However, this tool's test is failing on the Test Tool Shed: http://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats/6564815949… Tool test results Automated test environment Time tested: ~ 11 hours ago System: Linux 3.8.0-30-generic Architecture: x86_64 Python version: 2.7.4 Galaxy revision: 11284:28469a503b56 Galaxy database version: 117 Tool shed revision: Tool shed database version: Tool shed mercurial version: Tests that failed Tool id: samtools_idxstats Tool version: samtools_idxstats Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Fatal error: Exit code 1 () Input BAI file not found: None Traceback: Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 216, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 28, in do_it self.__verify_outputs( testdef, shed_tool_id, data_list ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 134, in __verify_outputs self.__verify_output( output_tuple, shed_tool_id, elem, maxseconds=maxseconds ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 141, in __verify_output self.verify_dataset_correctness( outfile, hid=elem_hid, attributes=attributes, shed_tool_id=shed_tool_id ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 782, in verify_dataset_correctness self._assert_dataset_state( elem, 'ok' ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 606, in _assert_dataset_state raise AssertionError( errmsg ) AssertionError: Expecting dataset state 'ok', but state is 'error'. Dataset blurb: error It appears that the upload has not generated the *.bai index and assigned it to the variable input_bam.metadata.bam_index (but this works via run_functional_tests.sh for me): $ ls test-data/ex1.* test-data/ex1.bam test-data/ex1.idxstats.tabular $ ./run_functional_tests.sh -id samtools_idxstats ... Ran 1 test in 32.400s OK ... (all fine) (Note that the bai file does not seem to be needed) Tested with this revision, which works: $ hg branch default $ hg log | head changeset: 12309:1df960b4892a tag: tip user: John Chilton <jmchilton(a)gmail.com> date: Sun Nov 10 23:37:56 2013 -0600 summary: PEP-8 cleanups of lib/galaxy/security/__init__.py. Updated to current tip, also works: $ hg branch default [galaxy@ppserver galaxy-central]$ hg log | head changeset: 12321:e12a10e5418d tag: tip user: guerler date: Mon Nov 11 16:00:10 2013 -0500 summary: UI: Fix tooltip placement for masthead icons Are there anyone known differences on the Test Tool Shed which could explain this failure? Thanks, Peter

5 12

Tool unit tests using composite datatypes
by Peter Cock 18 Nov '13

18 Nov '13

Hello all, I'd like to be able to write some simple <test> entries for some of the BLAST+ tools using composite datatypes as input or output (i.e. small BLAST databases). This doesn't seem to be mentioned or hinted at on the wiki: http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax?action=show&redi… Is it possible to use a composite datatype as a test input? If so how? Normal datatypes are loaded into the test history using using the upload tool - does that mean I first need to extend the relevant datatypes to allow them to be uploaded? Example: Run blastp using a small query FASTA file and a small database, check the output (eg tabular). Is it possible to use a composite datatype as a test output? If so how? Example: Run makeblastdb using a small FASTA file, and check the output (a small BLAST database). Thanks, Peter

7 26

tardis job splitter
by McCulloch, Alan 18 Nov '13

18 Nov '13

dear all, There have been a few posts lately about doing distributed computing via Galaxy - i.e. job splitters etc - below a contribution of some ideas we have developed and applied in our work, where we have arranged for some Galaxy tools to execute in parallel on our cluster. We have developed a job-splitter script "tardis.py" (available from https://bitbucket.org/agr-bifo/tardis) which takes marked-up standard unix commands that run an application or tool. The mark-up is prefixed to the input and output command-line options. Tardis strips off the mark-up, and re-writes the commands to refer to split inputs and outputs, which are then executed in parallel e.g. on a distributed compute resource. Tardis knows the output files to expect and how to join them back together. (This was referred to in our GCC2013 talk http://wiki.galaxyproject.org/Events/GCC2013/Abstracts#Events.2FGCC2013.2FA… ) Any reasonable unix based data processing or analysis command may be marked up and run using tardis, though of course tardis needs to know how to split and join the data. Our approach also assumes a "symmetrical" HPC cluster configuration, in the sense that each node sees the same view of the file system (and has the required underlying application installed). We use tardis to support both Galaxy and command-line based compute. Background / design pattern / motivating analogy: Galaxy provides a high level "end to end" view of a workflow; the HPC cluster resource that one uses then involves spraying chunks of data out into parallel processes, usually in the form of some kind of distributed compute cluster - but an end-user looking at a Galaxy history, should ideally not be able to tell whether the workflow was run as a single process on the server, or via many parallel processes on the cluster (apart from the fact that when run in parallel on the cluster, its alot faster!). We noticed that the TCP / IP layered networking protocol stack provides a useful metaphor and design pattern - with the "end to end" topology of a Galaxy workflow corresponding to the transport layer of TCP/ IP; and the distribution of computation across a cluster corresponding to the next TCP/IP layer down - the packet-routing layer. This picture suggested a strongly layered approach to provisioning Galaxy with parallelised compute on split data, and hence to an approach in which the footprint in the Galaxy code-base, of parallel / distributed compute support, should ideally (from the layered-design point of view) be minimal and superficial. Thus in our approach so far, the only footprint is in the tool config files, where we arrange the templating to (optionally) prefix the required tardis mark-up to the input and output command options, and the tardis script name to the command as a whole. tardis then takes care of rewriting and launching all of the jobs, and finally joining the results back together and putting them where galaxy expects them to be (and also housekeeping such as collating and passing up stderr and stdout , and appropriate process exit codes). (For each galaxy job, tardis creates a working folder in a designated scratch area, where input files are uncompressed and split; job files and their output are stored; logging is done etc. Split data is cleaned up at the end unless there was an error in some part of the job, in which case everything is retained for debugging and in some cases restart) (We modify Galaxy tool-configs so that the user can optionally choose to run the tool on our HPC cluster - there are three HPC related input fields, appended to the input section of a tool. Here the user selects whether they want to use our cluster and if so, they specify the chunk size, and can also at that point specify a sampling rate, since we often find it useful to be able to run preliminary analyses on a random sample of (for example) single or paired-end NGS sequence data, to obtain a fairly quick snapshot of the data, before the expense of a complete run. We found it convenient to include support for input sampling in tardis). The pdf document at https://bitbucket.org/agr-bifo/tardis includes a number of examples of marking up a command, and also a simple example of a galaxy tool-config that has been modified to include support for optionally running the job on our HPC cluster via the tardis pre-processor. Known limitations: * we have not yet attempted to integrate our approach with the existing Galaxy job-splitting distributed compute support, partly because of our "layered" design goal (admittedly also partly because of ignorance about its details ! ) * our current implementation is quite naive in the distributed compute API it uses - it supports launching condor job files (and also native sub-processes) - our plan is to replace that with using the drmaa API * we would like to integrate it better with the galaxy type system, probably via a galaxy-tardis wrapper We would be keen to contribute our approach to Galaxy if people are interested. Cheers Alan McCulloch Bioinformatics Software Engineer AgResearch NZ

3 5

auto removal of datasets when number of outputs cannot be determined until tool is run
by Nikhil Joshi 18 Nov '13

18 Nov '13

Hi all, So I am using this documentation<http://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_…>to create a tool that has a variable number of output files based upon the input file. I have gotten it to work, however, the intermediate files which I create in the $__new_file_path__ directory (which are then copied into the database as history files) are not being deleted upon tool completion. Is there a way to get galaxy to automatically delete these files, or is there some programmatic way to do it? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 1

Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
by Peter Cock 18 Nov '13

18 Nov '13

Hi Dave, I think this is a new regression from the Test Tool Shed installation framework, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/f2478dc77ccb Tool test results Automated test environment Tools missing tests or test data Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.27 Error ("Invalid file %s specified, ignoring set_environment_for_install action.",/ToolDepsTest/blast+/2.2.27/iuc/package_blast_plus_2_2_27/eab09bc4d63e/env.sh) Is that really a comma in the path? Is this a simple typo in a config file? Note this tool is using: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27/eab09b… (using platform specific actions) -- Meanwhile, over on the main Tool Shed, which I would like to update to use arch-specific actions for the packages (since that is now in the stable Galaxy releases), and switch from BLAST+ 2.2.26 to 2.2.27, we have a different install failure: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/70e7dcbf6573 still using 2.2.26 via shell magic, http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/40c69b76b4… /bin/sh: 1: blastn: not found Peter

2 3

wiki contributions: Admin/Config/ProFTPd_with_AD
by Eric Rasche 16 Nov '13

16 Nov '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'd like to contribute what I've learnt today to this particular page. As it is "locked", I am unsure how to contribute my information, so I'm posting here in the hopes that someone with rights will update it. Formatted in (pandoc compatible) Markdown for your ease. # Configuring ProFTPD with OpenLDAP I've found a set of working options for using ProFTPD with OpenLDAP servers (instead of AD). This configuration file can be modified and placed in `/etc/proftpd/conf.d/galaxy.conf Using the /conf.d/ directory, you can allow the ProFTPd to serve both local users (with PAM authentication) in the main configuration file, AND galaxy users on another port. ``` <VirtualHost xxx.yyy.zzz> RequireValidShell off User galaxy Group galaxy Umask 137 027 AllowOverwrite on # Ensure auth is LDAP AuthPAM off AuthOrder mod_ldap.c # Serve this VirtualHost on port 4000 Port 4000 # LDAP Bind information LDAPServer ldaps://xxx.yyy.zzz/??sub LDAPUsers "ou=People,dc=yyy,dc=zzz" "(uid=%u)" LDAPAuthBinds on # Force those numbers even if LDAP finds a valid UID/GID LDAPDefaultUID 1003 LDAPDefaultGID 1003 LDAPForceDefaultUID on LDAPForceDefaultGID on # Please generate home dir with user/group rwx permissions. Could probably be stricter CreateHome on 770 LDAPGenerateHomedir on 770 # Force this homedir even if LDAP said something different LDAPForceGeneratedHomedir on LDAPGenerateHomedirPrefix "/home/galaxy/galaxy/database/ftp/%u(a)cpt.tamu.edu" # The username is already incorporated in the %u, use this or it will get appended again LDAPGenerateHomedirPrefixNoUsername on TransferLog /var/log/proftpd/xfer-galaxy.log # Cause every FTP user to be "jailed" (chrooted) into their home directory DefaultRoot "/home/galaxy/galaxy/database/ftp/%u(a)cpt.tamu.edu" # Allow users to resume interrupted uploads AllowStoreRestart on # I set these as my passive ports because I run a very strict firewall. Change as needed PassivePorts 49152 50000 </VirtualHost> ``` Notably, this configuration allows a galaxy virtualhost to coexist with the normal FTP capabilities provided by ProFTPd, so users can still access their home directories AND galaxy users can upload to galaxy. Authentication can of course be changed to suit one's needs. # TLS Configuration If you're running the galaxy FTP portion under a VirtualHost, like described above, you'll notice that TLS directives placed in the main proftpd.conf file do not apply to VirtualHosts. As such, you can add a section that looks like this to every VirtualHost that needs to be secured ``` <IfModule mod_tls.c> TLSEngine on TLSLog /var/log/proftpd/tls.galaxy.log # Your cert and private key TLSRSACertificateFile /etc/ssl/certs/my.crt TLSRSACertificateKeyFile /etc/ssl/private/my.key TLSCACertificateFile /etc/ssl/certs/ca.bundle # I've found that this is required for FileZilla TLSOptions NoCertRequest EnableDiags NoSessionReuseRequired # Most clients won't be sending certs TLSVerifyClient off TLSRequired on </IfModule> ``` Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJShTqNAAoJEMqDXdrsMcpVZh8QAL1PvZTtTco+hBeJ+2o9jCyp DpasNtMm0PTKjmBR7Cq5lxNJeGlAcAJmafGKxnf7EEGPhJnw8xWUDwGolmjJmzik o9kl/4vASKQPz6+SX7zqz5Fn2155FrSfgZXruSc3N/56UR6N1mbUdJ0fOdm83vfi hlmVOErujrCkx5S8zSALf7UgTfDT3aPsCyLmy6wy+keNUhpDp5jY2Kvzfm133PIM YIxKM93rPA+IZb99h2BHRNOQjGIcIIM5cWhQ+NSd1lrRmSKZHFvvfVRvKbjb7uxL A+JJ86A3QEsfJm9Krch55KKYpWoom3l53xw+EMLBsO6Surerc6hZcsZsEhPaK/sq GiM33nGZ7DUulJE3OW3lKgilSZY07d3C7ol1fPhovsI20XN3ESdaHAliOSQdT4hn VqomH8qw8rWxKR1omP6MGfvWw1Sg8d8NylvyehylTOwLHO1iRGKT/HmzqEJSEVzb TReA9r85d35tIRlnuuNcHPIdAQreH1fp4Pz1F3sCzn3at9Y2WHNvc9ySHaZXMo6M /KvfdUFGQlDMtWIE3moK1mz5/IsIgDQiZm6Jc+hTcOTXueZ1RTIynLD4n6BHih6r UrdCdHdwIb5WGLyQbO+scn5YybmYSLtbcc5UBS1PvgdQr61/QA9J0XI8SeRUrSX+ gNFhUh3T5bfrnA0eXnaq =GZx9 -----END PGP SIGNATURE-----

2 1

reasonable linux box specs
by Andrew Norman 16 Nov '13

16 Nov '13

Hi all I'd like to set up a local installation of Galaxy on a dedicated linux machine. I've taken a look at the Galaxy "appliance", but since I'm the only one in my lab who will be using it, it's outside my price range. I'm going to be doing RNA seq read mapping and differential expression analysis. Can anyone recommend some minimum specs for the machine? I understand that more memory/cores will enable faster processing, but I'm willing to wait overnight for results to save some cash. Any advice you have about this will be helpful! Thanks, Andy

2 2

how to add openstack in biocloudcentral ?
by Jennifer Jackson 15 Nov '13

15 Nov '13

Hello Zeeshan, Let's post this over to the proper list, galaxy-dev(a)bx.psu.edu, and see if that helps. You could also contact Brad, the git repository owner, directly about questions that involve biocloud. Best, Jen Galaxy team http://wiki.galaxyproject.org/MailingLists#The_lists On 11/15/13 5:24 AM, Zeeshan Ali Shah wrote: > Hi, > I followed the guide from https://github.com/chapmanb/biocloudcentraland > managed to run locally, But only see AWS in combo box, How to add private > Openstack in it? > > -- > > Regards > > Zeeshan Ali Shah > System Administrator - PDC HPC > PhD researcher (IT security) > Kungliga Tekniska Hogskolan > +46 8 790 9115 > http://www.pdc.kth.se/members/zashah > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

1 0

Setting a queue for torque submission
by Jennifer Jackson 15 Nov '13

15 Nov '13

Hello, I am going to move your question over to the galaxt-dev(a)bx.psu.edu mailing list to give it better visibility to the development/local install community. http://wiki.galaxyproject.org/MailingLists#The_lists I didn't find it over here already, but apologies if double posted. Also, since I don't know the best answer, will let others jump in! Best, Jen Galaxy team On 11/14/13 6:35 AM, Xian Su wrote: > Greetings > > I'm having a hard time getting job submission to work via torque. > Posted below is the relevant part of my universe file and job_conf.xml. > > The main cause comes from our torque server requiring a queue to be > requested. I can submit jobs just fine via job .sub files via command > line using qsub -q M30 > > Running jobs via webgui results in the following: > galaxy.jobs.runners.pbs WARNING 2013-11-14 13:54:23,722 (28) > pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all > destinations > galaxy.jobs.runners.pbs ERROR 2013-11-14 13:54:25,724 (28) All > attempts to submit job failed > > On the torque server's pbs logs, I see the corresponding > 11/14/2013 13:54:23;0080;PBS_Server.66159;Req;req_reject;Reject reply > code=15039(No default queue specified MSG=requested queue not found), > aux=0, type=QueueJob, from xians(a)podmt1-100-62.novalocal > > In my universe_wsgi.ini, I'm currently using the settings: > start_job_runners = pbs > default_cluster_job_runner = pbs:///-q M30 -l nodes=1:ppn=12/ > ^I mainly have no idea how to find the correct syntax for this pbs:/// > > But when I comment out default_cluster_job_runner and instead > uncomment the line for the following job conf, I get the same error: > <?xml version="1.0"?> > <job_conf> > <plugins workers="4"> > <plugin id="local" type="runner" > load="galaxy.jobs.runners.local:LocalJobRunner"/> > <plugin id="pbs" type="runner" > load="galaxy.jobs.runners.pbs:PBSJobRunner" workers="2"/> > </plugins> > <handlers default="handlers"> > <handler id="main" tags="handlers"/> > </handlers> > <destinations default="pbs"> > <destination id="local" runner="local"/> > <destination id="pbs" runner="pbs" tags="mycluster"> > <param > id="Resource_List">walltime=72:00:00,nodes=1:ppn=12</param> > <param id="-q">M30</param> > </destination> > </destinations> > <tools> > <tool id="main" handler="main" destination="pbs"/> > </tools> > <limits> > </limits> > </job_conf> > > Thank you in advance for any help. > > Regards, > Xian > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1