November 2013 - galaxy-dev - lists.galaxyproject.org

auto removal of datasets when number of outputs cannot be determined until tool is run
by Nikhil Joshi 18 Nov '13

18 Nov '13

Hi all, So I am using this documentation<http://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_…>to create a tool that has a variable number of output files based upon the input file. I have gotten it to work, however, the intermediate files which I create in the $__new_file_path__ directory (which are then copied into the database as history files) are not being deleted upon tool completion. Is there a way to get galaxy to automatically delete these files, or is there some programmatic way to do it? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 1

Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
by Peter Cock 18 Nov '13

18 Nov '13

Hi Dave, I think this is a new regression from the Test Tool Shed installation framework, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/f2478dc77ccb Tool test results Automated test environment Tools missing tests or test data Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.27 Error ("Invalid file %s specified, ignoring set_environment_for_install action.",/ToolDepsTest/blast+/2.2.27/iuc/package_blast_plus_2_2_27/eab09bc4d63e/env.sh) Is that really a comma in the path? Is this a simple typo in a config file? Note this tool is using: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27/eab09b… (using platform specific actions) -- Meanwhile, over on the main Tool Shed, which I would like to update to use arch-specific actions for the packages (since that is now in the stable Galaxy releases), and switch from BLAST+ 2.2.26 to 2.2.27, we have a different install failure: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/70e7dcbf6573 still using 2.2.26 via shell magic, http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/40c69b76b4… /bin/sh: 1: blastn: not found Peter

2 3

wiki contributions: Admin/Config/ProFTPd_with_AD
by Eric Rasche 16 Nov '13

16 Nov '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'd like to contribute what I've learnt today to this particular page. As it is "locked", I am unsure how to contribute my information, so I'm posting here in the hopes that someone with rights will update it. Formatted in (pandoc compatible) Markdown for your ease. # Configuring ProFTPD with OpenLDAP I've found a set of working options for using ProFTPD with OpenLDAP servers (instead of AD). This configuration file can be modified and placed in `/etc/proftpd/conf.d/galaxy.conf Using the /conf.d/ directory, you can allow the ProFTPd to serve both local users (with PAM authentication) in the main configuration file, AND galaxy users on another port. ``` <VirtualHost xxx.yyy.zzz> RequireValidShell off User galaxy Group galaxy Umask 137 027 AllowOverwrite on # Ensure auth is LDAP AuthPAM off AuthOrder mod_ldap.c # Serve this VirtualHost on port 4000 Port 4000 # LDAP Bind information LDAPServer ldaps://xxx.yyy.zzz/??sub LDAPUsers "ou=People,dc=yyy,dc=zzz" "(uid=%u)" LDAPAuthBinds on # Force those numbers even if LDAP finds a valid UID/GID LDAPDefaultUID 1003 LDAPDefaultGID 1003 LDAPForceDefaultUID on LDAPForceDefaultGID on # Please generate home dir with user/group rwx permissions. Could probably be stricter CreateHome on 770 LDAPGenerateHomedir on 770 # Force this homedir even if LDAP said something different LDAPForceGeneratedHomedir on LDAPGenerateHomedirPrefix "/home/galaxy/galaxy/database/ftp/%u(a)cpt.tamu.edu" # The username is already incorporated in the %u, use this or it will get appended again LDAPGenerateHomedirPrefixNoUsername on TransferLog /var/log/proftpd/xfer-galaxy.log # Cause every FTP user to be "jailed" (chrooted) into their home directory DefaultRoot "/home/galaxy/galaxy/database/ftp/%u(a)cpt.tamu.edu" # Allow users to resume interrupted uploads AllowStoreRestart on # I set these as my passive ports because I run a very strict firewall. Change as needed PassivePorts 49152 50000 </VirtualHost> ``` Notably, this configuration allows a galaxy virtualhost to coexist with the normal FTP capabilities provided by ProFTPd, so users can still access their home directories AND galaxy users can upload to galaxy. Authentication can of course be changed to suit one's needs. # TLS Configuration If you're running the galaxy FTP portion under a VirtualHost, like described above, you'll notice that TLS directives placed in the main proftpd.conf file do not apply to VirtualHosts. As such, you can add a section that looks like this to every VirtualHost that needs to be secured ``` <IfModule mod_tls.c> TLSEngine on TLSLog /var/log/proftpd/tls.galaxy.log # Your cert and private key TLSRSACertificateFile /etc/ssl/certs/my.crt TLSRSACertificateKeyFile /etc/ssl/private/my.key TLSCACertificateFile /etc/ssl/certs/ca.bundle # I've found that this is required for FileZilla TLSOptions NoCertRequest EnableDiags NoSessionReuseRequired # Most clients won't be sending certs TLSVerifyClient off TLSRequired on </IfModule> ``` Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJShTqNAAoJEMqDXdrsMcpVZh8QAL1PvZTtTco+hBeJ+2o9jCyp DpasNtMm0PTKjmBR7Cq5lxNJeGlAcAJmafGKxnf7EEGPhJnw8xWUDwGolmjJmzik o9kl/4vASKQPz6+SX7zqz5Fn2155FrSfgZXruSc3N/56UR6N1mbUdJ0fOdm83vfi hlmVOErujrCkx5S8zSALf7UgTfDT3aPsCyLmy6wy+keNUhpDp5jY2Kvzfm133PIM YIxKM93rPA+IZb99h2BHRNOQjGIcIIM5cWhQ+NSd1lrRmSKZHFvvfVRvKbjb7uxL A+JJ86A3QEsfJm9Krch55KKYpWoom3l53xw+EMLBsO6Surerc6hZcsZsEhPaK/sq GiM33nGZ7DUulJE3OW3lKgilSZY07d3C7ol1fPhovsI20XN3ESdaHAliOSQdT4hn VqomH8qw8rWxKR1omP6MGfvWw1Sg8d8NylvyehylTOwLHO1iRGKT/HmzqEJSEVzb TReA9r85d35tIRlnuuNcHPIdAQreH1fp4Pz1F3sCzn3at9Y2WHNvc9ySHaZXMo6M /KvfdUFGQlDMtWIE3moK1mz5/IsIgDQiZm6Jc+hTcOTXueZ1RTIynLD4n6BHih6r UrdCdHdwIb5WGLyQbO+scn5YybmYSLtbcc5UBS1PvgdQr61/QA9J0XI8SeRUrSX+ gNFhUh3T5bfrnA0eXnaq =GZx9 -----END PGP SIGNATURE-----

2 1

reasonable linux box specs
by Andrew Norman 16 Nov '13

16 Nov '13

Hi all I'd like to set up a local installation of Galaxy on a dedicated linux machine. I've taken a look at the Galaxy "appliance", but since I'm the only one in my lab who will be using it, it's outside my price range. I'm going to be doing RNA seq read mapping and differential expression analysis. Can anyone recommend some minimum specs for the machine? I understand that more memory/cores will enable faster processing, but I'm willing to wait overnight for results to save some cash. Any advice you have about this will be helpful! Thanks, Andy

2 2

how to add openstack in biocloudcentral ?
by Jennifer Jackson 15 Nov '13

15 Nov '13

Hello Zeeshan, Let's post this over to the proper list, galaxy-dev(a)bx.psu.edu, and see if that helps. You could also contact Brad, the git repository owner, directly about questions that involve biocloud. Best, Jen Galaxy team http://wiki.galaxyproject.org/MailingLists#The_lists On 11/15/13 5:24 AM, Zeeshan Ali Shah wrote: > Hi, > I followed the guide from https://github.com/chapmanb/biocloudcentraland > managed to run locally, But only see AWS in combo box, How to add private > Openstack in it? > > -- > > Regards > > Zeeshan Ali Shah > System Administrator - PDC HPC > PhD researcher (IT security) > Kungliga Tekniska Hogskolan > +46 8 790 9115 > http://www.pdc.kth.se/members/zashah > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

1 0

Setting a queue for torque submission
by Jennifer Jackson 15 Nov '13

15 Nov '13

Hello, I am going to move your question over to the galaxt-dev(a)bx.psu.edu mailing list to give it better visibility to the development/local install community. http://wiki.galaxyproject.org/MailingLists#The_lists I didn't find it over here already, but apologies if double posted. Also, since I don't know the best answer, will let others jump in! Best, Jen Galaxy team On 11/14/13 6:35 AM, Xian Su wrote: > Greetings > > I'm having a hard time getting job submission to work via torque. > Posted below is the relevant part of my universe file and job_conf.xml. > > The main cause comes from our torque server requiring a queue to be > requested. I can submit jobs just fine via job .sub files via command > line using qsub -q M30 > > Running jobs via webgui results in the following: > galaxy.jobs.runners.pbs WARNING 2013-11-14 13:54:23,722 (28) > pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all > destinations > galaxy.jobs.runners.pbs ERROR 2013-11-14 13:54:25,724 (28) All > attempts to submit job failed > > On the torque server's pbs logs, I see the corresponding > 11/14/2013 13:54:23;0080;PBS_Server.66159;Req;req_reject;Reject reply > code=15039(No default queue specified MSG=requested queue not found), > aux=0, type=QueueJob, from xians(a)podmt1-100-62.novalocal > > In my universe_wsgi.ini, I'm currently using the settings: > start_job_runners = pbs > default_cluster_job_runner = pbs:///-q M30 -l nodes=1:ppn=12/ > ^I mainly have no idea how to find the correct syntax for this pbs:/// > > But when I comment out default_cluster_job_runner and instead > uncomment the line for the following job conf, I get the same error: > <?xml version="1.0"?> > <job_conf> > <plugins workers="4"> > <plugin id="local" type="runner" > load="galaxy.jobs.runners.local:LocalJobRunner"/> > <plugin id="pbs" type="runner" > load="galaxy.jobs.runners.pbs:PBSJobRunner" workers="2"/> > </plugins> > <handlers default="handlers"> > <handler id="main" tags="handlers"/> > </handlers> > <destinations default="pbs"> > <destination id="local" runner="local"/> > <destination id="pbs" runner="pbs" tags="mycluster"> > <param > id="Resource_List">walltime=72:00:00,nodes=1:ppn=12</param> > <param id="-q">M30</param> > </destination> > </destinations> > <tools> > <tool id="main" handler="main" destination="pbs"/> > </tools> > <limits> > </limits> > </job_conf> > > Thank you in advance for any help. > > Regards, > Xian > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1

Re: [galaxy-dev] Galaxy environment on local resources
by John Chilton 15 Nov '13

15 Nov '13

I am responding to galaxy-dev because others may have better answers than me. On Wed, Nov 13, 2013 at 9:38 PM, Alistair Chilcott <Alistair.Chilcott(a)utas.edu.au> wrote: > John, > > Thanks for the advice, a dose of realism is always useful. :D > > I suspect that our current installation is fiddly primarily because it has been built on top of a Linux distro that was not installed initially just for Galaxy. > It is running SUSE which varies enough in its behaviour from UBUNTU or BioLinux to make the job of interpreting/implementing the various instructions for "how to setup/maintain" much more difficult. > > As I suspected a full blown private cloud is optimistic at best, the question still needed to be asked. I still need to come up with a solution of some kind for the researchers. > > If I could pick your brain for a few minutes ..... > In your experience: > What is the best (easiest to setup and maintain) way to deploy a galaxy instance (including tools the NGS suite in particular) onto a local VM or physical host? The easiest way to setup everything is probably BioLinux (which you mention below). I am not sure if it results in the easiest maintenance however. There is not a turn key solution yet and with things like data managers and the tool shed the Galaxy team is constantly trying to make these things more automated. When I was working on the Galaxy-P project I put together getgalaxyp.org/install.html which is a fairly automated way to configure Galaxy-P environment on Ubuntu and CentOS - dozens of OS packages, custom tools, etc... It was built on CloudBIoLinux. This is a superset of Galaxy, so it could (and should) be trimmed back for a genomics focused setup. This install wouldn't include the data, nginx server, or Galaxy itself but all of this could be added to the procedure - the recipes are available in CloudBioLinux becuase CloudMan uses them. I just need to find a free day sometime to put all the pieces together and document them. > What Linux distribution (and version) is most compatible with galaxy? My recommended stack would be Ubuntu / postgres / nginx. CloudMan runs this stack and so you can always pop a cloud image and see what a working configuration looks like, also the community will likely have the most answers for the configuration as well. Also, even if you don't use the automated configuration scripts in CloudBioLinux - you can always look at them to see what to do as well. There is also this: http://bioteam.net/slipstream/galaxy-edition/. Hope this helps, -John > > > > > Regards, > > Alistair > > > > > -----Original Message----- > From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of John Chilton > Sent: Thursday, 14 November 2013 5:10 AM > To: Alistair Chilcott > Cc: Galaxy Dev > Subject: Re: [galaxy-dev] Galaxy environment on local resources > > On Tue, Nov 12, 2013 at 1:41 AM, Alistair Chilcott <Alistair.Chilcott(a)utas.edu.au> wrote: >> Hello all, >> >> Feel free to point it out if I have missed something obvious, I have done a fair bit of investigation and haven't quite found the solution yet. >> >> We have some hardware that has been around for a while for the purpose of processing Genetic data and other related tasks. >> To this end Galaxy fits the bill nicely in that it enables researchers to analyse data without being Linux geeks. >> >> The problem I have is that while the hand built galaxy server (running on SUSE for historical reasons) we have works to a point it is difficult to maintain and installing new tools and reference genomes is fiddly at best given that our server doesn't conform to the way the instructions for other systems expect it to work. >> >> We have had success using Cloudman on AWS to run training on how to use galaxy, and I would like to know more about how to customise and instance to contain all of the tools (mostly the NGS tools) we need by default. >> >> Ultimately we wont be able to use AWS to process much of the "real" data we have, because of a need to keep the data we are processing in-house due to ethics agreements. Fortunately we do have access to a modest pool of hardware (which is about to get bigger) to implement some kind of private cloud solution. >> >> How would I go about setting up a "private cloud" version of the cloudman style "galaxy instance on demand" system where researchers can start an instance, have it connect to a shared storage volume and process some data then terminate the instance? And is this even the best way to go? >> >> I have found it should be possible to use the scripts to install and configure the galaxy instances but I have not found any information on how to setup the environment that is required to make this work as a private cloud. >> >> Conversely I have found information about some private cloud scenarios such as Eucalyptis, and OpenStack but have not been able to join the dots to determine how to and if I can make the cloudman/galaxy usecase work on it. > > I think the way to do this would be to setup OpenStack and then install and configure CloudMan on your OpenStack cloud. I am not aware of something like CloudMan that deploys Galaxy without an existing cloud infrastructure. I have done this to some extent, I have put together some (dated) scripts for bootstrapping CloudMan on OpenStack still used by my old employer (MSI): > > https://github.com/jmchilton/cloudman_openstack_bootstrap > > A more modern starting point would be to extend the CloudBioLinux deployer instructions and scripts targeting Amazon for OpenStack (this will require some (small?) development effort though): > > https://github.com/chapmanb/cloudbiolinux/blob/master/deploy/cloudman.md > > Either path will take a lot of sys-admin-y work though. > > I hope I don't get in trouble for saying this, but my personal opinion is you should not do this though :). If you consider managing Galaxy fiddly (a fair criticism) than redoing everything an a cloudy manner is going to be several orders of magnitude more fiddly and more work. > I have deployed OpenStack and worked with others doing another deployment, it will take a good system administrator months of effort to do this well and then it will take some amount of her or his time each week ongoing to maintain that layer of the infrastructure. Then instead of installing and maintaining Galaxy, you will have to do that still but with the added complexity of CloudMan. > > CloudMan works well on Amazon because you have a corporation full of amazing engineers operating the infrastructure, you have Dannon and Enis doing a heroic job configuring Galaxy and CloudMan, and dozens of users find and reporting problems. Trying to replicate that all yourself is quite difficult... > >> >> I should mention that I'm primarily a Windows Sys Admin (who dabbles in Linux) who is looking at this due to a lack of a dedicated Linux admin. > > This is going to require more than dabbling... > >> >> At the end of the day I need to be able to setup this system and make it as low maintenance as possible whilst being useful and accessible to the researchers who aren't Linux admins. > > Ekkk.... > >> >> Any advice gratefully accepted. > > It is all going to be on Amazon someday, work to make that happen sooner? > > If you are more comfortable with Windows, you could buy some sort of VMware server solution and try to create a simple gold standard VMware image of something that just contains Galaxy and just spin up single instances on a per project basis. Ultimately though the easiest number of Galaxy instances to manage is 1 :). > > Sorry this e-mail is kind of pessimistic, hopefully more well adjusted people will respond with happier advice. > > -John > >> >> Regards, >> >> Alistair >> >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/

1 0

Failed to generate job destination
by Björn Grüning 15 Nov '13

15 Nov '13

Hi, anyone has seen the following bug in Galaxy November release? galaxy.jobs.handler ERROR 2013-11-15 00:07:41,008 Failed to generate job destination Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 311, in __check_if_ready_to_run self.job_wrappers[job.id].job_destination File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 620, in job_destination return self.job_runner_mapper.get_job_destination(self.params) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/mapper.py", line 163, in get_job_destination self.__cache_job_destination( params ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/mapper.py", line 148, in __cache_job_destination raw_job_destination = self.job_wrapper.tool.get_job_destination( params ) AttributeError: 'NoneType' object has no attribute 'get_job_destination' Thanks, Bjoern

2 3

Galaxy wrappers for CLC Assembly Cell (CLCbio)
by Peter Cock 15 Nov '13

15 Nov '13

Hello all, This is just to announce I am working on a wrapper for "CLC Assembly Cell" which is the CLCbio commercial command line assembly tool suite. http://www.clcbio.com/products/clc-assembly-cell/ Our institute bought a licence primarily for use on plant genomes where other assemblers at the time required too much RAM to complete. This assembler is both fast and low memory, which can be very useful. Wrapper development here: https://github.com/peterjc/pico_galaxy/tree/master/tools/clc_assembly_cell Prototype releases will be on the Test Tool Shed (soon): http://testtoolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell Stable Tool Shed releases will be here (later): http://toolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell I would be interested to hear from anyone else with access to a licensed copy of the tool interested in using it from Galaxy. e.g. Is it reasonable to assume the tools are on the $PATH, or is using a specific environment variable more helpful? Regards, Peter

1 1

LDAP user auth
by Jon Ambler 15 Nov '13

15 Nov '13

Good morning! I am setting up a galaxy instance at our department and I was hoping to use our LDAP database to authenticate users to use galaxy, So my questions are: How would one tell galaxy to refer to the LDAP database? Would this allow new users to be automatically created on first login? Thanks! Sorry about the not-so-great-to-deal-with question, I have dug through a lot of forms and the galaxy pages and didn't get very far... Love galaxy in general though, keep up the great work! Cheers! Jon Jon Ambler MSc Bioinformatics student Bioinformatics and Computational Biology Unit Department of Biochemistry Tree Protection Co-operative Program (TPCP) Research Group Forestry and Agricultural Biotechnology Institute (FABI) University of Pretoria

2 1