- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Galaxy environment on local resources
by John Chilton 15 Nov '13

15 Nov '13

I am responding to galaxy-dev because others may have better answers than me. On Wed, Nov 13, 2013 at 9:38 PM, Alistair Chilcott <Alistair.Chilcott(a)utas.edu.au> wrote: > John, > > Thanks for the advice, a dose of realism is always useful. :D > > I suspect that our current installation is fiddly primarily because it has been built on top of a Linux distro that was not installed initially just for Galaxy. > It is running SUSE which varies enough in its behaviour from UBUNTU or BioLinux to make the job of interpreting/implementing the various instructions for "how to setup/maintain" much more difficult. > > As I suspected a full blown private cloud is optimistic at best, the question still needed to be asked. I still need to come up with a solution of some kind for the researchers. > > If I could pick your brain for a few minutes ..... > In your experience: > What is the best (easiest to setup and maintain) way to deploy a galaxy instance (including tools the NGS suite in particular) onto a local VM or physical host? The easiest way to setup everything is probably BioLinux (which you mention below). I am not sure if it results in the easiest maintenance however. There is not a turn key solution yet and with things like data managers and the tool shed the Galaxy team is constantly trying to make these things more automated. When I was working on the Galaxy-P project I put together getgalaxyp.org/install.html which is a fairly automated way to configure Galaxy-P environment on Ubuntu and CentOS - dozens of OS packages, custom tools, etc... It was built on CloudBIoLinux. This is a superset of Galaxy, so it could (and should) be trimmed back for a genomics focused setup. This install wouldn't include the data, nginx server, or Galaxy itself but all of this could be added to the procedure - the recipes are available in CloudBioLinux becuase CloudMan uses them. I just need to find a free day sometime to put all the pieces together and document them. > What Linux distribution (and version) is most compatible with galaxy? My recommended stack would be Ubuntu / postgres / nginx. CloudMan runs this stack and so you can always pop a cloud image and see what a working configuration looks like, also the community will likely have the most answers for the configuration as well. Also, even if you don't use the automated configuration scripts in CloudBioLinux - you can always look at them to see what to do as well. There is also this: http://bioteam.net/slipstream/galaxy-edition/. Hope this helps, -John > > > > > Regards, > > Alistair > > > > > -----Original Message----- > From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of John Chilton > Sent: Thursday, 14 November 2013 5:10 AM > To: Alistair Chilcott > Cc: Galaxy Dev > Subject: Re: [galaxy-dev] Galaxy environment on local resources > > On Tue, Nov 12, 2013 at 1:41 AM, Alistair Chilcott <Alistair.Chilcott(a)utas.edu.au> wrote: >> Hello all, >> >> Feel free to point it out if I have missed something obvious, I have done a fair bit of investigation and haven't quite found the solution yet. >> >> We have some hardware that has been around for a while for the purpose of processing Genetic data and other related tasks. >> To this end Galaxy fits the bill nicely in that it enables researchers to analyse data without being Linux geeks. >> >> The problem I have is that while the hand built galaxy server (running on SUSE for historical reasons) we have works to a point it is difficult to maintain and installing new tools and reference genomes is fiddly at best given that our server doesn't conform to the way the instructions for other systems expect it to work. >> >> We have had success using Cloudman on AWS to run training on how to use galaxy, and I would like to know more about how to customise and instance to contain all of the tools (mostly the NGS tools) we need by default. >> >> Ultimately we wont be able to use AWS to process much of the "real" data we have, because of a need to keep the data we are processing in-house due to ethics agreements. Fortunately we do have access to a modest pool of hardware (which is about to get bigger) to implement some kind of private cloud solution. >> >> How would I go about setting up a "private cloud" version of the cloudman style "galaxy instance on demand" system where researchers can start an instance, have it connect to a shared storage volume and process some data then terminate the instance? And is this even the best way to go? >> >> I have found it should be possible to use the scripts to install and configure the galaxy instances but I have not found any information on how to setup the environment that is required to make this work as a private cloud. >> >> Conversely I have found information about some private cloud scenarios such as Eucalyptis, and OpenStack but have not been able to join the dots to determine how to and if I can make the cloudman/galaxy usecase work on it. > > I think the way to do this would be to setup OpenStack and then install and configure CloudMan on your OpenStack cloud. I am not aware of something like CloudMan that deploys Galaxy without an existing cloud infrastructure. I have done this to some extent, I have put together some (dated) scripts for bootstrapping CloudMan on OpenStack still used by my old employer (MSI): > > https://github.com/jmchilton/cloudman_openstack_bootstrap > > A more modern starting point would be to extend the CloudBioLinux deployer instructions and scripts targeting Amazon for OpenStack (this will require some (small?) development effort though): > > https://github.com/chapmanb/cloudbiolinux/blob/master/deploy/cloudman.md > > Either path will take a lot of sys-admin-y work though. > > I hope I don't get in trouble for saying this, but my personal opinion is you should not do this though :). If you consider managing Galaxy fiddly (a fair criticism) than redoing everything an a cloudy manner is going to be several orders of magnitude more fiddly and more work. > I have deployed OpenStack and worked with others doing another deployment, it will take a good system administrator months of effort to do this well and then it will take some amount of her or his time each week ongoing to maintain that layer of the infrastructure. Then instead of installing and maintaining Galaxy, you will have to do that still but with the added complexity of CloudMan. > > CloudMan works well on Amazon because you have a corporation full of amazing engineers operating the infrastructure, you have Dannon and Enis doing a heroic job configuring Galaxy and CloudMan, and dozens of users find and reporting problems. Trying to replicate that all yourself is quite difficult... > >> >> I should mention that I'm primarily a Windows Sys Admin (who dabbles in Linux) who is looking at this due to a lack of a dedicated Linux admin. > > This is going to require more than dabbling... > >> >> At the end of the day I need to be able to setup this system and make it as low maintenance as possible whilst being useful and accessible to the researchers who aren't Linux admins. > > Ekkk.... > >> >> Any advice gratefully accepted. > > It is all going to be on Amazon someday, work to make that happen sooner? > > If you are more comfortable with Windows, you could buy some sort of VMware server solution and try to create a simple gold standard VMware image of something that just contains Galaxy and just spin up single instances on a per project basis. Ultimately though the easiest number of Galaxy instances to manage is 1 :). > > Sorry this e-mail is kind of pessimistic, hopefully more well adjusted people will respond with happier advice. > > -John > >> >> Regards, >> >> Alistair >> >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/

1 0

Failed to generate job destination
by Björn Grüning 15 Nov '13

15 Nov '13

Hi, anyone has seen the following bug in Galaxy November release? galaxy.jobs.handler ERROR 2013-11-15 00:07:41,008 Failed to generate job destination Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 311, in __check_if_ready_to_run self.job_wrappers[job.id].job_destination File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 620, in job_destination return self.job_runner_mapper.get_job_destination(self.params) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/mapper.py", line 163, in get_job_destination self.__cache_job_destination( params ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/mapper.py", line 148, in __cache_job_destination raw_job_destination = self.job_wrapper.tool.get_job_destination( params ) AttributeError: 'NoneType' object has no attribute 'get_job_destination' Thanks, Bjoern

2 3

Galaxy wrappers for CLC Assembly Cell (CLCbio)
by Peter Cock 15 Nov '13

15 Nov '13

Hello all, This is just to announce I am working on a wrapper for "CLC Assembly Cell" which is the CLCbio commercial command line assembly tool suite. http://www.clcbio.com/products/clc-assembly-cell/ Our institute bought a licence primarily for use on plant genomes where other assemblers at the time required too much RAM to complete. This assembler is both fast and low memory, which can be very useful. Wrapper development here: https://github.com/peterjc/pico_galaxy/tree/master/tools/clc_assembly_cell Prototype releases will be on the Test Tool Shed (soon): http://testtoolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell Stable Tool Shed releases will be here (later): http://toolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell I would be interested to hear from anyone else with access to a licensed copy of the tool interested in using it from Galaxy. e.g. Is it reasonable to assume the tools are on the $PATH, or is using a specific environment variable more helpful? Regards, Peter

1 1

LDAP user auth
by Jon Ambler 15 Nov '13

15 Nov '13

Good morning! I am setting up a galaxy instance at our department and I was hoping to use our LDAP database to authenticate users to use galaxy, So my questions are: How would one tell galaxy to refer to the LDAP database? Would this allow new users to be automatically created on first login? Thanks! Sorry about the not-so-great-to-deal-with question, I have dug through a lot of forms and the galaxy pages and didn't get very far... Love galaxy in general though, keep up the great work! Cheers! Jon Jon Ambler MSc Bioinformatics student Bioinformatics and Computational Biology Unit Department of Biochemistry Tree Protection Co-operative Program (TPCP) Research Group Forestry and Agricultural Biotechnology Institute (FABI) University of Pretoria

2 1

Re: [galaxy-dev] Converting .gff3 to 12-column .bed
by Jennifer Jackson 15 Nov '13

15 Nov '13

Hi Lindsay, Nice to hear that Vipin's server worked out for you. A reduction in the number of lines is completely expected. You should have the same number of lines in the BED12 file as you have unique "ID"s in the GFF3 file, if you want to double check. This is the same as the number of unique transcripts in any of the files, if summed up correctly. Why is this? A GFF3 file has one line per feature. Part of an exon, etc. with further (this can vary) annotation that tells you if it is 5' UTR, or CDS, etc. When converting GFF3 -> BED12, all the features for a particular transcript are rolled up into a single line of output. Transcripts will have a unique "ID" by GFF3 specification - and all features that belong to that transcript will have it in the GFF3 file's attributes field (it is usually the first attribute listed). When you converted just GFF3 -> BED6, each feature was individually converted into a distinct line of BED formatted output, there was no summing up to group the information for the transcript as a whole. The Galaxy wiki has more - GFF3 can represent other kinds of data, and that is covered in the link-outs to the original spec, but this is the gist of the meaning for your specific data. Glad I could help before, and that Vipin had the tool online & available to simplify things! Best, Jen Galaxy team On 11/12/13 4:12 PM, Lindsay Rutter wrote: > Hello Jennifer: > > Thank you for your very helpful and detailed explanation! > > I ended up using the site that Vipin provided in his message > (https://galaxy.cbio.mskcc.org/tool_runner?tool_id=fml_gff2bed) > > Indeed, it produced a file with 12 columns that appears to be in a > reasonable bed format (including the 10th column being an integer, > with the 11th and 12th column consisting of that integer number of > items separated by commas). > > However, I noticed that the number of lines in the file went > from 183,748 (in the .gff3 file) to 11,506 (in the 12-column .bed file). > > In your opinion, does this seem like a reasonable reduction in number > of lines? I was not expecting that (in fact, I was expecting the > number of lines would stay the same, as they did going from .gff3 to > 6-column .bed file), but I am quite inexperienced using these files. > > Thanking you... > Lindsay > > > > > On Mon, Nov 11, 2013 at 11:15 AM, Jennifer Jackson <jen(a)bx.psu.edu > <mailto:jen@bx.psu.edu>> wrote: > > Hello, > > There are no tools directly on the public Galaxy site to transform > a GFF3 dataset into a BED12 dataset. However, the Tool Shed has a > repository called ' fml_gff3togtf' that includes a tool for this > purpose, for use in a local install. The description is a bit > bothersome in that it a slightly incorrect datatype statement, so > be sure to test out the results. (the word "wiggle" has no place > in this statement: " > gff3_to_bed_converter.py: This tool converts gene transcript annotation from GFF3 format to UCSC wiggle 12 column BED format.") > http://getgalaxy.org > http://usegalaxy.org/toolshed > > I see your post at Biostar, and it might be helpful to let you > know what a BED12 file represents (plus I'll post this there, may > help others): > http://www.biostars.org/p/85869/ > > A BED12 file describes the complete, often spliced, alignment of a > sequence to a reference genome. This does not include minor base > variation, it is a macro alignment. You can think of each of the > blocks as being "exons", although there is no magic here - if the > sequence or genome had quality problems, or significant variation > (large insertion or deletion), that could cause the alignment to > fragment as well. > Here is the data description: > http://wiki.galaxyproject.org/Learn/Datatypes#Bed > > To see examples, at UCSC (genome.ucsc.edu > <http://genome.ucsc.edu>), EST or mRNA track will have this as the > primary table format. All gene track can also be in BED12 format, > or in a related one, genePred: > http://genome.ucsc.edu/FAQ/FAQformat.html#format9 > > UCSC also has line-command utilities to convert between the > formats, pre-compiled versions are here: > http://hgdownload.cse.ucsc.edu/downloads.html#source_downloads > > Either way, you can convert the data, then load up into the public > Galaxy (usegalaxy.org <http://usegalaxy.org>) and proceed with > your analysis. BEDTools works well with BED12 files. There is > definitely information loss attempting to transform BED6 -> BED12, > as the global alignment is lost. And adjusting attributes such as > score or name are often a preference, so you can alter these > however you want, as long as the attribute formatting rules for > the columns are followed. > > Hopefully this helps, > > Jen > Galaxy team > > > On 11/9/13 3:29 PM, lrutter @iastate.edu <http://iastate.edu> wrote: >> Hello Galaxy: >> >> I am trying overall to convert a .gff3 file to 12-column .bed file. >> >> I first tried GFF-to-BED converter, but it gave a 6-column .bed file. >> >> Then, I tried BED-to-bigBed converter by inputting the 6-column >> .bed file. I get an error "Unspecified genome build, click the >> pencil icon in the history item to set the genome build". >> >> So, I click the pencil icon, and see 4 tabs at the top. I set the >> "Attributes" tab as in the attached image (Attributes.png). >> >> But then, when I select "Convert Format", I am only seeing an >> option that outputs .bed12 file as "Convert Genomic Intervals to >> Strict BED12". I am a bit confused about this because I specified >> the input file as a .bed file (and not genomic intervals, unless >> I am misunderstanding something). >> >> In any case, when I select "Convert Genomic Intervals to Strict >> BED12", I do get a .bed file with 12 columns. But I would like to >> ask if I may have lost information going from the .gff3 to >> .bed(6) to .bed(12)? >> >> (I feel that scores were all set to "0" from .gff3 to .bed(6), >> and columns 10, 11, 12 (block counts, sizes, and starting >> positions) were all set to zero going from .bed(6) to .bed(12)). >> >> If I am correct that there is information loss, is there a system >> in Galaxy to prevent this, and transfer as much information as >> possible from .gff3 to .bed(12)? >> >> Thank you. >> L. Rutter >> >> ** Below is a head of my three files (the species is P. dominula): >> >> .gff3 file >> >> ##gff-version 3 >> ##date Mon Nov 4 14:54:42 2013 >> ##source gbrowse gbgff gff3 dumper >> PdomScaf0001 maker gene 15 1963 . - . >> Name=PdomGene00025;ID=1;Dbxref=MAKER:maker-PdomScaf0001-snap-gene-0.274 >> PdomScaf0001 maker mRNA 15 1963 . - . >> Name=PdomMRNA00025.1;Parent=1;ID=2;_QI=216%7C0%7C0.2%7C0.6%7C0.5%7C0.6%7C5%7C0%7C98;_eAED=0.43;_AED=0.43;Dbxref=MAKER:maker-PdomScaf0001-snap-gene-0.274-mRNA-1 >> PdomScaf0001 maker exon 15 100 -0.094 - . >> Parent=2;ID=3 >> PdomScaf0001 maker CDS 15 100 . - 2 >> Parent=2;ID=4 >> PdomScaf0001 maker exon 223 300 21.8 - . >> Parent=2;ID=5 >> PdomScaf0001 maker CDS 223 300 . - 2 >> Parent=2;ID=6 >> PdomScaf0001 maker exon 717 765 22.4 - . >> Parent=2;ID=7 >> >> .bed(6) file >> >> PdomScaf0001 14 1963 gene 0 - >> PdomScaf0001 14 1963 mRNA 0 - >> PdomScaf0001 14 100 exon 0 - >> PdomScaf0001 14 100 CDS 0 - >> PdomScaf0001 222 300 exon 0 - >> PdomScaf0001 222 300 CDS 0 - >> PdomScaf0001 716 765 exon 0 - >> PdomScaf0001 716 765 CDS 0 - >> PdomScaf0001 906 947 exon 0 - >> PdomScaf0001 906 947 CDS 0 - >> >> .bed(12) file >> >> PdomScaf0001 14 1963 gene 0 - 14 1963 >> 0 0 , , >> PdomScaf0001 14 1963 mRNA 0 - 14 1963 >> 0 0 , , >> PdomScaf0001 14 100 exon 0 - 14 100 >> 0 0 , , >> PdomScaf0001 14 100 CDS 0 - 14 100 >> 0 0 , , >> PdomScaf0001 222 300 exon 0 - 222 300 >> 0 0 , , >> PdomScaf0001 222 300 CDS 0 - 222 300 >> 0 0 , , >> PdomScaf0001 716 765 exon 0 - 716 765 >> 0 0 , , >> PdomScaf0001 716 765 CDS 0 - 716 765 >> 0 0 , , >> PdomScaf0001 906 947 exon 0 - 906 947 >> 0 0 , , >> PdomScaf0001 906 947 CDS 0 - 906 947 >> 0 0 , , >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > -- > Jennifer Hillman-Jackson > http://galaxyproject.org > > -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1

Errors running DRMAA and PBS on remote server running Torque 4
by Ganote, Carrie L 14 Nov '13

14 Nov '13

Hi List, I've sprouted some grays in the last week after my Galaxy instances all simultaneously ceased to submit jobs to our main cluster. Some Galaxy instances are running the PBS job runner, and others use DRMAA. For the DRMAA runner I was getting: galaxy.jobs.runners ERROR 2013-10-15 08:40:14,942 (1024) Unhandled exception calling queue_job Traceback (most recent call last): File "galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 60, in run_next method(arg) File "galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 188, in queue_job external_job_id = self.ds.runJob(jt) File "build/bdist.linux-x86_64/egg/drmaa/__init__.py", line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File "build/bdist.linux-x86_64/egg/drmaa/helpers.py", line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "build/bdist.linux-x86_64/egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) InternalException: code 1: (qsub) cannot access script file: Unauthorized Request MSG=can not authorize request (0-Success) And in my PBS runner: galaxy.jobs.runners.pbs WARNING 2013-10-14 17:13:07,319 (550) pbs_submit failed (try 1/5), PBS error 15044: Resources temporarily unavailable To give some background, I had recently requested a new virtual machine to put my test/dev Galaxy on. I copied our production Galaxy to this new VM. I secured a new domain name for it and set it running. Everything was going well until I tried to hook it up to the cluster; at first I got an error saying that I didn't have permission to submit jobs. Makes sense, the new VM was not a qualified submit host for the cluster. I asked the sysadmins to add the VM as a submit host to the cluster using qmgr. As soon as this was done, not only could I still not submit jobs from the test Galaxy, but no Galaxy was able to submit jobs to the cluster. The issue isn't with Galaxy here but the underlying calls that it makes - for drmaa, I tracked it back to pbs-drmaa/bin/drmaa-run. For PBS, I'm sure it's somewhere in with libtorque. In every case, I could call qsub from the command line and it would correctly submit jobs, which was more perplexing. I re-installed python, drmaa.egg, pbs-drmaa, and rebooted the VM. I of course restarted Galaxy with each step, to no avail. I worked with the admins to see what was happening in the server logs, but the same cryptic error showed up - cannot authorize request. I've had this issue before in the past, more or less, but usually just gave up on it. It seemed to come and go sporadically, but rebooting the clusters seemed to help. This time, with our production server no longer functioning, I begged for help and the admins looked through the pbs_server config but couldn't find any mistypes or problems. Reloading the config by sending hangup signals to pbs_server didn't help. Then we tried pausing the scheduler and restarting pbs_server completely - and eureka, all problems went away. PBS and DRMAA runners are back up and working fine. This really seems to be a bug in Torque 4.1.5.1. I hope this saves someone a lot of headache! Newer versions of Torque may be the answer. I would also advise against making changes to the pbs_server configuration while in production - we have monthly maintenance, and I don't think I'll ever request changes when there won't be an immediate reboot to flush the server! Cheers, Carrie

2 3

Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
by Dave Bouvier 14 Nov '13

14 Nov '13

Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. On 08/26/2013 09:44 AM, Peter Cock wrote: > (Off list) > > Hi Dave, Greg, > > Just a reminder - without a bit more information about the Tool Shed > cluster I am stuck about how to debug this install issue. > > Thanks, > > Peter > > > On Wed, Aug 21, 2013 at 2:05 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >>> Peter, >>> >>> Here is the relevant log output for the installation of ncbi_blast_plus. A >>> quick google informs me that exit code 64 means "command line usage error", >>> for what it's worth. >>> >>> ... >>> tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row >>> (or updating an existing row) for repository 'package_blast_plus_2_2_26' in >>> the tool_shed_repository table, status set to 'New'. >>> >>> Warning: local() encountered an error (return code 64) while executing ' >>> if [[ "$(uname)" == "Linux" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi && >>> if [[ "$(arch)" == "x86_64" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi && >>> echo Fetching $FILENAME && >>> if [[ "$(uname)" == "Linux" ]]; then wget >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then curl -O >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> tar -zxvf $FILENAME && >>> echo Downloaded and decompressed >>> ' >>> >> >> Thanks Dave, >> >> I've made a note on this Trello card about exposing the log information >> of unsuccessful builds on the Tool Shed, >> https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-pa… >> >> Can you tell me what version of Linux this is, and which shell? >> >> $ uname >> ... >> >> $ arch >> ... >> >> $ echo $0 >> ... >> >> That might give me a clue about the nature of the failure. My >> hunch is that I am using something bash specific. >> >> The simplest solution would be if you could run each of these >> command by hand to see if it triggers the same error: >> >> if [[ "$(uname)" == "Linux" ]]; then export FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi >> if [[ "$(arch)" == "x86_64" ]]; then export FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi >> if [[ "$(uname)" == "Darwin" ]]; then export FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi >> echo Fetching $FILENAME >> if [[ "$(uname)" == "Linux" ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> if [[ "$(uname)" == "Darwin" ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> tar -zxvf $FILENAME >> echo Downloaded and decompressed >> >> Thanks, >> >> Peter

9 23

Fwd: Managing Data Locality
by John Chilton 14 Nov '13

14 Nov '13

Forgot to cc galaxy-dev on this, it is too riveting not to post :). -John ---------- Forwarded message ---------- From: John Chilton <chil0060(a)umn.edu> Date: Thu, Nov 14, 2013 at 10:00 AM Subject: Re: [galaxy-dev] Managing Data Locality To: "Paniagua, Eric" <epaniagu(a)cshl.edu> Hey Eric, Sorry for the delayed response, I have pushed some updates to galaxy-central and the lwr to close some loop and fill out documentation based on your comments. I worry my last e-mail didn't make it clear that what you want to do is very ... ambitious. I didn't mean to make it sound like this was a solved problem, just that it was a problem that people were working on various parts of. The additional wrinkle that you would like run these jobs as the actual user is another significant hurdle. All of that said, you are certainly not alone or unreasonable in wanting this functionality - many large computing centers have very similar use cases and have made various degree of progress including my former employer the Minnesota Supercomputing Institute and I doubt anyone is currently using the LWR in this capacity at such centers (it is still mostly used for submitting jobs to Windows servers). On Fri, Nov 8, 2013 at 4:04 PM, Paniagua, Eric <epaniagu(a)cshl.edu> wrote: > Hi John, > > I have now read the top-level documentation for LWR, and gone through the sample configurations. I would appreciate if you would answer a few technical questions for me. > > 1) How exactly is the "staging_directory" in "server.ini.sample" used? Is that intended to be the (final) location at which to put files on the remote server? How is the relative path structure under $GALAXY_ROOT/databases/files handled? Depending on the configuration, either the LWR client or the LWR will copy/transfer files out of $GALAXY_ROOT/database/files into ${staging_directory}/${job_identifier}.In your case $GALAXY_ROOT/database/files will not be mounted on the large compute cluster but staging_directory should be. Here, job_identifier can be either the Galaxy job id or a UUID if you want to allow multiple Galaxy instances to submit to the same LWR (see assign_uuid option in server.ini.sample). > > 2) What exactly does "persistence_directory" in "server.ini.sample" mean? Where should it be located, how will it be used? The LWR doesn't talk to a "real" database, it just uses a directory to persist various internal mappings that should persist beyond an LWR restart. You shouldn't need to modify this unless the $LWR_ROOT/persisted_data is not writable to the LWR user (in your case the LWR user should likely be the Galaxy user). I have filled out the documentation in server.ini.sample to reflect this. > > 3) What exactly does "file_cache_dir" in "server.ini.sample" mean? It is an experimental feature - it may help you down the road to cache large files on the file system available to the whole cluster on that file system so they only need to be transferred out of $GALAXY_ROOT/database/files once. This option is not used unless it is specified however, so I would try to get it working in a simpler (though still very complicated :) ) configuration first. I have filled out the documentation in server.ini.sample to reflect this. > > 4) Does LWR preserve some relative path (e.g. to GALAXY_ROOT) under the above directories? No. > > 5) Are files renamed when cached? If so, are they eventually restored to their original names? They are put in new directory structures (e.g. ${staging_directory}/${job_id}/{inputs,outputs}), but I believe they should have the same names. They do eventually get plopped back into $GALAXY_ROOT/database/files. I have never used the LWR with an object store - so none of this might work. Hopefully, if there is a fix needed it will be easy. > > 6) Is it possible to customize the DRMAA and/or qsub requests made by LWR, for example to include additional settings such as Project or a memory limit? Is it possible to customize this on a case by case basis, rather than globally? Yes, there are lots of possibilities here (probably too many) and obviously none particularly well documented. There is an LWR way of doing this, but I the best thing to do is going to be piggy backing on job_conf.xml (the Galaxy way). You will want to review the documentation for how to setup job_conf.xml and ponit to it, but once you set up an LWR destination you can specify a native specification to pass along to the LWR by adding the following tag: <param id="submit_native_specification">-P bignodes -R y -pe threads 8</param> This will only work if your are targeting a queued_drmaa or queued_external_drmaa job manager on the LWR side... > > 7) Are there any options for the "queued_drmaa" manager in "job_managers.ini.sample" which are not listed in that file? Yes, in particular native_specification can be specified just likely Galaxy here. I have updated the sample. However, given your setup (run as real user) you will want the manager type 'queued_external_drmaa' which runs DRMAA communications in a separate process that can be run as a different user. This has some additional options "production", chown_working_directory_script, drmaa_kill_script, drmaa_launch_script. The defaults for all these should just work but you can modify them if you want. You will need to add the following rules or some variant of them to your sudoers file on the LWR server. galaxy ALL = (root) NOPASSWD: SETENV: /home/galaxy/lwr/scripts/drmaa_external_runner.sh galaxy ALL = (root) NOPASSWD: SETENV: /home/galaxy/lwr/scripts/drmaa_external_killer.py galaxy ALL = (root) NOPASSWD: SETENV: /usr/bin/chown This last rule can be significantly restricted since all chowns will be of the form "chown -R '{user}' '/path/to/staging_directory/job_id'". I need to work documentation of all this though. If using the external version of the drmaa runner, you will need to add another param element to the destination to pass along the user, specifically: <param id="submit_user">$__user_name__</param> > > 8) What exactly are the differences between the "queued_drmaa" manager and the "queued_cli" manager? Are there any options for the latter which are not in the "job_managers.ini.sample" file? queued_drmaa/queued_external_drmaa uses the DRMAA api to communicate with the DRM, queued_cli uses the qsub and qstat commands (this feels less clean, but is probably just as good). There is no queue_external_cli, but it would probably be easy to add. > > 9) When I attempt to run LWR (not having completed all the mentioned preparation steps, namely without setting DRMAA_LIBRARY_PATH), I get a Seg fault. Is this because it can't find DRMAA or is it potentially unrelated? In the latter case, here's the error being output to the console: > > ./run.sh: line 65: 26277 Segmentation fault paster serve server.ini "$@" Can you try to simplify the configuration, do not specify a job_managers file for instance, this would narrow it down to a DRMAA_LIBRARY_PATH problem. Also, sometimes you need to update the LD_LIBRARY_PATH to place the underlying (pbs?) shared library on it. > > Lastly, a simple comment, hopefully helpful. It would be nice if the LWR install docs at least mentioned the dependency of PyOpenSSL 0.13 (or later) on OpenSSL 0.9.8f (or later), maybe even with a comment that "pip" will listen to the environment variables CFLAGS and LDFLAGS in the event one is creating a local installation of the OpenSSL library for LWR to use. I have tried to update the documentation to reflect this, though in my defense it mentions pyopenssl a couple places :). If you have more specific recommendations, can you send me a patch for the README.rst file or issue a pull request and I will consider additional changes. > > Thank you for your time and assistance. > > Best, > Eric > ________________________________________ > From: jmchilton(a)gmail.com [jmchilton(a)gmail.com] on behalf of John Chilton [chilton(a)msi.umn.edu] > Sent: Tuesday, November 05, 2013 11:58 AM > To: Paniagua, Eric > Cc: Galaxy Dev [galaxy-dev(a)bx.psu.edu] > Subject: Re: [galaxy-dev] Managing Data Locality > > Hey Eric, > > I think what you are purposing would be a major development effort and > mirrors major development efforts ongoing. There are sortof ways to > do this already, with various trade-offs, and none particularly well > documented. So before undertaking this efforts I would dig into some > alternatives. > > If you are using PBS, the PBS runner contains some logic for > delegating to PBS for doing this kind of thing - I have never tried > it. > > https://bitbucket.org/galaxy/galaxy-central/src/default/lib/galaxy/jobs/run… > > In may be possible to use a specially configured handler and the > Galaxy object store to stage files to a particular mount before > running jobs - not sure it makes sense in this case. It might be worth > looking into this (having the object store stage your files, instead > of solving it at the job runner level). > > My recommendation however would be to investigate the LWR job runner. > There are a bunch of fairly recent developments to enable something > like what you are describing. For specificity lets say you are using > DRMAA to talk to some HPC cluster and Galaxy's file data is stored in > /galaxy/data on the galaxy web server but not on the HPC and there is > some scratch space (/scratch) that is mounted on both the Galaxy web > server and your HPC cluster. > > I would stand up an LWR (http://lwr.readthedocs.org/en/latest/) server > right beside Galaxy on your web server. The LWR has a concept of > managers that sort of mirrors the concept of runners in Galaxy - see > the sample config for guidance on how to get it to talk with your > cluster. It could use DRMAA, torque command-line tools, or condor at > this time (I could add new methods e.g. PBS library if that would > help). https://bitbucket.org/jmchilton/lwr/src/default/job_managers.ini.sample?at=… > > On the Galaxy side, I would then create a job_conf.xml file telling > certain HPC tools to be sent to the LWR. Be sure to enable the LWR > runner at the top (see advanced example config) and then add at least > one LWR destination. > > <destinations> > .... > <destination id="lwr" runner="lwr"> > <param id="url">http://localhost:8913/</param> >  > <param id="default_file_action">none</param> >  > <param id="file_action_config">file_actions.json</param> > </destination> > > Then create a file_actions.json file in the Galaxy root directory > (structure of this file is subject to change, current json layout > doesn't feel very Galaxy-ish). > > {"paths": [ > {"path": "/galaxy/data", "action": "copy"} > ] } > > More details on the structure of this file_actions.json file can be > found in the following changeset: > https://bitbucket.org/galaxy/galaxy-central/commits/b0b83be30136e2939a4a4f5… > > I am really eager to see the LWR gain adoption and tackle tricky cases > like this, so if there is anything I can do to help please let me know > and contributions in terms of development or documentation would be > greatly appreciated as well. > > Hope this helps, > -John > > On Tue, Nov 5, 2013 at 8:23 AM, Paniagua, Eric <epaniagu(a)cshl.edu> wrote: >> Dear Galaxy Developers, >> >> I administer a Galaxy instance at Cold Spring Harbor Laboratory, which servers around 200 laboratory members. While our initial hardware purchase has scaled well for the last 3 years, we are finding that we can't quite keep up with rising the demand for compute-intensive jobs, such as mapping. We are hesitant to consider buying more hardware to support the load, since we can't expect that solution to scale. >> >> Rather, we are attempting to set up Galaxy to queue jobs (especially mappers) out to the lab's HPCC to accommodate the increasing load. While there is a good number of technical challenges involved in this strategy, I am only writing to ask about one: data locality. >> >> Normally, all Galaxy datasets are stored directly on the private server hosting our Galaxy instance. The HPCC cannot mount our Galaxy server's storage (ie: for the purpose of running jobs reading/writing datasets) for security reasons. However, we can mount a small portion of the HPCC file system to our Galaxy server. Storage on the HPCC is at a premium, so we can't afford to just let newly created (or copied) datasets just sit there. It follows that we need a mechanism for maintaining temporary storage in the (restricted) HPCC space which allows for transfer of input datasets to the HPCC (so they will be visible to jobs running there) and transfer of output datasets back to persistent storage on our server. >> >> I am in the process of analyzing when/where/how exact path names are substituted into tool command lines, looking for potential hooks to facilitate the staging/unstaging of data before/after job execution on the HPCC. I have found a few places where I might try to insert logic for handling this case. >> >> Before modifying too much of Galaxy's core code, I would like to know if there is a recommended method for handling this situation and whether other members of the Galaxy community have implemented fixes or workarounds for this or similar data locality issues. If you can offer either type of information, I shall be most grateful. Of course, if the answer were that there were no recommended or known technique, then that would be valuable information too. >> >> Thank you in advance, >> Eric Paniagua >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/

1 0

Galaxy environment on local resources
by Alistair Chilcott 13 Nov '13

13 Nov '13

Hello all, Feel free to point it out if I have missed something obvious, I have done a fair bit of investigation and haven't quite found the solution yet. We have some hardware that has been around for a while for the purpose of processing Genetic data and other related tasks. To this end Galaxy fits the bill nicely in that it enables researchers to analyse data without being Linux geeks. The problem I have is that while the hand built galaxy server (running on SUSE for historical reasons) we have works to a point it is difficult to maintain and installing new tools and reference genomes is fiddly at best given that our server doesn't conform to the way the instructions for other systems expect it to work. We have had success using Cloudman on AWS to run training on how to use galaxy, and I would like to know more about how to customise and instance to contain all of the tools (mostly the NGS tools) we need by default. Ultimately we wont be able to use AWS to process much of the "real" data we have, because of a need to keep the data we are processing in-house due to ethics agreements. Fortunately we do have access to a modest pool of hardware (which is about to get bigger) to implement some kind of private cloud solution. How would I go about setting up a "private cloud" version of the cloudman style "galaxy instance on demand" system where researchers can start an instance, have it connect to a shared storage volume and process some data then terminate the instance? And is this even the best way to go? I have found it should be possible to use the scripts to install and configure the galaxy instances but I have not found any information on how to setup the environment that is required to make this work as a private cloud. Conversely I have found information about some private cloud scenarios such as Eucalyptis, and OpenStack but have not been able to join the dots to determine how to and if I can make the cloudman/galaxy usecase work on it. I should mention that I'm primarily a Windows Sys Admin (who dabbles in Linux) who is looking at this due to a lack of a dedicated Linux admin. At the end of the day I need to be able to setup this system and make it as low maintenance as possible whilst being useful and accessible to the researchers who aren't Linux admins. Any advice gratefully accepted. Regards, Alistair

2 1

api script import_library_dataset_to_history.py problem
by Olivia Doppelt 13 Nov '13

13 Nov '13

Hello to all, I'm trying to use the script import_library_dataset_to_history.py in order to upload a file from a dataset in a history. I get an error, but I'm not sure that I use it the right way. The command line I use is: python import_library_dataset_to_history.py cd065330048989af4ae266cf94685e6b http://127.0.0.1:8080 /api/libraries/f597429621d6eb2b/contents/f597429621d6eb2b I'm not sur if the url (http://127.0.0.1:8080) is the one to use. I'm not sure if the library_file_id is the right one. At the url http://127.0.0.1:8080/api/libraries/f597429621d6eb2b/contents/f597429621d6e… The Json: { "data_type": "fasta", "date_uploaded": "2012-11-16T13:29:37.051910", "file_name": "/home/odoppelt/localStuff_VM/tools/galaxy-dist/database/files/000/dataset_2.dat", "file_size": 5492113, "folder_id": "f597429621d6eb2b", "genome_build": "?", "id": "f597429621d6eb2b", "ldda_id": "f597429621d6eb2b", "message": "", "metadata_data_lines": 90357, "metadata_dbkey": "?", "metadata_sequences": 246, "misc_blurb": "246 sequences", "misc_info": "uploaded fasta file", "model_class": "LibraryDataset", "name": "Galaxy64-[Fasta_file_containing_result_contigs].fasta", "template_data": {}, "uploaded_by": "odoppelt(a)pasteur.fr", "uuid": null } could you please tell me if I use it the wrong way or if it doesn't work. FYI, the error i'm getting is: Traceback (most recent call last): File "import_library_dataset_to_history.py", line 16, in <module> submit( sys.argv[1], sys.argv[2], data ) File "common.py", line 105, in submit r = post( api_key, url, data ) File "common.py", line 47, in post return simplejson.loads( urllib2.urlopen( req ).read() ) File "/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/__init__.py", line 384, in loads File "/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py", line 402, in decode File "/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) Thanks in advance, Cheers, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur

1 0