November 2013 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Converting .gff3 to 12-column .bed
by Jennifer Jackson 15 Nov '13

15 Nov '13

Hi Lindsay, Nice to hear that Vipin's server worked out for you. A reduction in the number of lines is completely expected. You should have the same number of lines in the BED12 file as you have unique "ID"s in the GFF3 file, if you want to double check. This is the same as the number of unique transcripts in any of the files, if summed up correctly. Why is this? A GFF3 file has one line per feature. Part of an exon, etc. with further (this can vary) annotation that tells you if it is 5' UTR, or CDS, etc. When converting GFF3 -> BED12, all the features for a particular transcript are rolled up into a single line of output. Transcripts will have a unique "ID" by GFF3 specification - and all features that belong to that transcript will have it in the GFF3 file's attributes field (it is usually the first attribute listed). When you converted just GFF3 -> BED6, each feature was individually converted into a distinct line of BED formatted output, there was no summing up to group the information for the transcript as a whole. The Galaxy wiki has more - GFF3 can represent other kinds of data, and that is covered in the link-outs to the original spec, but this is the gist of the meaning for your specific data. Glad I could help before, and that Vipin had the tool online & available to simplify things! Best, Jen Galaxy team On 11/12/13 4:12 PM, Lindsay Rutter wrote: > Hello Jennifer: > > Thank you for your very helpful and detailed explanation! > > I ended up using the site that Vipin provided in his message > (https://galaxy.cbio.mskcc.org/tool_runner?tool_id=fml_gff2bed) > > Indeed, it produced a file with 12 columns that appears to be in a > reasonable bed format (including the 10th column being an integer, > with the 11th and 12th column consisting of that integer number of > items separated by commas). > > However, I noticed that the number of lines in the file went > from 183,748 (in the .gff3 file) to 11,506 (in the 12-column .bed file). > > In your opinion, does this seem like a reasonable reduction in number > of lines? I was not expecting that (in fact, I was expecting the > number of lines would stay the same, as they did going from .gff3 to > 6-column .bed file), but I am quite inexperienced using these files. > > Thanking you... > Lindsay > > > > > On Mon, Nov 11, 2013 at 11:15 AM, Jennifer Jackson <jen(a)bx.psu.edu > <mailto:jen@bx.psu.edu>> wrote: > > Hello, > > There are no tools directly on the public Galaxy site to transform > a GFF3 dataset into a BED12 dataset. However, the Tool Shed has a > repository called ' fml_gff3togtf' that includes a tool for this > purpose, for use in a local install. The description is a bit > bothersome in that it a slightly incorrect datatype statement, so > be sure to test out the results. (the word "wiggle" has no place > in this statement: " > gff3_to_bed_converter.py: This tool converts gene transcript annotation from GFF3 format to UCSC wiggle 12 column BED format.") > http://getgalaxy.org > http://usegalaxy.org/toolshed > > I see your post at Biostar, and it might be helpful to let you > know what a BED12 file represents (plus I'll post this there, may > help others): > http://www.biostars.org/p/85869/ > > A BED12 file describes the complete, often spliced, alignment of a > sequence to a reference genome. This does not include minor base > variation, it is a macro alignment. You can think of each of the > blocks as being "exons", although there is no magic here - if the > sequence or genome had quality problems, or significant variation > (large insertion or deletion), that could cause the alignment to > fragment as well. > Here is the data description: > http://wiki.galaxyproject.org/Learn/Datatypes#Bed > > To see examples, at UCSC (genome.ucsc.edu > <http://genome.ucsc.edu>), EST or mRNA track will have this as the > primary table format. All gene track can also be in BED12 format, > or in a related one, genePred: > http://genome.ucsc.edu/FAQ/FAQformat.html#format9 > > UCSC also has line-command utilities to convert between the > formats, pre-compiled versions are here: > http://hgdownload.cse.ucsc.edu/downloads.html#source_downloads > > Either way, you can convert the data, then load up into the public > Galaxy (usegalaxy.org <http://usegalaxy.org>) and proceed with > your analysis. BEDTools works well with BED12 files. There is > definitely information loss attempting to transform BED6 -> BED12, > as the global alignment is lost. And adjusting attributes such as > score or name are often a preference, so you can alter these > however you want, as long as the attribute formatting rules for > the columns are followed. > > Hopefully this helps, > > Jen > Galaxy team > > > On 11/9/13 3:29 PM, lrutter @iastate.edu <http://iastate.edu> wrote: >> Hello Galaxy: >> >> I am trying overall to convert a .gff3 file to 12-column .bed file. >> >> I first tried GFF-to-BED converter, but it gave a 6-column .bed file. >> >> Then, I tried BED-to-bigBed converter by inputting the 6-column >> .bed file. I get an error "Unspecified genome build, click the >> pencil icon in the history item to set the genome build". >> >> So, I click the pencil icon, and see 4 tabs at the top. I set the >> "Attributes" tab as in the attached image (Attributes.png). >> >> But then, when I select "Convert Format", I am only seeing an >> option that outputs .bed12 file as "Convert Genomic Intervals to >> Strict BED12". I am a bit confused about this because I specified >> the input file as a .bed file (and not genomic intervals, unless >> I am misunderstanding something). >> >> In any case, when I select "Convert Genomic Intervals to Strict >> BED12", I do get a .bed file with 12 columns. But I would like to >> ask if I may have lost information going from the .gff3 to >> .bed(6) to .bed(12)? >> >> (I feel that scores were all set to "0" from .gff3 to .bed(6), >> and columns 10, 11, 12 (block counts, sizes, and starting >> positions) were all set to zero going from .bed(6) to .bed(12)). >> >> If I am correct that there is information loss, is there a system >> in Galaxy to prevent this, and transfer as much information as >> possible from .gff3 to .bed(12)? >> >> Thank you. >> L. Rutter >> >> ** Below is a head of my three files (the species is P. dominula): >> >> .gff3 file >> >> ##gff-version 3 >> ##date Mon Nov 4 14:54:42 2013 >> ##source gbrowse gbgff gff3 dumper >> PdomScaf0001 maker gene 15 1963 . - . >> Name=PdomGene00025;ID=1;Dbxref=MAKER:maker-PdomScaf0001-snap-gene-0.274 >> PdomScaf0001 maker mRNA 15 1963 . - . >> Name=PdomMRNA00025.1;Parent=1;ID=2;_QI=216%7C0%7C0.2%7C0.6%7C0.5%7C0.6%7C5%7C0%7C98;_eAED=0.43;_AED=0.43;Dbxref=MAKER:maker-PdomScaf0001-snap-gene-0.274-mRNA-1 >> PdomScaf0001 maker exon 15 100 -0.094 - . >> Parent=2;ID=3 >> PdomScaf0001 maker CDS 15 100 . - 2 >> Parent=2;ID=4 >> PdomScaf0001 maker exon 223 300 21.8 - . >> Parent=2;ID=5 >> PdomScaf0001 maker CDS 223 300 . - 2 >> Parent=2;ID=6 >> PdomScaf0001 maker exon 717 765 22.4 - . >> Parent=2;ID=7 >> >> .bed(6) file >> >> PdomScaf0001 14 1963 gene 0 - >> PdomScaf0001 14 1963 mRNA 0 - >> PdomScaf0001 14 100 exon 0 - >> PdomScaf0001 14 100 CDS 0 - >> PdomScaf0001 222 300 exon 0 - >> PdomScaf0001 222 300 CDS 0 - >> PdomScaf0001 716 765 exon 0 - >> PdomScaf0001 716 765 CDS 0 - >> PdomScaf0001 906 947 exon 0 - >> PdomScaf0001 906 947 CDS 0 - >> >> .bed(12) file >> >> PdomScaf0001 14 1963 gene 0 - 14 1963 >> 0 0 , , >> PdomScaf0001 14 1963 mRNA 0 - 14 1963 >> 0 0 , , >> PdomScaf0001 14 100 exon 0 - 14 100 >> 0 0 , , >> PdomScaf0001 14 100 CDS 0 - 14 100 >> 0 0 , , >> PdomScaf0001 222 300 exon 0 - 222 300 >> 0 0 , , >> PdomScaf0001 222 300 CDS 0 - 222 300 >> 0 0 , , >> PdomScaf0001 716 765 exon 0 - 716 765 >> 0 0 , , >> PdomScaf0001 716 765 CDS 0 - 716 765 >> 0 0 , , >> PdomScaf0001 906 947 exon 0 - 906 947 >> 0 0 , , >> PdomScaf0001 906 947 CDS 0 - 906 947 >> 0 0 , , >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > -- > Jennifer Hillman-Jackson > http://galaxyproject.org > > -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1

Errors running DRMAA and PBS on remote server running Torque 4
by Ganote, Carrie L 14 Nov '13

14 Nov '13

Hi List, I've sprouted some grays in the last week after my Galaxy instances all simultaneously ceased to submit jobs to our main cluster. Some Galaxy instances are running the PBS job runner, and others use DRMAA. For the DRMAA runner I was getting: galaxy.jobs.runners ERROR 2013-10-15 08:40:14,942 (1024) Unhandled exception calling queue_job Traceback (most recent call last): File "galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 60, in run_next method(arg) File "galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 188, in queue_job external_job_id = self.ds.runJob(jt) File "build/bdist.linux-x86_64/egg/drmaa/__init__.py", line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File "build/bdist.linux-x86_64/egg/drmaa/helpers.py", line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "build/bdist.linux-x86_64/egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) InternalException: code 1: (qsub) cannot access script file: Unauthorized Request MSG=can not authorize request (0-Success) And in my PBS runner: galaxy.jobs.runners.pbs WARNING 2013-10-14 17:13:07,319 (550) pbs_submit failed (try 1/5), PBS error 15044: Resources temporarily unavailable To give some background, I had recently requested a new virtual machine to put my test/dev Galaxy on. I copied our production Galaxy to this new VM. I secured a new domain name for it and set it running. Everything was going well until I tried to hook it up to the cluster; at first I got an error saying that I didn't have permission to submit jobs. Makes sense, the new VM was not a qualified submit host for the cluster. I asked the sysadmins to add the VM as a submit host to the cluster using qmgr. As soon as this was done, not only could I still not submit jobs from the test Galaxy, but no Galaxy was able to submit jobs to the cluster. The issue isn't with Galaxy here but the underlying calls that it makes - for drmaa, I tracked it back to pbs-drmaa/bin/drmaa-run. For PBS, I'm sure it's somewhere in with libtorque. In every case, I could call qsub from the command line and it would correctly submit jobs, which was more perplexing. I re-installed python, drmaa.egg, pbs-drmaa, and rebooted the VM. I of course restarted Galaxy with each step, to no avail. I worked with the admins to see what was happening in the server logs, but the same cryptic error showed up - cannot authorize request. I've had this issue before in the past, more or less, but usually just gave up on it. It seemed to come and go sporadically, but rebooting the clusters seemed to help. This time, with our production server no longer functioning, I begged for help and the admins looked through the pbs_server config but couldn't find any mistypes or problems. Reloading the config by sending hangup signals to pbs_server didn't help. Then we tried pausing the scheduler and restarting pbs_server completely - and eureka, all problems went away. PBS and DRMAA runners are back up and working fine. This really seems to be a bug in Torque 4.1.5.1. I hope this saves someone a lot of headache! Newer versions of Torque may be the answer. I would also advise against making changes to the pbs_server configuration while in production - we have monthly maintenance, and I don't think I'll ever request changes when there won't be an immediate reboot to flush the server! Cheers, Carrie

2 3

Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
by Dave Bouvier 14 Nov '13

14 Nov '13

Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. On 08/26/2013 09:44 AM, Peter Cock wrote: > (Off list) > > Hi Dave, Greg, > > Just a reminder - without a bit more information about the Tool Shed > cluster I am stuck about how to debug this install issue. > > Thanks, > > Peter > > > On Wed, Aug 21, 2013 at 2:05 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >>> Peter, >>> >>> Here is the relevant log output for the installation of ncbi_blast_plus. A >>> quick google informs me that exit code 64 means "command line usage error", >>> for what it's worth. >>> >>> ... >>> tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row >>> (or updating an existing row) for repository 'package_blast_plus_2_2_26' in >>> the tool_shed_repository table, status set to 'New'. >>> >>> Warning: local() encountered an error (return code 64) while executing ' >>> if [[ "$(uname)" == "Linux" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi && >>> if [[ "$(arch)" == "x86_64" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi && >>> echo Fetching $FILENAME && >>> if [[ "$(uname)" == "Linux" ]]; then wget >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then curl -O >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> tar -zxvf $FILENAME && >>> echo Downloaded and decompressed >>> ' >>> >> >> Thanks Dave, >> >> I've made a note on this Trello card about exposing the log information >> of unsuccessful builds on the Tool Shed, >> https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-pa… >> >> Can you tell me what version of Linux this is, and which shell? >> >> $ uname >> ... >> >> $ arch >> ... >> >> $ echo $0 >> ... >> >> That might give me a clue about the nature of the failure. My >> hunch is that I am using something bash specific. >> >> The simplest solution would be if you could run each of these >> command by hand to see if it triggers the same error: >> >> if [[ "$(uname)" == "Linux" ]]; then export FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi >> if [[ "$(arch)" == "x86_64" ]]; then export FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi >> if [[ "$(uname)" == "Darwin" ]]; then export FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi >> echo Fetching $FILENAME >> if [[ "$(uname)" == "Linux" ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> if [[ "$(uname)" == "Darwin" ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> tar -zxvf $FILENAME >> echo Downloaded and decompressed >> >> Thanks, >> >> Peter

9 23

Fwd: Managing Data Locality
by John Chilton 14 Nov '13

14 Nov '13

Forgot to cc galaxy-dev on this, it is too riveting not to post :). -John ---------- Forwarded message ---------- From: John Chilton <chil0060(a)umn.edu> Date: Thu, Nov 14, 2013 at 10:00 AM Subject: Re: [galaxy-dev] Managing Data Locality To: "Paniagua, Eric" <epaniagu(a)cshl.edu> Hey Eric, Sorry for the delayed response, I have pushed some updates to galaxy-central and the lwr to close some loop and fill out documentation based on your comments. I worry my last e-mail didn't make it clear that what you want to do is very ... ambitious. I didn't mean to make it sound like this was a solved problem, just that it was a problem that people were working on various parts of. The additional wrinkle that you would like run these jobs as the actual user is another significant hurdle. All of that said, you are certainly not alone or unreasonable in wanting this functionality - many large computing centers have very similar use cases and have made various degree of progress including my former employer the Minnesota Supercomputing Institute and I doubt anyone is currently using the LWR in this capacity at such centers (it is still mostly used for submitting jobs to Windows servers). On Fri, Nov 8, 2013 at 4:04 PM, Paniagua, Eric <epaniagu(a)cshl.edu> wrote: > Hi John, > > I have now read the top-level documentation for LWR, and gone through the sample configurations. I would appreciate if you would answer a few technical questions for me. > > 1) How exactly is the "staging_directory" in "server.ini.sample" used? Is that intended to be the (final) location at which to put files on the remote server? How is the relative path structure under $GALAXY_ROOT/databases/files handled? Depending on the configuration, either the LWR client or the LWR will copy/transfer files out of $GALAXY_ROOT/database/files into ${staging_directory}/${job_identifier}.In your case $GALAXY_ROOT/database/files will not be mounted on the large compute cluster but staging_directory should be. Here, job_identifier can be either the Galaxy job id or a UUID if you want to allow multiple Galaxy instances to submit to the same LWR (see assign_uuid option in server.ini.sample). > > 2) What exactly does "persistence_directory" in "server.ini.sample" mean? Where should it be located, how will it be used? The LWR doesn't talk to a "real" database, it just uses a directory to persist various internal mappings that should persist beyond an LWR restart. You shouldn't need to modify this unless the $LWR_ROOT/persisted_data is not writable to the LWR user (in your case the LWR user should likely be the Galaxy user). I have filled out the documentation in server.ini.sample to reflect this. > > 3) What exactly does "file_cache_dir" in "server.ini.sample" mean? It is an experimental feature - it may help you down the road to cache large files on the file system available to the whole cluster on that file system so they only need to be transferred out of $GALAXY_ROOT/database/files once. This option is not used unless it is specified however, so I would try to get it working in a simpler (though still very complicated :) ) configuration first. I have filled out the documentation in server.ini.sample to reflect this. > > 4) Does LWR preserve some relative path (e.g. to GALAXY_ROOT) under the above directories? No. > > 5) Are files renamed when cached? If so, are they eventually restored to their original names? They are put in new directory structures (e.g. ${staging_directory}/${job_id}/{inputs,outputs}), but I believe they should have the same names. They do eventually get plopped back into $GALAXY_ROOT/database/files. I have never used the LWR with an object store - so none of this might work. Hopefully, if there is a fix needed it will be easy. > > 6) Is it possible to customize the DRMAA and/or qsub requests made by LWR, for example to include additional settings such as Project or a memory limit? Is it possible to customize this on a case by case basis, rather than globally? Yes, there are lots of possibilities here (probably too many) and obviously none particularly well documented. There is an LWR way of doing this, but I the best thing to do is going to be piggy backing on job_conf.xml (the Galaxy way). You will want to review the documentation for how to setup job_conf.xml and ponit to it, but once you set up an LWR destination you can specify a native specification to pass along to the LWR by adding the following tag: <param id="submit_native_specification">-P bignodes -R y -pe threads 8</param> This will only work if your are targeting a queued_drmaa or queued_external_drmaa job manager on the LWR side... > > 7) Are there any options for the "queued_drmaa" manager in "job_managers.ini.sample" which are not listed in that file? Yes, in particular native_specification can be specified just likely Galaxy here. I have updated the sample. However, given your setup (run as real user) you will want the manager type 'queued_external_drmaa' which runs DRMAA communications in a separate process that can be run as a different user. This has some additional options "production", chown_working_directory_script, drmaa_kill_script, drmaa_launch_script. The defaults for all these should just work but you can modify them if you want. You will need to add the following rules or some variant of them to your sudoers file on the LWR server. galaxy ALL = (root) NOPASSWD: SETENV: /home/galaxy/lwr/scripts/drmaa_external_runner.sh galaxy ALL = (root) NOPASSWD: SETENV: /home/galaxy/lwr/scripts/drmaa_external_killer.py galaxy ALL = (root) NOPASSWD: SETENV: /usr/bin/chown This last rule can be significantly restricted since all chowns will be of the form "chown -R '{user}' '/path/to/staging_directory/job_id'". I need to work documentation of all this though. If using the external version of the drmaa runner, you will need to add another param element to the destination to pass along the user, specifically: <param id="submit_user">$__user_name__</param> > > 8) What exactly are the differences between the "queued_drmaa" manager and the "queued_cli" manager? Are there any options for the latter which are not in the "job_managers.ini.sample" file? queued_drmaa/queued_external_drmaa uses the DRMAA api to communicate with the DRM, queued_cli uses the qsub and qstat commands (this feels less clean, but is probably just as good). There is no queue_external_cli, but it would probably be easy to add. > > 9) When I attempt to run LWR (not having completed all the mentioned preparation steps, namely without setting DRMAA_LIBRARY_PATH), I get a Seg fault. Is this because it can't find DRMAA or is it potentially unrelated? In the latter case, here's the error being output to the console: > > ./run.sh: line 65: 26277 Segmentation fault paster serve server.ini "$@" Can you try to simplify the configuration, do not specify a job_managers file for instance, this would narrow it down to a DRMAA_LIBRARY_PATH problem. Also, sometimes you need to update the LD_LIBRARY_PATH to place the underlying (pbs?) shared library on it. > > Lastly, a simple comment, hopefully helpful. It would be nice if the LWR install docs at least mentioned the dependency of PyOpenSSL 0.13 (or later) on OpenSSL 0.9.8f (or later), maybe even with a comment that "pip" will listen to the environment variables CFLAGS and LDFLAGS in the event one is creating a local installation of the OpenSSL library for LWR to use. I have tried to update the documentation to reflect this, though in my defense it mentions pyopenssl a couple places :). If you have more specific recommendations, can you send me a patch for the README.rst file or issue a pull request and I will consider additional changes. > > Thank you for your time and assistance. > > Best, > Eric > ________________________________________ > From: jmchilton(a)gmail.com [jmchilton(a)gmail.com] on behalf of John Chilton [chilton(a)msi.umn.edu] > Sent: Tuesday, November 05, 2013 11:58 AM > To: Paniagua, Eric > Cc: Galaxy Dev [galaxy-dev(a)bx.psu.edu] > Subject: Re: [galaxy-dev] Managing Data Locality > > Hey Eric, > > I think what you are purposing would be a major development effort and > mirrors major development efforts ongoing. There are sortof ways to > do this already, with various trade-offs, and none particularly well > documented. So before undertaking this efforts I would dig into some > alternatives. > > If you are using PBS, the PBS runner contains some logic for > delegating to PBS for doing this kind of thing - I have never tried > it. > > https://bitbucket.org/galaxy/galaxy-central/src/default/lib/galaxy/jobs/run… > > In may be possible to use a specially configured handler and the > Galaxy object store to stage files to a particular mount before > running jobs - not sure it makes sense in this case. It might be worth > looking into this (having the object store stage your files, instead > of solving it at the job runner level). > > My recommendation however would be to investigate the LWR job runner. > There are a bunch of fairly recent developments to enable something > like what you are describing. For specificity lets say you are using > DRMAA to talk to some HPC cluster and Galaxy's file data is stored in > /galaxy/data on the galaxy web server but not on the HPC and there is > some scratch space (/scratch) that is mounted on both the Galaxy web > server and your HPC cluster. > > I would stand up an LWR (http://lwr.readthedocs.org/en/latest/) server > right beside Galaxy on your web server. The LWR has a concept of > managers that sort of mirrors the concept of runners in Galaxy - see > the sample config for guidance on how to get it to talk with your > cluster. It could use DRMAA, torque command-line tools, or condor at > this time (I could add new methods e.g. PBS library if that would > help). https://bitbucket.org/jmchilton/lwr/src/default/job_managers.ini.sample?at=… > > On the Galaxy side, I would then create a job_conf.xml file telling > certain HPC tools to be sent to the LWR. Be sure to enable the LWR > runner at the top (see advanced example config) and then add at least > one LWR destination. > > <destinations> > .... > <destination id="lwr" runner="lwr"> > <param id="url">http://localhost:8913/</param> >  > <param id="default_file_action">none</param> >  > <param id="file_action_config">file_actions.json</param> > </destination> > > Then create a file_actions.json file in the Galaxy root directory > (structure of this file is subject to change, current json layout > doesn't feel very Galaxy-ish). > > {"paths": [ > {"path": "/galaxy/data", "action": "copy"} > ] } > > More details on the structure of this file_actions.json file can be > found in the following changeset: > https://bitbucket.org/galaxy/galaxy-central/commits/b0b83be30136e2939a4a4f5… > > I am really eager to see the LWR gain adoption and tackle tricky cases > like this, so if there is anything I can do to help please let me know > and contributions in terms of development or documentation would be > greatly appreciated as well. > > Hope this helps, > -John > > On Tue, Nov 5, 2013 at 8:23 AM, Paniagua, Eric <epaniagu(a)cshl.edu> wrote: >> Dear Galaxy Developers, >> >> I administer a Galaxy instance at Cold Spring Harbor Laboratory, which servers around 200 laboratory members. While our initial hardware purchase has scaled well for the last 3 years, we are finding that we can't quite keep up with rising the demand for compute-intensive jobs, such as mapping. We are hesitant to consider buying more hardware to support the load, since we can't expect that solution to scale. >> >> Rather, we are attempting to set up Galaxy to queue jobs (especially mappers) out to the lab's HPCC to accommodate the increasing load. While there is a good number of technical challenges involved in this strategy, I am only writing to ask about one: data locality. >> >> Normally, all Galaxy datasets are stored directly on the private server hosting our Galaxy instance. The HPCC cannot mount our Galaxy server's storage (ie: for the purpose of running jobs reading/writing datasets) for security reasons. However, we can mount a small portion of the HPCC file system to our Galaxy server. Storage on the HPCC is at a premium, so we can't afford to just let newly created (or copied) datasets just sit there. It follows that we need a mechanism for maintaining temporary storage in the (restricted) HPCC space which allows for transfer of input datasets to the HPCC (so they will be visible to jobs running there) and transfer of output datasets back to persistent storage on our server. >> >> I am in the process of analyzing when/where/how exact path names are substituted into tool command lines, looking for potential hooks to facilitate the staging/unstaging of data before/after job execution on the HPCC. I have found a few places where I might try to insert logic for handling this case. >> >> Before modifying too much of Galaxy's core code, I would like to know if there is a recommended method for handling this situation and whether other members of the Galaxy community have implemented fixes or workarounds for this or similar data locality issues. If you can offer either type of information, I shall be most grateful. Of course, if the answer were that there were no recommended or known technique, then that would be valuable information too. >> >> Thank you in advance, >> Eric Paniagua >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/

1 0

Galaxy environment on local resources
by Alistair Chilcott 13 Nov '13

13 Nov '13

Hello all, Feel free to point it out if I have missed something obvious, I have done a fair bit of investigation and haven't quite found the solution yet. We have some hardware that has been around for a while for the purpose of processing Genetic data and other related tasks. To this end Galaxy fits the bill nicely in that it enables researchers to analyse data without being Linux geeks. The problem I have is that while the hand built galaxy server (running on SUSE for historical reasons) we have works to a point it is difficult to maintain and installing new tools and reference genomes is fiddly at best given that our server doesn't conform to the way the instructions for other systems expect it to work. We have had success using Cloudman on AWS to run training on how to use galaxy, and I would like to know more about how to customise and instance to contain all of the tools (mostly the NGS tools) we need by default. Ultimately we wont be able to use AWS to process much of the "real" data we have, because of a need to keep the data we are processing in-house due to ethics agreements. Fortunately we do have access to a modest pool of hardware (which is about to get bigger) to implement some kind of private cloud solution. How would I go about setting up a "private cloud" version of the cloudman style "galaxy instance on demand" system where researchers can start an instance, have it connect to a shared storage volume and process some data then terminate the instance? And is this even the best way to go? I have found it should be possible to use the scripts to install and configure the galaxy instances but I have not found any information on how to setup the environment that is required to make this work as a private cloud. Conversely I have found information about some private cloud scenarios such as Eucalyptis, and OpenStack but have not been able to join the dots to determine how to and if I can make the cloudman/galaxy usecase work on it. I should mention that I'm primarily a Windows Sys Admin (who dabbles in Linux) who is looking at this due to a lack of a dedicated Linux admin. At the end of the day I need to be able to setup this system and make it as low maintenance as possible whilst being useful and accessible to the researchers who aren't Linux admins. Any advice gratefully accepted. Regards, Alistair

2 1

api script import_library_dataset_to_history.py problem
by Olivia Doppelt 13 Nov '13

13 Nov '13

Hello to all, I'm trying to use the script import_library_dataset_to_history.py in order to upload a file from a dataset in a history. I get an error, but I'm not sure that I use it the right way. The command line I use is: python import_library_dataset_to_history.py cd065330048989af4ae266cf94685e6b http://127.0.0.1:8080 /api/libraries/f597429621d6eb2b/contents/f597429621d6eb2b I'm not sur if the url (http://127.0.0.1:8080) is the one to use. I'm not sure if the library_file_id is the right one. At the url http://127.0.0.1:8080/api/libraries/f597429621d6eb2b/contents/f597429621d6e… The Json: { "data_type": "fasta", "date_uploaded": "2012-11-16T13:29:37.051910", "file_name": "/home/odoppelt/localStuff_VM/tools/galaxy-dist/database/files/000/dataset_2.dat", "file_size": 5492113, "folder_id": "f597429621d6eb2b", "genome_build": "?", "id": "f597429621d6eb2b", "ldda_id": "f597429621d6eb2b", "message": "", "metadata_data_lines": 90357, "metadata_dbkey": "?", "metadata_sequences": 246, "misc_blurb": "246 sequences", "misc_info": "uploaded fasta file", "model_class": "LibraryDataset", "name": "Galaxy64-[Fasta_file_containing_result_contigs].fasta", "template_data": {}, "uploaded_by": "odoppelt(a)pasteur.fr", "uuid": null } could you please tell me if I use it the wrong way or if it doesn't work. FYI, the error i'm getting is: Traceback (most recent call last): File "import_library_dataset_to_history.py", line 16, in <module> submit( sys.argv[1], sys.argv[2], data ) File "common.py", line 105, in submit r = post( api_key, url, data ) File "common.py", line 47, in post return simplejson.loads( urllib2.urlopen( req ).read() ) File "/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/__init__.py", line 384, in loads File "/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py", line 402, in decode File "/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) Thanks in advance, Cheers, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur

1 0

Converting .gff3 to 12-column .bed
by lrutter ＠iastate.edu 13 Nov '13

13 Nov '13

Hello Galaxy: I am trying overall to convert a .gff3 file to 12-column .bed file. I first tried GFF-to-BED converter, but it gave a 6-column .bed file. Then, I tried BED-to-bigBed converter by inputting the 6-column .bed file. I get an error "Unspecified genome build, click the pencil icon in the history item to set the genome build". So, I click the pencil icon, and see 4 tabs at the top. I set the "Attributes" tab as in the attached image (Attributes.png). But then, when I select "Convert Format", I am only seeing an option that outputs .bed12 file as "Convert Genomic Intervals to Strict BED12". I am a bit confused about this because I specified the input file as a .bed file (and not genomic intervals, unless I am misunderstanding something). In any case, when I select "Convert Genomic Intervals to Strict BED12", I do get a .bed file with 12 columns. But I would like to ask if I may have lost information going from the .gff3 to .bed(6) to .bed(12)? (I feel that scores were all set to "0" from .gff3 to .bed(6), and columns 10, 11, 12 (block counts, sizes, and starting positions) were all set to zero going from .bed(6) to .bed(12)). If I am correct that there is information loss, is there a system in Galaxy to prevent this, and transfer as much information as possible from .gff3 to .bed(12)? Thank you. L. Rutter ** Below is a head of my three files (the species is P. dominula): .gff3 file ##gff-version 3 ##date Mon Nov 4 14:54:42 2013 ##source gbrowse gbgff gff3 dumper PdomScaf0001 maker gene 15 1963 . - . Name=PdomGene00025;ID=1;Dbxref=MAKER:maker-PdomScaf0001-snap-gene-0.274 PdomScaf0001 maker mRNA 15 1963 . - . Name=PdomMRNA00025.1;Parent=1;ID=2;_QI=216%7C0%7C0.2%7C0.6%7C0.5%7C0.6%7C5%7C0%7C98;_eAED=0.43;_AED=0.43;Dbxref=MAKER:maker-PdomScaf0001-snap-gene-0.274-mRNA-1 PdomScaf0001 maker exon 15 100 -0.094 - . Parent=2;ID=3 PdomScaf0001 maker CDS 15 100 . - 2 Parent=2;ID=4 PdomScaf0001 maker exon 223 300 21.8 - . Parent=2;ID=5 PdomScaf0001 maker CDS 223 300 . - 2 Parent=2;ID=6 PdomScaf0001 maker exon 717 765 22.4 - . Parent=2;ID=7 .bed(6) file PdomScaf0001 14 1963 gene 0 - PdomScaf0001 14 1963 mRNA 0 - PdomScaf0001 14 100 exon 0 - PdomScaf0001 14 100 CDS 0 - PdomScaf0001 222 300 exon 0 - PdomScaf0001 222 300 CDS 0 - PdomScaf0001 716 765 exon 0 - PdomScaf0001 716 765 CDS 0 - PdomScaf0001 906 947 exon 0 - PdomScaf0001 906 947 CDS 0 - .bed(12) file PdomScaf0001 14 1963 gene 0 - 14 1963 0 0 , , PdomScaf0001 14 1963 mRNA 0 - 14 1963 0 0 , , PdomScaf0001 14 100 exon 0 - 14 100 0 0 , , PdomScaf0001 14 100 CDS 0 - 14 100 0 0 , , PdomScaf0001 222 300 exon 0 - 222 300 0 0 , , PdomScaf0001 222 300 CDS 0 - 222 300 0 0 , , PdomScaf0001 716 765 exon 0 - 716 765 0 0 , , PdomScaf0001 716 765 CDS 0 - 716 765 0 0 , , PdomScaf0001 906 947 exon 0 - 906 947 0 0 , , PdomScaf0001 906 947 CDS 0 - 906 947 0 0 , ,

4 3

Re: [galaxy-dev] dynamic_options selected value as input for second dynamic_options function call
by Hans van Leeuwen 12 Nov '13

12 Nov '13

Hi, I found a working workaround using the deprecated <page> tags: <inputs> <page> <param name = "State_id" type="select" dynamic_options="getStates()" label="Select State" display="radio" multiple="false"/> </page> <page> <param name = "Town_id" type="select" dynamic_options="getTowns(State_id)" label="Select Town" display="radio" multiple="false"/> </page> </inputs> This works; both select lists (on separate GUI pages) are dynamically generated when the called function connects to a database to retrieve the options. However, as it is recommended not to use <page> tags any longer, can anyone suggest how to write above example using <condition> and <when> tags instead? I have searched a lot but can't find out how to do it. Thank you, Hans

1 0

Error editing users associated with a quota
by Peter Cock 12 Nov '13

12 Nov '13

Hi all, I'm trying to remove a user from our default quota (while they run a large analysis), but am getting an error: Internal Server Error Galaxy was unable to successfully complete your request An error occurred. This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. The error has been logged to our team. Has anyone else seen this? Thanks, Peter $ hg branch stable [galaxy@ppserver galaxy-dist]$ hg head | more changeset: 11219:5c789ab4144a branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Mon Nov 04 15:04:42 2013 -0500 summary: Added tag release_2013.11.04 for changeset 26f58e05aa10 changeset: 11216:c458a0fe1ba8 parent: 11213:6d633418ecfa parent: 11215:f79149dd3d35 user: Nate Coraor <nate(a)bx.psu.edu> date: Mon Nov 04 14:56:57 2013 -0500 summary: Merge security fix for filtering tools from stable/next-stable. >From the pasteur.log file, xxx.xxx.xxx.xxx - - [11/Nov/2013:12:04:45 +0100] "POST /galaxy/admin/manage_users_and_groups_for_quota?id=7a72206f0ccffc2e HTTP/1.1" 500 - "http://xxx/galaxy/admin/manage_users_and_groups_for_quota?webapp=galaxy&id=…" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0" Error - <type 'exceptions.AttributeError'>: 'NoQuotaAgent' object has no attribute 'set_entity_quota_associations' URL: http://ppserver/galaxy/admin/manage_users_and_groups_for_quota?id=7a72206f0… File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/mnt/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/mnt/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 229 in decorator return func( self, trans, *args, **kwargs ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin.py', line 531 in manage_users_and_groups_for_quota quota, params = self._quota_op( trans, 'quota_members_edit_button', self._manage_users_and_groups_for_quota, kwd ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin.py', line 664 in _quota_op message = op_method( quota, params ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/actions/admin.py', line 80 in _manage_users_and_groups_for_quota self.app.quota_agent.set_entity_quota_associations( quotas=[ quota ], users=in_users, groups=in_groups ) AttributeError: 'NoQuotaAgent' object has no attribute 'set_entity_quota_associations' CGI Variables ------------- CONTENT_LENGTH: '-1' CONTENT_TYPE: 'application/x-www-form-urlencoded' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-gb,en;q=0.7,en-us;q=0.3' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: 'galaxysession=fed9265fae01dfd85c30fe1bbf5a988ae27edcb17a4961e7cef7bef77b0eeb9be91a902d5f0aacdf' HTTP_DNT: '1' HTTP_HOST: 'ppserver' HTTP_REFERER: 'http://ppserver/galaxy/admin/manage_users_and_groups_for_quota?webapp=galax…' HTTP_USER_AGENT: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0' PATH_INFO: '/admin/manage_users_and_groups_for_quota' QUERY_STRING: 'id=7a72206f0ccffc2e' REMOTE_ADDR: '143.234.97.120' REQUEST_METHOD: 'POST' SCRIPT_NAME: '/galaxy' SERVER_NAME: 'xxx' SERVER_PORT: '8080' SERVER_PROTOCOL: 'HTTP/1.1' WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0x910c890> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='fed9265fae01dfd85c30fe1bbf5a988ae27edcb17a4961e7cef7bef77b0eeb9be91a902d5f0aacdf'>, 'galaxysession=fed9265fae01dfd85c30fe1bbf5a988ae27edcb17a4961e7cef7bef77b0eeb9be91a902d5f0aacdf') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x783fd50> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x30eea90> paste.parsed_querystring: ([('id', '7a72206f0ccffc2e')], 'id=7a72206f0ccffc2e') paste.recursive.forward: <paste.recursive.Forwarder from /galaxy> paste.recursive.include: <paste.recursive.Includer from /galaxy> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy> paste.recursive.script_name: '/galaxy' paste.throw_errors: True request_id: '744ee0cc4ac911e3bbcc0026b95a4d69' webob._parsed_post_vars: (MultiDict([('in_users', '22'), ('in_users', '6'), ('in_users', '12'), ('in_users', '14'), ('in_users', '15'), ('in_users', '4'), ('in_users', '5'), ('in_users', '20'), ('in_users', '9'), ('in_users', '16'), ('in_users', '11'), ('in_users', '7'), ('in_users', '17'), ('in_users', '23'), ('in_users', '19'), ('in_users', '10'), ('in_users', '32'), ('out_users', '21'), ('quota_members_edit_button', 'Save')]), <FakeCGIBody at 7f3708159910 viewing MultiDict([('in...e')])>) webob._parsed_query_vars: (MultiDict([('id', '7a72206f0ccffc2e')]), 'id=7a72206f0ccffc2e') wsgi process: 'Multithreaded' ------------------------------------------------------------

2 4

dynamic_options selected value as input for second dynamic_options function call
by Hans van Leeuwen 12 Nov '13

12 Nov '13

Hello, I would like to have a tool form with a two parameters: 1. Select State 2. Select Town The options for 'Select State' are dynamically generated with dynamic_options which calls a python function, which connects to a database and returns the list of States. This works fine in my form xml code: <param name = "State_id" type="select" dynamic_options="getStates()" > label="Select State" display="radio" multiple="false"> Now I want to dynamically create the options for '2. Select Town'. To make this work I need to take the selected option from '1. Select State', pass it as an argument to a dynamic_options function call that would then connect to a database and retrieve the Towns for the provided State. These Towns would then be the options for '2. Select Town'. So, the second parameter '2. Select Town' should do something like this: <param name = "Town_id" type="select" dynamic_options="getTowns([user > selected value in 1. Select State])" label="Select Town" display="radio" > multiple="false"> The part in square brackets is pseudo code; this is the part that I cannot get to work . That part should also be automatically refreshed if the user changes the selection in '1. Select State'. I looked into the use of <conditional> but I don't see how I can use that if I don't know beforehand what values I can have (determined in real time based on content of database), so I cannot code the <when> tags. Any help is greatly appreciated. Thank you in advance, Hans

1 0