October 2013 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
by Dave Bouvier 14 Nov '13

14 Nov '13

Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. On 08/26/2013 09:44 AM, Peter Cock wrote: > (Off list) > > Hi Dave, Greg, > > Just a reminder - without a bit more information about the Tool Shed > cluster I am stuck about how to debug this install issue. > > Thanks, > > Peter > > > On Wed, Aug 21, 2013 at 2:05 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >>> Peter, >>> >>> Here is the relevant log output for the installation of ncbi_blast_plus. A >>> quick google informs me that exit code 64 means "command line usage error", >>> for what it's worth. >>> >>> ... >>> tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row >>> (or updating an existing row) for repository 'package_blast_plus_2_2_26' in >>> the tool_shed_repository table, status set to 'New'. >>> >>> Warning: local() encountered an error (return code 64) while executing ' >>> if [[ "$(uname)" == "Linux" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi && >>> if [[ "$(arch)" == "x86_64" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi && >>> echo Fetching $FILENAME && >>> if [[ "$(uname)" == "Linux" ]]; then wget >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then curl -O >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> tar -zxvf $FILENAME && >>> echo Downloaded and decompressed >>> ' >>> >> >> Thanks Dave, >> >> I've made a note on this Trello card about exposing the log information >> of unsuccessful builds on the Tool Shed, >> https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-pa… >> >> Can you tell me what version of Linux this is, and which shell? >> >> $ uname >> ... >> >> $ arch >> ... >> >> $ echo $0 >> ... >> >> That might give me a clue about the nature of the failure. My >> hunch is that I am using something bash specific. >> >> The simplest solution would be if you could run each of these >> command by hand to see if it triggers the same error: >> >> if [[ "$(uname)" == "Linux" ]]; then export FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi >> if [[ "$(arch)" == "x86_64" ]]; then export FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi >> if [[ "$(uname)" == "Darwin" ]]; then export FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi >> echo Fetching $FILENAME >> if [[ "$(uname)" == "Linux" ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> if [[ "$(uname)" == "Darwin" ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> tar -zxvf $FILENAME >> echo Downloaded and decompressed >> >> Thanks, >> >> Peter

9 23

Contributing to genome indexes on rsync server
by Brad Chapman 08 Nov '13

08 Nov '13

Hi all; Is there a way for community members to contribute indexes to the rsync server? This resource is awesome and I'm working on migrating the CloudBioLinux retrieval scripts to use this instead of the custom S3 buckets we'd set up previously: https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/biodata/gala… It's great to have this as a public shared resource and I'd like to be able to contribute back. From an initial pass, here are the things I'd like to do: - Include bowtie2 indexes for more genomes. - Include novoalign indexes for a number of commonly used genomes. - Clean up hg19 to include a full canonically sorted hg19, with indexes. Broad has a nice version prepped so GATK will be happy with it, and you need to stick with this ordering if you're ever going to use a GATK tool on it. Right now there is a partial hg19canon (without the random/haplotype chromosomes) and the structure is a bit complex. What's the best way to contribute these? Right now I have a lot of the indexes on S3. For instance, the hg19 indexes are here: https://s3.amazonaws.com/biodata/genomes/hg19-bowtie.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bowtie2.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bwa.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-novoalign.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-seq.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-ucsc.tar.xz I'm happy to format these differently or upload somewhere that would make it easy to include. Thanks again for setting this up, I'm looking forward to working off a shared repository of data, Brad

6 8

Bowtie2 mm9 index
by Davis, Mary 08 Nov '13

08 Nov '13

Greetings- I tried to run an alignment using Bowtie2 and got this message- format: bam, database: mm9 Could not locate a Bowtie index corresponding to basename "/galaxy/data/mm9/mm9canon/bowtie2_index/mm9canon" Error: Encountered internal Bowtie 2 exception (#1) Command: /galaxy/software/linux2.6-x86_64/pkg/bowtie2-2.1.0/bin/bowtie2-align --wrapper basic I imported Illumina fastq data, groomed them, and then did the analysis using the built-in index mouse, both full and male, and had the same error message. I'm relatively new to this, and don't see what I missed. Thanks Mary E. Davis, Ph.D. Professor Department of Physiology & Pharmacology West Virginia University Health Sciences Center PO Box 9229 Morgantown, WV 26506-9229

2 2

Re: [galaxy-dev] datacache & bowtie2 for mm9 ?
by Curtis Hendrickson (Campus) 07 Nov '13

07 Nov '13

Jennifer, What's the status of bowtie2/mm9 index on PSU main? When I select tophat2, it offers me mm9 as a choice for built-in indexes. However, when the job runs, I get the following error, indicating the bowtie2/mm9 indexes are missing (below). Any insight into whether this is expected, or what the ETA is until the index would be installed, would be great. I'm trying to reproduce work on PSU I ran on my local galaxy, so that we can link to it for supplemental materials for a paper. Thanks, Curtis PS - I clicked the submit bug button a few days ago, but haven't received a response yet. Fatal error: Tool execution failed [2013-10-29 10:13:27] Beginning TopHat run (v2.0.9) ----------------------------------------------- [2013-10-29 10:13:27] Checking for Bowtie Bowtie version: 2.1.0.0 [2013-10-29 10:13:27] Checking for Samtools Samtools version: 0.1.18.0 [2013-10-29 10:13:27] Checking for Bowtie index files (genome).. Error: Could not find Bowtie 2 index files (/galaxy/data/mm9/mm9full/bowtie2_index/mm9full.*.bt2) From: Jennifer Jackson [mailto:jen@bx.psu.edu] Sent: Friday, September 20, 2013 4:00 PM To: Curtis Hendrickson (Campus) Subject: Re: [galaxy-dev] datacache & bowtie2 for mm9 ? Thanks Curtis, I am actually working to try to get mm9 out there right now. No promises, but is just one (well, three, including variants)! If technical is a go, then will do it. Ideally others soonish. We'll see. The last news brief has help for the Data manager, it may be that you need to do some config changes to get it going. I am certainly no expert - this is Dan's and under active development - but is where I would start. Jen On 9/20/13 1:25 PM, Curtis Hendrickson (Campus) wrote: Thanks for the rapid reply! I have some questions and comments, but need to read up on Data Managers (that admin page seems non-functional in our local galaxy, despite being on latest code) first. Regards, Curtis From: Jennifer Jackson [mailto:jen@bx.psu.edu] Sent: Friday, September 20, 2013 2:34 PM To: Curtis Hendrickson (Campus) Cc: galaxy-dev(a)bx.psu.edu<mailto:galaxy-dev@bx.psu.edu> Subject: Re: [galaxy-dev] datacache & bowtie2 for mm9 ? Hello Curtis, The datacache was originally pointed to the data staging area and is now pointed to the data published area. The difference is that the published area contains data and location (.loc) files that are in synch and have completed final testing. It is your choice about whether to use the staged-only data - it depends how risk tolerant your project is and if you plan on testing. But, that said, I think it is almost certainly fine or our team wouldn't have staged it yet. A vanishingly small number of datasets are pulled back once they make it to staging, and this is why we were comfortable pointing datacache there in the first place (were unable to point to the published area at first, but wanted to make the data available ASAP). Going forward - I can let you know that these indexes are very easy to create: one command-line execution, then add one line to the associated .loc file. Instructions are here, see "Bowtie and Tophat": http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup For one or few genomes, not a problem. For hundreds of genomes with variants, can become tedious even with helper tools and in our case, the processing interacted with disk that was undergoing changes (as we have been working on system configuration most of the summer). Also, with the Data Manager is now available, creating batch indexes for use via rsync become lower priority. Even so, I would expect more indexes to be fully published once the final configuration is in place, as many are already staged or close being staged (watch the yellow banner on Main). Hopefully this helps to explain the data, guides you to making an informed decision, and aids with creating your own indexes as needed, Thanks! Jen Galaxy team On 9/18/13 1:04 PM, Curtis Hendrickson (Campus) wrote: Folks, First, I wanted to thank you for making the datacache available (http://wiki.galaxyproject.org/Admin/Data%20Integration; rsync://datacache.g2.bx.psu.edu) It's a great resource. However, what is the best way to stay abreast of changes to what's in datacache, and understand how these indexes are computed? We are currently upgrading to bowtie2, but I notice that the bowtie2 indices for mm9, which used to be in rsync://datacache.g2.bx.psu.edu/indexes/mm9/mm9*/bowtie2_index have been removed, and only the hg19 genome has bowtie2 indices. Why only that one, and not the others? Where are the scripts you use to make these indices, in case I want to create bowtie2 indices for other So, how do I find out *why* they were removed? (Can I safely use the copy I have, or was there a problem with them?) More generally, how do I understand the policies and logic behind the datacache indices, and be notified of changes, short of running my own periodic rsync/diff? Finally, since I'm doing "reproducible research" is anything planned for systematically versioning genome indices, so I can easily tell what version of a system (ie, what BWA version) was used to create the index, and be sure that an index will not suddenly disappear. Thanks, Curtis Research Associate/CTSA-Informatics Team University of Alabama at Birmingham ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1

Comments on and a fix for John Chilton's tool-dependency-resolver-plugins
by Guest, Simon 07 Nov '13

07 Nov '13

Hi John, This is in regard to this: https://bitbucket.org/galaxy/galaxy-central/pull-request/228/tool-dependenc… Overall, this is very useful, just what I need, thanks. I'd *really* like to see this feature in the mainline Galaxy. Is there some voting necessary on Trello to achieve this, or is it enough to be enthusiastic here? I tested the ModuleDependencyResolver, and fixed three problems: 1. Fixed up module loading to work properly. The problem is that 'module' is not a first class command, it's a shell function. And it only works from interactive shells. The solution is to use the underlying modulecmd command. This requires deeper knowledge in the modules resolver of how environment modules work, which obviates the DEFAULT_MODULE_COMMAND and the flexibility to override it. 2. Made versionless fallback work, i.e. use the matching version if it exists, and only fallback to a generic match if it doesn't. 3. Enhanced the DirectoryModuleChecker to look along the modulepath, not just in a single directory. The default path is initialised appropriately from environment variables MODULEPATH, MODULESHOME, as per module(1). This can be overridden with the attribute modulepath rather than directory in the config file. Fix attached - I presume a Mercurial export is all you need? It may be better to default prefetch to false (but I didn't change that). Otherwise the Galaxy server needs restarting after new system packages become available. Now, there's one more thing required, which I'm not sure how to achieve. I intend to run with this config: <dependency_resolvers> <modules prefetch="false" versionless="true"/> </dependency_resolvers> So in particular I'm not interested in tool_shed_packages. However, when I install from the toolshed, say, the emboss tool, it still downloads the source tarball and tries to compile it locally (which fails, as I don't have make installed on my production Galaxy, nor do I want it). The emboss tool status in the "Manage installed tool shed repositories" list is "Installed, missing tool dependencies", but actually my installed modules mean the tool dependencies are satisfied. The behaviour I'm after is not even to try to do the actions in a tool_dependency.xml package spec in the toolshed, if I have dependency resolvers configured without tool_shed_packages. What are your thoughts on that? cheers, Simon

4 7

Galaxy dropping jobs?
by Nikolai Vazov 05 Nov '13

05 Nov '13

Hi, I discovered a weird issue in the job behaviour : Galaxy is running a long job on a cluster (more than 24h), about 15 hours later it misses the connection with SLURM on the cluster and throws the following message : [root@galaxy-prod01 galaxy-dist]# grep 3715200 paster.log galaxy.jobs.runners.drmaa INFO 2013-10-30 10:51:54,149 (555) queued as 3715200 galaxy.jobs.runners.drmaa DEBUG 2013-10-30 10:51:55,149 (555/3715200) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2013-10-30 10:52:13,516 (555/3715200) state change: job is running galaxy.jobs.runners.drmaa INFO 2013-10-31 03:29:33,090 (555/3715200) job left DRM queue with following message: code 1: slurm_load_jobs error: Unable to contact slurm controller (connect failure),job_id: 3715200 Is there a timeout in Galaxy for contacting slurm? Yet, the job is still running properly on the cluster ... Thanks for help, it's really urgent :) Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo

2 2

Re: [galaxy-dev] [galaxy-user] Inquiring
by Nate Coraor 04 Nov '13

04 Nov '13

Hi Yan, I've moved this discussion to the galaxy-dev list since it pertains to a local installation of Galaxy. Responses to your questions follow, in-line. Yan Luo wrote: > Dear Sir, > > (1)We installed Galaxy, but recently the user can't registered and got the > following error, how can we fix it? > > Sever error > An error occurred. See the error logs for more information.(To turn debug on > to display ...). Since debug = False in universe_wsgi.ini, you should be able to find a more detailed error message in the log file. If starting Galaxy with: % sh run.sh --daemon The default log file is 'paster.log' in Galaxy's root directory. > (2) Could you please let me know if there is any command to stop galaxy? If starting with the --daemon flag (as above), you can use: % sh run.sh --stop-daemon If running in the foreground, you can use Ctrl-C to terminate the process. There is a recent bug whereby Ctrl-C is ineffective on some platforms under Python 2.6 - in this case you will have to kill/pkill the process manually. We are working on a fix for the latter. > (3) If I reset universe_wsgi.ini file and want to set an administrator > user(I can add a line in the above file), how can I get the password? Should > I stop galaxy(See question 2) first? then run "./setup.sh" and "./run.sh". setup.sh would have only been necessary prior to running Galaxy the first time, however, this step has recently been removed. If you are referencing documentation that still refers to setup.sh, please let us know so we can update it - I did notice this was still on the "Production Server" page, so I removed it from there. You no longer need to run setup.sh at all. > (4) If I run "setup.sh", will a new file "universe_wsgi.ini" be generated? > if I want to change this file,should I edit it before "run.sh" and after > "setup.sh". Is it right? setup.sh and its replacements in run.sh and the Galaxy application itself never overwrite files, they only create files from sample files if they do not exist. > (5) I read some of your docs, command "sh setup.sh"(sh run.sh) and > "./setup.sh"(./run.sh), which one is correct under Linux? Both syntaxes are effectively the same in most cases. --nate > > Looking forward to hearing from you. > > Best Wises, > > Yan Luo, Ph.D. > NIH > <http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=By%20the%20way,> > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

4 7

Installing toolshed repositories with set_environment dependency
by S.D. Hiltemann 02 Nov '13

02 Nov '13

I updated my Galaxies to the latest version today, and when I try to install a toolshed repository which defines an environment variable via the <set_environment> tags in the tool_dependencies.xml file, I get an error (see below). tool_dependencies.xml: <?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="CONDEL_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> </tool_dependency> (before the update these repositories installed without problems) ..doing the same as an action within a package dependency does still work, so for now I have just changed the tools I needed to set the environment this way, e.g.: <tool_dependency> <package name="condel" version="1"> <install version="1.0"> <actions> <action type="set_environment"> <environment_variable name="CONDEL_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </action> </actions> </install> <readme> Set condel environment variable </readme> </package> </tool_dependency> Is this the preferred way of doing it, and should I change all my tools this way? or should both ways still be working? Saskia Error Traceback: View as: Interactive<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> | Text<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> | XML<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> (full)<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> ⇝ NameError: global name 'env_var_elem' is not defined URL: http://galaxy.trait-ctmm.cloudlet.sara.nl/admin_toolshed/manage_tool_depend… Module weberror.evalexception.middleware:364 in respond [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> app_iter = self.application(environ, detect_start_response) Module paste.recursive:84 in __call__ [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> body = method( trans, **kwargs ) Module galaxy.web.framework:229 in decorator [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:757 in manage_tool_dependencies [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> self.initiate_tool_dependency_installation( trans, tool_dependencies_for_installation ) Module galaxy.web.framework:229 in decorator [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:433 in initiate_tool_dependency_installation [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> tool_dependencies=tool_dependencies ) Module tool_shed.util.common_install_util:486 in handle_tool_dependencies [http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/media/plus.jpg] <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> view<http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> >> <http://galaxy.trait-ctmm.cloudlet.sara.nl/_debug/view/1382536593#> env_var_name = env_var_elem.get( 'name', None ) NameError: global name 'env_var_elem' is not defined ________________________________ From: S.D. Hiltemann Sent: Friday, August 02, 2013 11:19 AM To: galaxy-dev(a)lists.bx.psu.edu Subject: Toolshed repository update error I am working on putting a tool into the toolshed. I have a bash script wrapper. I uploaded the first version fine, but when I wanted to upload a revision, some unwanted characters are inserted around the revised function of my code (see bottom) ..the inserted "<<<<<< local" etc strings are causing my script to fail of course. How can I prevent this from happening? Saskia snippet of the code: if [[ ! -s $rfile ]] then dummycol=${addcols:2} outputcol=${dummycol//",B."/" "} <<<<<<< local echo -e "${col_chr_name}\t${col_start_name}\t${col_end_name}\t${col_ref_name}\t${col_obs_name}\t$outputcol" > $rfile cat $rfile ======= numcommas=`echo "$addcols" | grep -o "," | wc -l` echo "numcolums: $numcommas" awk 'BEGIN{FS="\t";OFS="\t"}{ if(FNR==1) print $0,"'"$outputcol"'"; else{ printf $0 for(i=0;i<="'"$numcommas"'"+1;i++) printf "\t" printf "\n" } }END{}' $ofile > tempofile mv tempofile $ofile return >>>>>>> other fi

6 17

Re: [galaxy-dev] Working with setup_ruby_environment tag
by Ira Cooke 01 Nov '13

01 Nov '13

Hi Bjoern, I've just done some more testing of my tool with a fresh install and a fresh checkout of galaxy. After install of this tool http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_protk_1_2_5 I have an env.sh that looks like GEM_PATH=/home/vagrant/tool_dependencies/protk/1.2.5/iracooke/package_protk_1_2_5/1100ac98dfd2:$GEM_PATH; export GEM_PATH PATH=/home/vagrant/tool_dependencies/protk/1.2.5/iracooke/package_protk_1_2_5/1100ac98dfd2/bin:$PATH; export PATH Which seems to be enough for me to run the executables in my gem. So I must be wrong about GEM_HOME needing to be set ... sorry for the false alarm there. The thing that I'm now having trouble with is that all the other set_environment tags in my tool definition are ignored. In my tool_dependencies.xml I have <action type="set_environment"> <environment_variable action="prepend_to" name="PATH">$INSTALL_DIR/bin</environment_variable> <environment_variable action="set_to" name="GEM_HOME">$INSTALL_DIR/</environment_variable> <environment_variable action="set_to" name="PROTK_BLAST_ROOT">FROMPATH</environment_variable> </action> Admittedly some of that stuff is redundant ( I now realise ) ... but other stuff (eg PROTK_BLAST_ROOT ) is important ... but it gets ignored. Is this intended? Seems like I should be able to use the set_environment action as well as the setup_ruby_environment action in the same tool definition? Hope you can clarify. Thanks Ira On 31/10/2013, at 6:06 AM, Ira Cooke <mudflatsoftware(a)gmail.com> wrote: > Hi Bjoern, > > I saw the same SO post ... but it says that GEM_PATH can be multiple paths but one of them should be GEM_HOME. Under your system (I think) GEM_HOME remains unset? > > Anyway ... I'll test again to double check that this is required for my tool to work. > > Cheers > Ira > > > On 31 Oct 2013, at 6:00 am, Björn Grüning <bjoern.gruening(a)pharmazie.uni-freiburg.de> wrote: > >> Hi Ira, >> >>> Hi Bjoern, >>> >>> I'll test, >> >> thanks! >> >>> but I wonder if the problem is that GEM_PATH is set instead of GEM_HOME. I think GEM_HOME needs to be set also. It's zoo of environment variables out there :)!! >> >> Mh according to >> >> http://stackoverflow.com/questions/11277227/whats-the-difference-between-ge… >> >> GEM_PATH will contain GEM_HOME and GEM_HOME is used for installation, >> and I use it for installation of GEMs, See the code. If you think >> GEM_HOME is necessary to export I will do so. Otherwise I would not >> export to much. >> >> Thanks! >> Bjoern >> >>> Also ... I think GEM_HOME should be set using "set_to" not "prepend_to" ... >>> >>> I am using the latest galaxy-central (default branch). >>> >>> Cheers >>> Ira >>> >>> >>> >>> >>> On 31 Oct 2013, at 12:33 am, Bjoern Gruening <bjoern.gruening(a)gmail.com> wrote: >>> >>>> Hi Ira, >>>> >>>> that is strange, because PATH and GEM_HOME should be set automatically >>>> for you. >>>> >>>> Please have a look at >>>> https://github.com/bgruening/galaxytools/blob/master/orphan_tool_dependenci… >>>> http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_ruby2_bioruby_1_4 >>>> >>>> That definition results on my test environment in: >>>> >>>> GEM_PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1:$GEM_PATH; export GEM_PATH >>>> PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1/bin:$PATH; export PATH >>>> >>>> And the code for that is here: >>>> https://bitbucket.org/galaxy/galaxy-central/src/77d58fdd1c2e0dfde3742762092… >>>> >>>> Are you using the latest galaxy central? >>>> >>>> Regarding RUBYLIB, I think we need to set it to specify the right ruby >>>> version, because many can be installed in parallel. Some holds for >>>> RUBYHOME or? >>>> >>>> I see your point in having it set automatically and came with the >>>> attached patch. Here I need to assume that package_ruby_x_x will have >>>> RUBYLIB and RUBYHOME specified, but I think that is a save assumption. >>>> Nevertheless, we should document it. >>>> >>>> With that new patch you will end up with >>>> >>>> RUBYLIB=/home/gruening/projects/code/galaxy-central/tool_deps/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/; export RUBYLIB >>>> RUBYHOME=/home/gruening/projects/code/galaxy-central/tool_deps/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63; export RUBYHOME >>>> GEM_PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1:$GEM_PATH; export GEM_PATH >>>> PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1/bin:$PATH; export PATH >>>> PATH=/home/gruening/projects/code/galaxy-central/tool_deps/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH >>>> >>>> >>>> Thanks very much for your feedback. I do not have any ruby repository so >>>> I really appropriate your testing! >>>> Bjoern >>>> >>>> >>>> >>>>> Hi Bjoern, >>>>> >>>>> Ross just gave me some pointers and I realised that I could just set these environment variables myself by doing >>>>> >>>>> <action type="set_environment"> >>>>> <environment_variable action="prepend_to" name="PATH">$INSTALL_DIR/bin</environment_variable> >>>>> <environment_variable action="set_to" name="GEM_HOME">$INSTALL_DIR/</environment_variable> >>>>> </action> >>>>> >>>>> In my protk tool definition. >>>>> >>>>> RUBYLIB isn't set ... hopefully this will not be an issue since I guess that's mostly just needed for Gem installation/compilation. >>>>> >>>>> This part works now :) >>>>> >>>>> Thanks >>>>> Ira >>>>> >>>>> >>>>> On 30/10/2013, at 3:51 PM, Ira Cooke <iracooke(a)gmail.com> wrote: >>>>> >>>>>> Hi Bjoern, >>>>>> >>>>>> Changing the title to reflect the proper topic >>>>>> >>>>>> I see now that the gem executables are installed under $INSTALLDIR/bin so I added this to the environment for my protk install. >>>>>> >>>>>> That got me to the point where my executable would run ... but now I'm seeing that in my env.sh file I don't really have my ruby environment setup properly (ie RUBYLIB and GEM_HOME are not set). This resulted in my tool running but it couldn't find my gem. >>>>>> >>>>>> So I tried adding ruby explicitly as a tool dependency for my downstream tool (this seems wrong though ... since it is required by protk ... and if I include protk I should get it automatically right?). After doing this though I was getting an error on installing protk about "List index out of range". >>>>>> >>>>>> Just to summarize, I can't see how to setup my dependencies properly so that the executables installed by protk get run with the correct environment. >>>>>> >>>>>> The repository I'm trying to install is >>>>>> >>>>>> http://testtoolshed.g2.bx.psu.edu/view/iracooke/omssa >>>>>> >>>>>> I'm sure I've just got muddled somewhere ... but it feels like I'm just messing around and getting it wrong. I'd rather do it in the way you envisioned. >>>>>> >>>>>> If you could give me some pointers on what I've done wrong it would help alot >>>>>> >>>>>> Thanks >>>>>> Ira >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 29/10/2013, at 8:14 PM, Bjoern Gruening <bjoern.gruening(a)gmail.com> wrote: >>>>>> >>>>>>> >>>>>>>> Ahhh ... Thanks .. I can't believe I missed that. Thanks for spotting it. >>>>>>> >>>>>>> No problem! >>>>>>> >>>>>>>> I'm not sure I understand enough about how this works to answer your ruby_home question. Ill play with this more tomorrow and see how things work with my tools and maybe have more feedback then >>>>>>> >>>>>>> Sure, thanks for working on that! >>>>>>> >>>>>>> Here is the promised trello card: https://trello.com/c/l9ZlGa8R >>>>>>> Ciao, >>>>>>> Bjoern >>>>>>> >>>>>>>> Cheers >>>>>>>> Ira >>>>>>>> >>>>>>>> >>>>>>>>> On 29 Oct 2013, at 7:17 pm, Björn Grüning <bjoern.gruening(a)pharmazie.uni-freiburg.de> wrote: >>>>>>>>> >>>>>>>>> Hi Ira, >>>>>>>>> >>>>>>>>> as far as I can see you missed the <actions> tag. That results in no log >>>>>>>>> message at all, since no action type is executed. Galaxy should probably >>>>>>>>> give an error here. Or better validate that during tool upload. I will >>>>>>>>> fill a Trello Card for it. >>>>>>>>> >>>>>>>>> The following code snipped worked for me: >>>>>>>>> >>>>>>>>> <?xml version="1.0"?> >>>>>>>>> <tool_dependency> >>>>>>>>> <package name="protk" version="1.2.5"> >>>>>>>>> <install version="1.0"> >>>>>>>>> <actions> >>>>>>>>> <action type="setup_ruby_environment"> >>>>>>>>> <repository changeset_revision="a0494c6e1c63" >>>>>>>>> name="package_ruby_2_0" owner="bgruening" >>>>>>>>> toolshed="http://testtoolshed.g2.bx.psu.edu"> >>>>>>>>> <package name="ruby" version="2.0" /> >>>>>>>>> </repository> >>>>>>>>> <package>protk=1.2.6.pre1</package> >>>>>>>>> </action> >>>>>>>>> </actions> >>>>>>>>> </install> >>>>>>>>> <readme> >>>>>>>>> Installs version 1.2.5 of the protk rubygem from rubygems.org >>>>>>>>> </readme> >>>>>>>>> </package> >>>>>>>>> </tool_dependency> >>>>>>>>> >>>>>>>>> Keep in mind that the repository tag should be filled. If you have a >>>>>>>>> actions and action tag it will be filled automatically by the TS during >>>>>>>>> upload. But better to check :) >>>>>>>>> >>>>>>>>> To populate the RUBY_HOME env var ... do you think its better to set it >>>>>>>>> via the installation automatically? My idea was to just "require" ruby >>>>>>>>> in the tool? If you want to set it in your protk definition you need to >>>>>>>>> use the $ENV[] construct. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Bjoern >>>>>>>>> >>>>>>>>>> Hi Bjoern, >>>>>>>>>> >>>>>>>>>> Thanks ... I thought I had to do that in order to make the GALAXY_RUBY_HOME variable available ... nice that it's not necessary. >>>>>>>>>> >>>>>>>>>> I've updated the tool on the test toolshed and have rerun my test but unfortunately the problem is still there. >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> Ira >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 29/10/2013, at 10:06 AM, Björn Grüning <bjoern.gruening(a)pharmazie.uni-freiburg.de> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Ira, >>>>>>>>>>> >>>>>>>>>>> I can reproduce that error here, will try to study it tomorrow more >>>>>>>>>>> deeply. For the meantime can you try to remove the following lines? >>>>>>>>>>> >>>>>>>>>>> <action type="set_environment_for_install"> >>>>>>>>>>> <repository name="package_ruby_2_0" owner="bgruening"> >>>>>>>>>>> <package name="ruby" version="2.0" /> >>>>>>>>>>> </repository> >>>>>>>>>>> </action> >>>>>>>>>>> >>>>>>>>>>> It's not necessary. >>>>>>>>>>> Will get back to you tomorrow. >>>>>>>>>>> Bjoern >>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> The short version of my problem is that I often seem to find myself in a situation where my tool gets stuck in the "Installing" phase ... and I don't know how to get feedback on it to see where it is failing. What I guess I'm after is just the stdout of the tool installation .. but since the tool install never completes I can never see this. What I usually do is try to recreate the environment that galaxy sees during tool install in a normal shell .. and then run it manually .. which sometimes tells me the problem ... but not always (see below). >>>>>>>>>>>> >>>>>>>>>>>> The longer version of my problem is below. >>>>>>>>>>>> >>>>>>>>>>>> In my case I am struggling with the following tool >>>>>>>>>>>> >>>>>>>>>>>> http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_protk_1_2_5 >>>>>>>>>>>> >>>>>>>>>>>> Which uses the new "setup_ruby_environment" tag ... so the tool should do the following >>>>>>>>>>>> >>>>>>>>>>>> (1) Install the dependency .. Ruby 2.0 ... which it does successfully >>>>>>>>>>>> (2) Install my protk rubygem from rubygems.org .. which it gets stuck doing. >>>>>>>>>>>> >>>>>>>>>>>> The last few lines from my galaxy server log during the failed install are; >>>>>>>>>>>> >>>>>>>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:26 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>>>>>>> [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>>>>>>> [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:30 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>>>>>>> [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>>>>>>> [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>>>>>>> [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdQaLI7O >>>>>>>>>>>> ruby version 2.0 installed in /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63 >>>>>>>>>>>> [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>>>>>>> [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>>>>>>> [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>>>>>>> [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>>>>>>> [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdJYyhO0 >>>>>>>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:34 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>>>>>>> 10.0.2.2 - - [28/Oct/2013:21:47:22 +0000] "POST /admin_toolshed/manage_repositories HTTP/1.1" 302 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:37 +0000] "GET /admin_toolshed/monitor_repository_installation?tool_shed_repository_ids=f597429621d6eb2b&tool_shed_repository_ids=f2db41e1fa331b3e HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> With a normal tool install I would see the last line of this output repeat lots of times as installation is monitored ... but for some reason in this case monitoring just shuts down and I don't see any further updates. >>>>>>>>>>>> >>>>>>>>>>>> If I recreate the situation on the command-line (eg by manually setting the RUBYLIB and GEM_HOME environment variables appropriately I can install my gem without any errors. >>>>>>>>>>>> >>>>>>>>>>>> More details on my setup: >>>>>>>>>>>> >>>>>>>>>>>> I'm using a vagrant VM for testing which is based on Ubuntu precise64 ... with a minimal number of packages installed ... >>>>>>>>>>>> (My VM is documented in full here: https://bitbucket.org/iracooke/protk_vagrant/src/0ef8d86cc3dcec4ee3a9a83edf… ) >>>>>>>>>>>> >>>>>>>>>>>> I'm testing based on the current default branch of galaxy-central .... which I regularly tear down and re-clone from scratch ... always with the same result. >>>>>>>>>>>> >>>>>>>>>>>> Any help or pointers to how to resolve this much appreciated. >>>>>>>>>>>> >>>>>>>>>>>> Cheers >>>>>>>>>>>> Ira >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> <add_more_env_vars_to_ruby_install_environment.patch> >>> >> >> >> >

2 1

Tool XML form building bug with <options from_data_table="..."> ?
by Jennifer Jackson 01 Nov '13

01 Nov '13

Moving this over to galaxy-dev(a)bx.psu.edu, the best list for tool design questions. Best! Jen, Galaxy team On 10/30/13 2:24 PM, damion(a)learningpoint.ca wrote: > Has anyone run into this? I'm building a general-purpose filter > control on my galaxy tool xml template for enabling numeric fields to > be filtered by > < etc. parameters - in a user friendly way. I have a > select list <para> driven by a data table: > ... > <param name="filter_column" type="select" label="Col"> > <options from_data_table="bccdc_blast_fields" /> > </param> > ... > This works fine in building a list of fields to select from. Then I add > <filter type="sort_by" column="1"/> > <filter type="add_value" name="TESTESTTEST" value="WHWHWHW" /> > > but nothing happens, sort remains incorrect and no extra value. I try > these same filters on a previous <param> in form that is driven > by <options from_file="bccdc_blast_bins.loc"> and they work fine. > > I also tried applying <fiilter type="static_value" ...> to no avail on > from_data_table options tag, but from from_file options, no problem. > > So I start to think there's a bug whereby NO filters work on <options > from_data_table="bccdc_blast_fields" /> input? I've surveyed the > galaxy.tools.parameters.dynamic_options > python code, but can't see a decision point there in which > from_data_table work vs from_file choice is made; is it in another > script file? > > I'm using a BioLinux 2012 install that includes Galaxy. > > Help appreciated! > > Regards, > > Damion Dooley > BC Centre for Disease Control > Vancouver, BC > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1