Hi Ira,
Hi Bjoern,
I've just done some more testing of my tool with a fresh install and a fresh checkout of galaxy. After install of this tool
http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_protk_1_2_5
I have an env.sh that looks like
GEM_PATH=/home/vagrant/tool_dependencies/protk/1.2.5/iracooke/package_protk_1_2_5/1100ac98dfd2:$GEM_PATH; export GEM_PATH PATH=/home/vagrant/tool_dependencies/protk/1.2.5/iracooke/package_protk_1_2_5/1100ac98dfd2/bin:$PATH; export PATH
Which seems to be enough for me to run the executables in my gem. So I must be wrong about GEM_HOME needing to be set ... sorry for the false alarm there.
The thing that I'm now having trouble with is that all the other set_environment tags in my tool definition are ignored.
In my tool_dependencies.xml I have
<action type="set_environment"> <environment_variable action="prepend_to" name="PATH">$INSTALL_DIR/bin</environment_variable> <environment_variable action="set_to" name="GEM_HOME">$INSTALL_DIR/</environment_variable> <environment_variable action="set_to" name="PROTK_BLAST_ROOT">FROMPATH</environment_variable> </action>Admittedly some of that stuff is redundant ( I now realise ) ... but other stuff (eg PROTK_BLAST_ROOT ) is important ... but it gets ignored.
Is this intended? Seems like I should be able to use the set_environment action as well as the setup_ruby_environment action in the same tool definition?
It's implemented in:
https://bitbucket.org/galaxy/galaxy-central/pull-request/248/add-more-env-va...
If my last PR will be merged, you should be able to do that. Thanks for spotting that use-case.
Cheers, Bjoern
Hope you can clarify.
Thanks
Ira
On 31/10/2013, at 6:06 AM, Ira Cooke mudflatsoftware@gmail.com wrote:
Hi Bjoern,
I saw the same SO post ... but it says that GEM_PATH can be multiple paths but one of them should be GEM_HOME. Under your system (I think) GEM_HOME remains unset?
Anyway ... I'll test again to double check that this is required for my tool to work.
Cheers Ira
On 31 Oct 2013, at 6:00 am, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote:
Hi Ira,
Hi Bjoern,
I'll test,
thanks!
but I wonder if the problem is that GEM_PATH is set instead of GEM_HOME. I think GEM_HOME needs to be set also. It's zoo of environment variables out there :)!!
Mh according to
http://stackoverflow.com/questions/11277227/whats-the-difference-between-gem...
GEM_PATH will contain GEM_HOME and GEM_HOME is used for installation, and I use it for installation of GEMs, See the code. If you think GEM_HOME is necessary to export I will do so. Otherwise I would not export to much.
Thanks! Bjoern
Also ... I think GEM_HOME should be set using "set_to" not "prepend_to" ...
I am using the latest galaxy-central (default branch).
Cheers Ira
On 31 Oct 2013, at 12:33 am, Bjoern Gruening bjoern.gruening@gmail.com wrote:
Hi Ira,
that is strange, because PATH and GEM_HOME should be set automatically for you.
Please have a look at https://github.com/bgruening/galaxytools/blob/master/orphan_tool_dependencie... http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_ruby2_bioruby_1_4
That definition results on my test environment in:
GEM_PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1:$GEM_PATH; export GEM_PATH PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1/bin:$PATH; export PATH
And the code for that is here: https://bitbucket.org/galaxy/galaxy-central/src/77d58fdd1c2e0dfde37427620927...
Are you using the latest galaxy central?
Regarding RUBYLIB, I think we need to set it to specify the right ruby version, because many can be installed in parallel. Some holds for RUBYHOME or?
I see your point in having it set automatically and came with the attached patch. Here I need to assume that package_ruby_x_x will have RUBYLIB and RUBYHOME specified, but I think that is a save assumption. Nevertheless, we should document it.
With that new patch you will end up with
RUBYLIB=/home/gruening/projects/code/galaxy-central/tool_deps/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/; export RUBYLIB RUBYHOME=/home/gruening/projects/code/galaxy-central/tool_deps/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63; export RUBYHOME GEM_PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1:$GEM_PATH; export GEM_PATH PATH=/home/gruening/projects/code/galaxy-central/tool_deps/bioruby/1.4/bgruening/package_ruby2_bioruby_1_4/0a30368657b1/bin:$PATH; export PATH PATH=/home/gruening/projects/code/galaxy-central/tool_deps/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH
Thanks very much for your feedback. I do not have any ruby repository so I really appropriate your testing! Bjoern
Hi Bjoern,
Ross just gave me some pointers and I realised that I could just set these environment variables myself by doing
<action type="set_environment"> <environment_variable action="prepend_to" name="PATH">$INSTALL_DIR/bin</environment_variable> <environment_variable action="set_to" name="GEM_HOME">$INSTALL_DIR/</environment_variable> </action>
In my protk tool definition.
RUBYLIB isn't set ... hopefully this will not be an issue since I guess that's mostly just needed for Gem installation/compilation.
This part works now :)
Thanks Ira
On 30/10/2013, at 3:51 PM, Ira Cooke iracooke@gmail.com wrote:
> Hi Bjoern, > > Changing the title to reflect the proper topic > > I see now that the gem executables are installed under $INSTALLDIR/bin so I added this to the environment for my protk install. > > That got me to the point where my executable would run ... but now I'm seeing that in my env.sh file I don't really have my ruby environment setup properly (ie RUBYLIB and GEM_HOME are not set). This resulted in my tool running but it couldn't find my gem. > > So I tried adding ruby explicitly as a tool dependency for my downstream tool (this seems wrong though ... since it is required by protk ... and if I include protk I should get it automatically right?). After doing this though I was getting an error on installing protk about "List index out of range". > > Just to summarize, I can't see how to setup my dependencies properly so that the executables installed by protk get run with the correct environment. > > The repository I'm trying to install is > > http://testtoolshed.g2.bx.psu.edu/view/iracooke/omssa > > I'm sure I've just got muddled somewhere ... but it feels like I'm just messing around and getting it wrong. I'd rather do it in the way you envisioned. > > If you could give me some pointers on what I've done wrong it would help alot > > Thanks > Ira > > > > > On 29/10/2013, at 8:14 PM, Bjoern Gruening bjoern.gruening@gmail.com wrote: > >> >>> Ahhh ... Thanks .. I can't believe I missed that. Thanks for spotting it. >> >> No problem! >> >>> I'm not sure I understand enough about how this works to answer your ruby_home question. Ill play with this more tomorrow and see how things work with my tools and maybe have more feedback then >> >> Sure, thanks for working on that! >> >> Here is the promised trello card: https://trello.com/c/l9ZlGa8R >> Ciao, >> Bjoern >> >>> Cheers >>> Ira >>> >>> >>>> On 29 Oct 2013, at 7:17 pm, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote: >>>> >>>> Hi Ira, >>>> >>>> as far as I can see you missed the <actions> tag. That results in no log >>>> message at all, since no action type is executed. Galaxy should probably >>>> give an error here. Or better validate that during tool upload. I will >>>> fill a Trello Card for it. >>>> >>>> The following code snipped worked for me: >>>> >>>> <?xml version="1.0"?> >>>> <tool_dependency> >>>> <package name="protk" version="1.2.5"> >>>> <install version="1.0"> >>>> <actions> >>>> <action type="setup_ruby_environment"> >>>> <repository changeset_revision="a0494c6e1c63" >>>> name="package_ruby_2_0" owner="bgruening" >>>> toolshed="http://testtoolshed.g2.bx.psu.edu%22%3E >>>> <package name="ruby" version="2.0" /> >>>> </repository> >>>> <package>protk=1.2.6.pre1</package> >>>> </action> >>>> </actions> >>>> </install> >>>> <readme> >>>> Installs version 1.2.5 of the protk rubygem from rubygems.org >>>> </readme> >>>> </package> >>>> </tool_dependency> >>>> >>>> Keep in mind that the repository tag should be filled. If you have a >>>> actions and action tag it will be filled automatically by the TS during >>>> upload. But better to check :) >>>> >>>> To populate the RUBY_HOME env var ... do you think its better to set it >>>> via the installation automatically? My idea was to just "require" ruby >>>> in the tool? If you want to set it in your protk definition you need to >>>> use the $ENV[] construct. >>>> >>>> Cheers, >>>> Bjoern >>>> >>>>> Hi Bjoern, >>>>> >>>>> Thanks ... I thought I had to do that in order to make the GALAXY_RUBY_HOME variable available ... nice that it's not necessary. >>>>> >>>>> I've updated the tool on the test toolshed and have rerun my test but unfortunately the problem is still there. >>>>> >>>>> Cheers >>>>> Ira >>>>> >>>>> >>>>>> On 29/10/2013, at 10:06 AM, Björn Grüning bjoern.gruening@pharmazie.uni-freiburg.de wrote: >>>>>> >>>>>> Hi Ira, >>>>>> >>>>>> I can reproduce that error here, will try to study it tomorrow more >>>>>> deeply. For the meantime can you try to remove the following lines? >>>>>> >>>>>> <action type="set_environment_for_install"> >>>>>> <repository name="package_ruby_2_0" owner="bgruening"> >>>>>> <package name="ruby" version="2.0" /> >>>>>> </repository> >>>>>> </action> >>>>>> >>>>>> It's not necessary. >>>>>> Will get back to you tomorrow. >>>>>> Bjoern >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> The short version of my problem is that I often seem to find myself in a situation where my tool gets stuck in the "Installing" phase ... and I don't know how to get feedback on it to see where it is failing. What I guess I'm after is just the stdout of the tool installation .. but since the tool install never completes I can never see this. What I usually do is try to recreate the environment that galaxy sees during tool install in a normal shell .. and then run it manually .. which sometimes tells me the problem ... but not always (see below). >>>>>>> >>>>>>> The longer version of my problem is below. >>>>>>> >>>>>>> In my case I am struggling with the following tool >>>>>>> >>>>>>> http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_protk_1_2_5 >>>>>>> >>>>>>> Which uses the new "setup_ruby_environment" tag ... so the tool should do the following >>>>>>> >>>>>>> (1) Install the dependency .. Ruby 2.0 ... which it does successfully >>>>>>> (2) Install my protk rubygem from rubygems.org .. which it gets stuck doing. >>>>>>> >>>>>>> The last few lines from my galaxy server log during the failed install are; >>>>>>> >>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:26 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>> [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>> [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:30 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>> [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>> [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh >>>>>>> [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdQaLI7O >>>>>>> ruby version 2.0 installed in /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63 >>>>>>> [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>> [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>> [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>> [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh >>>>>>> [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdJYyhO0 >>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:34 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>> 10.0.2.2 - - [28/Oct/2013:21:47:22 +0000] "POST /admin_toolshed/manage_repositories HTTP/1.1" 302 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>> 10.0.2.2 - - [28/Oct/2013:21:54:37 +0000] "GET /admin_toolshed/monitor_repository_installation?tool_shed_repository_ids=f597429621d6eb2b&tool_shed_repository_ids=f2db41e1fa331b3e HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" >>>>>>> >>>>>>> >>>>>>> With a normal tool install I would see the last line of this output repeat lots of times as installation is monitored ... but for some reason in this case monitoring just shuts down and I don't see any further updates. >>>>>>> >>>>>>> If I recreate the situation on the command-line (eg by manually setting the RUBYLIB and GEM_HOME environment variables appropriately I can install my gem without any errors. >>>>>>> >>>>>>> More details on my setup: >>>>>>> >>>>>>> I'm using a vagrant VM for testing which is based on Ubuntu precise64 ... with a minimal number of packages installed ... >>>>>>> (My VM is documented in full here: https://bitbucket.org/iracooke/protk_vagrant/src/0ef8d86cc3dcec4ee3a9a83edf2... ) >>>>>>> >>>>>>> I'm testing based on the current default branch of galaxy-central .... which I regularly tear down and re-clone from scratch ... always with the same result. >>>>>>> >>>>>>> Any help or pointers to how to resolve this much appreciated. >>>>>>> >>>>>>> Cheers >>>>>>> Ira >>>> >>>> >>>> >> >> >> >
<add_more_env_vars_to_ruby_install_environment.patch>