Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-comma... --Dave B. On 08/26/2013 10:32 AM, Peter Cock wrote:

On Mon, Aug 26, 2013 at 3:10 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

...

Peter,

I apologize for the delay, here is the information you requested:

ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash

--Dave B.

Thanks Dave,

Sadly that looks just like my server where it works,

$ uname Linux $ arch x86_64 $ echo $0 -bash

I was considering trying to have the tool_dependencies.xml install recipe call a shell or Python script instead - but the new Tool Shed "Tool dependency definition" repository type prevents that (unless using a workaround like downloading the script first).

Hmm. Does anyone have any ideas on this error return code 64?

Regards,

Peter

Simon wrote:

...

...

An example is the NCBI BLAST+ suite, which failed to build on my (EL6) system, because it couldn't run /usr/bin/touch. That's pretty dumb, and pretty simple to solve in isolation - it needs to be running /bin/touch instead.

This was an oddity in the NCBI compile scripts for BLAST 2.2.26+ which can be avoided via a symlink, editing their code, or more simply just downloading their provided binaries.

I prefer the later because it is simpler, faster, and should better ensure reproducibility by eliminating things like different compilers and their settings. Unfortunately the new install process to do this from within the Tool Shed is glitching on the Tool Shed's nightly tests:

Nevertheless, the simple binary install is working for me and Bjoern. Could you try the Test Tool Shed package and let us know if that works for you?

Hi Peter, It seems to me that there's quite a bit still to do on the Tool Shed dependency packaging machinery. I don't think I'm inclined to be part of that testing process, sorry. (Rather short of time, having to choose carefully where I get most bang for buck.) As an alternative, I note your requirement for having multiple versions of a package installed side-by-side, which isn't traditionally offered by package repositories, although there is nothing to preclude that. In fact, I am currently in discussion with the CentOS project leader about collaborating on a scientific repo for Enterprise Linux (RHEL, CentOS, Scientific Linux, Oracle Linux). I will be feeding in my ideas for multiple version installs, and will see where that takes us. Who knows, you may have better choices soon on how to install packages for the Tool Shed. cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

On Wed, Sep 18, 2013 at 11:47 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Tue, Aug 27, 2013 at 2:18 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

...

Peter,

I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you.

I've added a trello card to track progress on this issue:

https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-comma...

--Dave B.

Hi Dave,

Something seems to have changed again on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/4ae1bac976f3

Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information.

Is the test cluster really missing /bin/sh? Can you check that and post the full installation log please?

I really would like to push an update to ncbi_blast_plus to the main Tool Shed with an updated README file (using RST) and the new citation information - plus of course handling the binary dependency via the new package:

http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26

Once this is done there is a backlog of updates I want to tackle following our productive BoF meeting at GCC2013: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast

Thanks,

Peter

A related bug for Greg (I think), despite this strange no /bin/sh error, this repository is not listed on the Test Tool Shed under "Latest revision: installation errors". Regards, Peter

Hi all, Some good news, the strange missing /bin/sh error has gone away on the Test Tool Shed: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc The BLAST+ wrapper tests also look good on the main Tool Shed: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573 Hooray! Peter P.S. Another plug for explicitly testing Tool Dependency Definition package installation on the Tool Shed: https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-pac...

Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. For sh, the corresponding syntax would look like: if [ "$string" = "value" ] then echo "string is equal to value" fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced <actions_group> feature, which would bypass shell-dependent conditional syntax entirely. --Dave B. On 10/07/2013 09:45 AM, Peter Cock wrote:

On Tue, Sep 24, 2013 at 10:58 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

Hi all,

Some good news, the strange missing /bin/sh error has gone away on the Test Tool Shed:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc

The BLAST+ wrapper tests also look good on the main Tool Shed:

http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573

Hooray!

Peter

P.S. Another plug for explicitly testing Tool Dependency Definition package installation on the Tool Shed: https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-pac...

Bad news, the strange missing /bin/sh error has come back again on both the main and Test Tool Shed :(

These are the same revisions of the BLAST+ wrapper, so the only changes are in Galaxy itself, the Tool Shed itself, or the Tool Shed test environment:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573

Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information.

Regards,

Peter

On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

...

Peter,

I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh.

Does that mean the test system has switched between sh and bash and back to sh again?

Is there anything written down about the expected shell(s) available for a standard Galaxy instance?

...

For sh, the corresponding syntax would look like:

if [ "$string" = "value" ] then echo "string is equal to value" fi

However, it strikes me that the platform detection and download url determination could also be done using the recently introduced <actions_group> feature, which would bypass shell-dependent conditional syntax entirely.

Yes, but is that in stable releases for galaxy-dist yet?

In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a... Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: <actions os="darwin" architecture="x86_64"> <actions os="darwin" architecture="i386"> into: <actions os="darwin" architecture="x86_64,i386"> or just: <actions os="darwin"> and similarly, combine these: <actions os="linux" architecture="i386"> <actions os="linux" architecture="i686"> into something like: <actions os="linux" architecture="i386,i686"> Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter

actions_group is neither in stable nor in default branch of galaxy-dist repository, it's present only in galaxy-central. Nicola Il giorno lun, 07/10/2013 alle 11.24 -0400, Dave Bouvier ha scritto:

Peter,

The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions.

--Dave B.

On 10/07/2013 11:22 AM, Peter Cock wrote:

...

On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

...

Peter,

I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh.

Does that mean the test system has switched between sh and bash and back to sh again?

Is there anything written down about the expected shell(s) available for a standard Galaxy instance?

...

...

For sh, the corresponding syntax would look like:

if [ "$string" = "value" ] then echo "string is equal to value" fi

However, it strikes me that the platform detection and download url determination could also be done using the recently introduced <actions_group> feature, which would bypass shell-dependent conditional syntax entirely.

Yes, but is that in stable releases for galaxy-dist yet?

In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before:

http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a...

Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action.

For example, combine these:

<actions os="darwin" architecture="x86_64"> <actions os="darwin" architecture="i386">

into:

<actions os="darwin" architecture="x86_64,i386">

or just:

<actions os="darwin">

and similarly, combine these:

<actions os="linux" architecture="i386"> <actions os="linux" architecture="i686">

into something like:

<actions os="linux" architecture="i386,i686">

Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well.

Thanks,

Peter

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

On Mon, Oct 7, 2013 at 10:22 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

...

Peter,

I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh.

Does that mean the test system has switched between sh and bash and back to sh again?

Is there anything written down about the expected shell(s) available for a standard Galaxy instance?

I don't think it is documented anywhere and there has been some disagreement on this in the past. I think we should do our best to reach some sort of conclusion on this and document it in the following card: https://trello.com/c/7VTlX9rD My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? I know there is some difference in bash behavior between platforms, but that is hopefully minimal and we can address inconsistencies one at a time. Thanks, -John

...

...

For sh, the corresponding syntax would look like:

if [ "$string" = "value" ] then echo "string is equal to value" fi

However, it strikes me that the platform detection and download url determination could also be done using the recently introduced <actions_group> feature, which would bypass shell-dependent conditional syntax entirely.

Yes, but is that in stable releases for galaxy-dist yet?

In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before:

http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a...

Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action.

For example, combine these:

<actions os="darwin" architecture="x86_64"> <actions os="darwin" architecture="i386">

into:

<actions os="darwin" architecture="x86_64,i386">

or just:

<actions os="darwin">

and similarly, combine these:

<actions os="linux" architecture="i386"> <actions os="linux" architecture="i686">

into something like:

<actions os="linux" architecture="i386,i686">

Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well.

Thanks,

Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

This is not an objection, but do we need bash? Can we live with posix sh? We should ask this question about every requirement we introduce. (bash is not part of the default installation of FreeBSD or OpenBSD for example. bash is unfortunately licensed under GPLV3, so if you are trying to create an OS not polluted by viral licensing you don't get bash). On Mon, Oct 7, 2013 at 11:36 AM, John Chilton <chilton@msi.umn.edu> wrote:

My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available?

-- James Taylor, Associate Professor, Biology/CS, Emory University

On Thu, Nov 14, 2013 at 5:13 PM, Nate Coraor <nate@bx.psu.edu> wrote:

Bash is easily obtained on these systems and I think the extra functionality available in post-Bourne shells ought to be allowed. I also question how many people are going to run tools on *BSD since most underlying analysis tools tend to only target Linux.

So could bash be declared an expected Galaxy dependency? i.e. The core system libraries and tools which Tool Authors may assume, and which Galaxy Administrators should install?

That said, shell code should be restricted to Bourne-compatible syntax whenever there’s no good reason to use non-Bourne features, e.g. if all you’re doing is `export FOO=foo`, you should not be forcing the use of bash. In cases where bash really is required (say, you’re using arrays), the script should explicitly specify '#!/bin/bash' (or '#!/usr/bin/env bash'?) rather than '#!/bin/sh'.

I agree that any shell script (e.g. a tool wrapper) which is bash specific should say that in the hash-bang line, rather than '#!/bin/sh'. What about command line magic like -num_threads \${GALAXY_SLOTS:-8} in a <command> tag using bash specific environment variable default values? What about bash specific if statements in <action type="shell_command"> sections of a tool_dependencies.xml file (which is what the BLAST+ packages currently use on the main Tool Shed, pending an update to use arch/os specific tags as tested on the Test Tool Shed)? Peter

If, by "current installation methods" you are not referring to Tool Shed repository installation methods for installing tool dependencies, then please read no further. For installing tool dependencies along with repositories from the Tool Shed, fabric is currently being used. However, it's been on my list for a while to eliminate fabric if possible so that I have more management of the threads and process ids during installation. If fabric / subprocess is a problem here, I can attempt to raise the prority of looking at this. Greg Von Kuster On Nov 14, 2013, at 12:38 PM, Nate Coraor <nate@bx.psu.edu> wrote:

This is a bit tricker, since there’s currently no way to specify installation actions to run on a system other than the Galaxy application server. However, if we are saying that bash should be available to tools, I’d think there is no reason to say that it’s not expected to be available to tool installation. Unfortunately, I believe the current installation methods use subprocesses’ default, which is sh, which is not going to be bash on some systems (on Debian, it’s dash).

—nate