Galaxy code run on main Tool Shed vs Test Tool Shed?
Splitting off from this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/016903.html On Mon, Oct 7, 2013 at 4:24 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions.
--Dave B.
Thank you for your reply Dave, but I hope there has been a miscommunication here. The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers). I had been under the impression that the main "Tool Shed" (and the nightly tool tests run from it) was tracking the stable branch on galaxy-dist and therefore equivalent to what most Galaxy instances are running. Are you saying that for both the main "Tool Shed" and the "Test Tool Shed" you are using the galaxy-dist default branch with the bleeding edge code to run the tests? If so, this worries me as that means the main "Tool Shed" nightly tests are not indicative of what would happen once the tools are downloaded and installed to a galaxy-dist stable branch instance. e.g. If I was to release a tool update using the latest <action> work then I would expect it to work fine on the "Test Tool Shed", but not to work on a Galaxy instance running the current stable release. Therefore, I would hope/expect it to fail on the main "Tool Shed" (due to using features not yet in the stable release). (The <action> enhancement is just an example, this is a general question.) Could you clarify the situation please? Thanks, Peter
On Tue, Oct 8, 2013 at 6:45 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers).
I have to say this was exactly my thoughts when developing a couple of tools recently. I would like to have a suitable place where I can test my tools before promoting them to the Main Tool Shed. Because of the difference in the code base between Main and Test Tool Sheds, the later is not such a place. I know I could always run a local Tool Shed, but I feel having a public place for testing where I can ask for other users to test my work would be more useful. Best, Carlos
On Tue, Oct 8, 2013 at 3:19 PM, Carlos Borroto <carlos.borroto@gmail.com> wrote:
On Tue, Oct 8, 2013 at 6:45 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers).
I have to say this was exactly my thoughts when developing a couple of tools recently.
I would like to have a suitable place where I can test my tools before promoting them to the Main Tool Shed. Because of the difference in the code base between Main and Test Tool Sheds, the later is not such a place. I know I could always run a local Tool Shed, but I feel having a public place for testing where I can ask for other users to test my work would be more useful.
I agree. There has been some discussion about related issues on the IUC list. One idea would be that the latest revision isn't automatically available for public installation (perhaps available only to those with the write access, i.e. the tool author or team), yet would still be automatically tested (and perhaps reviewed). Speaking for myself as a tool author, I would be happy to upload a fix to the tool shed, wait for the automated test results, and only then flag the revision as "released" for public usage (or upload a tweaked version to fix whatever wasn't quite right and try again). This would solve the problem of trying to use the Test Tool Shed for development and testing, when it doesn't match the current stable release and thus the environment into which the tool would be downloaded, installed, and used. (Of course, I'm assuming that the main Tool Shed test framework should/would be testing the tools using the stable release Galaxy code - the main query in the previous email) Regards, Peter
Peter, The following wiki page should provide additional information on the testing environments: http://wiki.galaxyproject.org/AutomatedToolTests#Automated_testing_framework... I should point out that I have not yet finished decoupling the revisions of Galaxy and the installation and testing scripts, but I hope to have that done today. --Dave B. On 10/08/2013 06:45 AM, Peter Cock wrote:
Splitting off from this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/016903.html
On Mon, Oct 7, 2013 at 4:24 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions.
--Dave B.
Thank you for your reply Dave, but I hope there has been a miscommunication here.
The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers).
I had been under the impression that the main "Tool Shed" (and the nightly tool tests run from it) was tracking the stable branch on galaxy-dist and therefore equivalent to what most Galaxy instances are running.
Are you saying that for both the main "Tool Shed" and the "Test Tool Shed" you are using the galaxy-dist default branch with the bleeding edge code to run the tests?
If so, this worries me as that means the main "Tool Shed" nightly tests are not indicative of what would happen once the tools are downloaded and installed to a galaxy-dist stable branch instance.
e.g. If I was to release a tool update using the latest <action> work then I would expect it to work fine on the "Test Tool Shed", but not to work on a Galaxy instance running the current stable release. Therefore, I would hope/expect it to fail on the main "Tool Shed" (due to using features not yet in the stable release).
(The <action> enhancement is just an example, this is a general question.)
Could you clarify the situation please?
Thanks,
Peter
On Tue, Oct 8, 2013 at 3:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
The following wiki page should provide additional information on the testing environments:
http://wiki.galaxyproject.org/AutomatedToolTests#Automated_testing_framework...
I should point out that I have not yet finished decoupling the revisions of Galaxy and the installation and testing scripts, but I hope to have that done today.
--Dave B.
Quoting the current wiki page contents,
Main Tool Shed
The main Tool Shed tracks the latest stable release of the Galaxy codebase, and the installation and testing framework instantiates a Galaxy instance running the same revision, into which repositories are installed from the main Tool Shed. The testing framework itself runs the latest revision of the galaxy-central repository in bitbucket.
That last sentence seems wrong (and to contradict the previous one). I expect the whole thing should be running the Galaxy stable release in order to be a fair test of what will happen when a tool from the Main Tool Shed is installed into a local Galaxy Instance running the stable release. Is this what you mean by you have not yet finished decoupling the revisions of Galaxy and the installation and testing scripts? Thanks, Peter
Peter, Yes, the testing framework needs to be maintained independently, in order to allow for enhancements to the test scripts without affecting the stable Galaxy that is instantiated by these scripts. --Dave B. On 10/08/2013 11:05 AM, Peter Cock wrote:
On Tue, Oct 8, 2013 at 3:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
The following wiki page should provide additional information on the testing environments:
http://wiki.galaxyproject.org/AutomatedToolTests#Automated_testing_framework...
I should point out that I have not yet finished decoupling the revisions of Galaxy and the installation and testing scripts, but I hope to have that done today.
--Dave B.
Quoting the current wiki page contents,
Main Tool Shed
The main Tool Shed tracks the latest stable release of the Galaxy codebase, and the installation and testing framework instantiates a Galaxy instance running the same revision, into which repositories are installed from the main Tool Shed. The testing framework itself runs the latest revision of the galaxy-central repository in bitbucket.
That last sentence seems wrong (and to contradict the previous one). I expect the whole thing should be running the Galaxy stable release in order to be a fair test of what will happen when a tool from the Main Tool Shed is installed into a local Galaxy Instance running the stable release.
Is this what you mean by you have not yet finished decoupling the revisions of Galaxy and the installation and testing scripts?
Thanks,
Peter
participants (3)
-
Carlos Borroto -
Dave Bouvier -
Peter Cock