I have read through the documentation a couple times, but I still have a few questions about the recent tool shed enhancements. At MSI we have a testing environment and a production environment and I want to make sure the tool versions and configurations don't get out of sync, I would also like to test everything in our testing environment before it reaches production. Is there a recommended way to accomplish this rather than just manually repeating the same set of UI interactions twice? Can I just import tools through the testing UI and run the ./scripts/migrate_tools/XXXX scripts on our testing repository and then move the resulting migrated_tools_conf.xml and integrated_tool_panel.xml files into production? I have follow up questions, but I will wait for a response on this point. Also as you are removing tools from Galaxy and placing them into our tool shed, what is the recommended course of actions for deployers that have made local minor tweaks to those tool configs and scripts and adapt them to our local environments? Along the same lines, what is the recommended course of action if we need to make minor tweaks to tools pulled into through the UI to adapt them to our institution. Thanks for your time, -John ------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223
Hi John, On Jun 7, 2012, at 11:55 PM, John Chilton wrote:
I have read through the documentation a couple times, but I still have a few questions about the recent tool shed enhancements.
At MSI we have a testing environment and a production environment and I want to make sure the tool versions and configurations don't get out of sync, I would also like to test everything in our testing environment before it reaches production.
Is there a recommended way to accomplish this rather than just manually repeating the same set of UI interactions twice?
Can I just import tools through the testing UI and run the ./scripts/migrate_tools/XXXX scripts on our testing repository and then move the resulting migrated_tools_conf.xml and integrated_tool_panel.xml files into production? I have follow up questions, but I will wait for a response on this point.
Tools that used to be in the Galaxy distribution but have been moved to the main Galaxy tool shed are automatically installed when you start up your Galaxy server and presented with the option of running the migration script to automatically install the tools that were migrated in the specific Galaxy distribution release. If you choose to install the tools, they are installed only in that specific Galaxy instance. Installation produces mercurial repositories that include the tools on disk in your Galaxy server environment. Several other things are produced as well, including database records for the installation. Each Galaxy instance consists of it's own separate set of components, this installation process must be done for each instance. The installation is fully automatic, requiring little interaction on the part of the Galaxy admin, and doesn't require much time, so performing the process for each Galaxy instance should not be too intensive. Also, the tools that are installed into each Galaxy instance's tool panel are only those tools that were originally defined in the tool panel configuration file (tool_conf.xml). This approach provides for the case where each Galaxy instance having different tools defined will not be altered by the migration process.
Also as you are removing tools from Galaxy and placing them into our tool shed, what is the recommended course of actions for deployers that have made local minor tweaks to those tool configs and scripts and adapt them to our local environments? Along the same lines, what is the recommended course of action if we need to make minor tweaks to tools pulled into through the UI to adapt them to our institution.
In both cases you should upload your proprietary tools to either a local Galaxy tool shed that you administer, or the main Galaxy tool shed if you want. You can choose to not execute any of the tool migration scripts, so the Galaxy tools that were migrated from the distribution will not be installed into your Galaxy environment. You can use the Galaxy admin UI to install your proprietary versions of the migrated tools from the tool shed in which you chose to upload and store them. New versions of the tools can be uploaded to respective tool shed repositories over time.
Thanks for your time, -John
------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hello Greg, Thanks for the prompt and detailed response (though it did make me sad). I think deploying tested, static components and configurations to production environments and having production environments not depending on outside services (like the tool shed) should be considered best practices. Oh well, I guess. Would there be a way to at least automate the pulling of tools in. For instance, would it make sense to tweak InstallManager to parse a new kind of migration file that is a lot like the "official" migration files, but with the sections defined in the file. For this new kind of migration, the InstallManager would then import everything in the file and not just the tools that are also in a tool_conf? Does that make sense? If yes, I imagine it could be modified to handle updates the same way? Rephrased, I guess the idea would be to have the sequence of official galaxy migrations that check tool_conf, and then have a sequence of migration defined by the deployer that could be used to install new tools or update existing ones. My concern isn't just with the dev to production transition, it is also the ability to sort of programmatically define Galaxy installations the way I am doing with the galaxy-vm-launcher (https://bitbucket.org/jmchilton/galaxy-vm-launcher) or the way mi-deployment works. Thanks again for your time and patience in explaining these things to me, -John On Fri, Jun 8, 2012 at 4:19 AM, Greg Von Kuster <greg@bx.psu.edu> wrote:
Hi John,
On Jun 7, 2012, at 11:55 PM, John Chilton wrote:
I have read through the documentation a couple times, but I still have a few questions about the recent tool shed enhancements.
At MSI we have a testing environment and a production environment and I want to make sure the tool versions and configurations don't get out of sync, I would also like to test everything in our testing environment before it reaches production.
Is there a recommended way to accomplish this rather than just manually repeating the same set of UI interactions twice?
Can I just import tools through the testing UI and run the ./scripts/migrate_tools/XXXX scripts on our testing repository and then move the resulting migrated_tools_conf.xml and integrated_tool_panel.xml files into production? I have follow up questions, but I will wait for a response on this point.
Tools that used to be in the Galaxy distribution but have been moved to the main Galaxy tool shed are automatically installed when you start up your Galaxy server and presented with the option of running the migration script to automatically install the tools that were migrated in the specific Galaxy distribution release. If you choose to install the tools, they are installed only in that specific Galaxy instance. Installation produces mercurial repositories that include the tools on disk in your Galaxy server environment. Several other things are produced as well, including database records for the installation. Each Galaxy instance consists of it's own separate set of components, this installation process must be done for each instance. The installation is fully automatic, requiring little interaction on the part of the Galaxy admin, and doesn't require much time, so performing the process for each Galaxy instance should not be too intensive. Also, the tools that are installed into each Galaxy instance's tool panel are only those tools that were originally defined in the tool panel configuration file (tool_conf.xml). This approach provides for the case where each Galaxy instance having different tools defined will not be altered by the migration process.
Also as you are removing tools from Galaxy and placing them into our tool shed, what is the recommended course of actions for deployers that have made local minor tweaks to those tool configs and scripts and adapt them to our local environments? Along the same lines, what is the recommended course of action if we need to make minor tweaks to tools pulled into through the UI to adapt them to our institution.
In both cases you should upload your proprietary tools to either a local Galaxy tool shed that you administer, or the main Galaxy tool shed if you want. You can choose to not execute any of the tool migration scripts, so the Galaxy tools that were migrated from the distribution will not be installed into your Galaxy environment. You can use the Galaxy admin UI to install your proprietary versions of the migrated tools from the tool shed in which you chose to upload and store them. New versions of the tools can be uploaded to respective tool shed repositories over time.
Thanks for your time, -John
------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi John, On Jun 8, 2012, at 1:22 PM, John Chilton wrote:
Hello Greg,
Thanks for the prompt and detailed response (though it did make me sad). I think deploying tested, static components and configurations to production environments and having production environments not depending on outside services (like the tool shed) should be considered best practices.
I'm not sure I understand this issue. What processes are you using to upgrade your test and production servers with new Galaxy distributions? If you are pulling new Galaxy distributions from our Galaxy dist repository in bitbucket, then pulling tools from the Galaxy tool shed is not much different - both are outside services. Updating your test environment, determining it is functionally correct, and then updating your production environment using the same approach would generally follow a best practice approach. This is the approach we are currently using for our public test and main Galaxy instances at Penn State.
Oh well, I guess. Would there be a way to at least automate the pulling of tools in.
The process is completely automated - all you need to do execute a script, something like this: sh ./scripts/migrate_tools/0002_tools.sh This is the same process used when the Galaxy database schema migrates as part of a new Galaxy release, except in that case you would run a script like this: sh manage_db.sh upgrade
For instance, would it make sense to tweak InstallManager to parse a new kind of migration file that is a lot like the "official" migration files, but with the sections defined in the file. For this new kind of migration, the InstallManager would then import everything in the file and not just the tools that are also in a tool_conf? Does that make sense? If yes, I imagine it could be modified to handle updates the same way?
If I understand this correctly, this is how the InstallManage works. The entire tool shed repository is installed into your local Galaxy environment, but only the tools that are currently defined in your tool_conf.xml file are loaded into your tool panel.
Rephrased, I guess the idea would be to have the sequence of official galaxy migrations that check tool_conf, and then have a sequence of migration defined by the deployer that could be used to install new tools or update existing ones.
My concern isn't just with the dev to production transition, it is also the ability to sort of programmatically define Galaxy installations the way I am doing with the galaxy-vm-launcher (https://bitbucket.org/jmchilton/galaxy-vm-launcher) or the way mi-deployment works.
You have complete control with these migrations. You can chose to not install any tools shed repositories, and just start your Galaxy server. If you choose to install the defined repositories, you have control over what specific tools included in the repository are loaded into your tool panel by having them defined in your tool_conf.xml prior to the installation. This whole process is associated only with tools that have moved from the Galaxy distribution to the tool shed.
Thanks again for your time and patience in explaining these things to me, -John
On Fri, Jun 8, 2012 at 4:19 AM, Greg Von Kuster <greg@bx.psu.edu> wrote:
Hi John,
On Jun 7, 2012, at 11:55 PM, John Chilton wrote:
I have read through the documentation a couple times, but I still have a few questions about the recent tool shed enhancements.
At MSI we have a testing environment and a production environment and I want to make sure the tool versions and configurations don't get out of sync, I would also like to test everything in our testing environment before it reaches production.
Is there a recommended way to accomplish this rather than just manually repeating the same set of UI interactions twice?
Can I just import tools through the testing UI and run the ./scripts/migrate_tools/XXXX scripts on our testing repository and then move the resulting migrated_tools_conf.xml and integrated_tool_panel.xml files into production? I have follow up questions, but I will wait for a response on this point.
Tools that used to be in the Galaxy distribution but have been moved to the main Galaxy tool shed are automatically installed when you start up your Galaxy server and presented with the option of running the migration script to automatically install the tools that were migrated in the specific Galaxy distribution release. If you choose to install the tools, they are installed only in that specific Galaxy instance. Installation produces mercurial repositories that include the tools on disk in your Galaxy server environment. Several other things are produced as well, including database records for the installation. Each Galaxy instance consists of it's own separate set of components, this installation process must be done for each instance. The installation is fully automatic, requiring little interaction on the part of the Galaxy admin, and doesn't require much time, so performing the process for each Galaxy instance should not be too intensive. Also, the tools that are installed into each Galaxy instance's tool panel are only those tools that were originally defined in the tool panel configuration file (tool_conf.xml). This approach provides for the case where each Galaxy instance having different tools defined will not be altered by the migration process.
Also as you are removing tools from Galaxy and placing them into our tool shed, what is the recommended course of actions for deployers that have made local minor tweaks to those tool configs and scripts and adapt them to our local environments? Along the same lines, what is the recommended course of action if we need to make minor tweaks to tools pulled into through the UI to adapt them to our institution.
In both cases you should upload your proprietary tools to either a local Galaxy tool shed that you administer, or the main Galaxy tool shed if you want. You can choose to not execute any of the tool migration scripts, so the Galaxy tools that were migrated from the distribution will not be installed into your Galaxy environment. You can use the Galaxy admin UI to install your proprietary versions of the migrated tools from the tool shed in which you chose to upload and store them. New versions of the tools can be uploaded to respective tool shed repositories over time.
Thanks for your time, -John
------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (3)
-
Greg Von Kuster -
John Chilton -
John Chilton