Publish tool with Python dependencies to the toolshed
Dear all I have a Python script that I would like to publish to the tool shed. It has a couple dependencies, all Python modules (argparse, PyVCF, PyYAML). It also needs a Python version 2.7<=X<3, which is not available per default everywhere. Firstly, how do I specify these dependencies in the tool <requirements> tag? And secondly, once I have a correct set of tool definition files with tests and test data, how do I best get it onto the tool shed? I had a look through the wiki and the planemo documentation. Unfortunately how to handle these dependencies did not become immediately obvious (to me). Thanks for any pointers and sorry if this has been answered some place I did not look. Cheers Ulf -- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England 61 Colindale Ave, London NW9 5EQ ulf.schaefer@phe.gov.uk http://www.gov.uk/phe Protecting and improving the nation’s health ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
Hi Ulf, you most probably want your tool to depend on package_python_2_7_10 from the iuc (it will ensure your tool has access to 2.7.10 no matter what python the Galaxy executing your tool is using) The package is being developed here https://github.com/galaxyproject/tools-iuc/tree/master/packages/package_pyth... and is available at the MTS here https://toolshed.g2.bx.psu.edu/view/iuc/package_python_2_7_10/73631e818d3d You will need to add to your wrapper's xml something like: <requirements> <requirement type="package" version="2.7.10">python</requirement> </requirements> And you have to include tool_dependencies.xml file which will look something like: <?xml version="1.0"?> <tool_dependency> <package name="python" version="2.7.10"> <repository name="package_python_2_7_10" owner="iuc" prior_installation_required="True" /> </package> </tool_dependency> Please let me know if you have more questions. M. On Tue, Dec 1, 2015 at 10:54 AM Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
I have a Python script that I would like to publish to the tool shed. It has a couple dependencies, all Python modules (argparse, PyVCF, PyYAML). It also needs a Python version 2.7<=X<3, which is not available per default everywhere.
Firstly, how do I specify these dependencies in the tool <requirements> tag?
And secondly, once I have a correct set of tool definition files with tests and test data, how do I best get it onto the tool shed?
I had a look through the wiki and the planemo documentation. Unfortunately how to handle these dependencies did not become immediately obvious (to me).
Thanks for any pointers and sorry if this has been answered some place I did not look.
Cheers Ulf
-- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England 61 Colindale Ave, London NW9 5EQ ulf.schaefer@phe.gov.uk http://www.gov.uk/phe Protecting and improving the nation’s health
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ************************************************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Ulf, just adding to Martins great explanation that you should set enable_beta_tool_command_isolation = True in your galaxy.ini file if you are using a strict Python dependency with package_python_2_7_10. Cheers, Bjoern Am 01.12.2015 um 17:07 schrieb Martin Čech:
Hi Ulf,
you most probably want your tool to depend on package_python_2_7_10 from the iuc (it will ensure your tool has access to 2.7.10 no matter what python the Galaxy executing your tool is using)
The package is being developed here https://github.com/galaxyproject/tools-iuc/tree/master/packages/package_pyth... and is available at the MTS here https://toolshed.g2.bx.psu.edu/view/iuc/package_python_2_7_10/73631e818d3d
You will need to add to your wrapper's xml something like:
<requirements> <requirement type="package" version="2.7.10">python</requirement> </requirements>
And you have to include tool_dependencies.xml file which will look something like:
<?xml version="1.0"?> <tool_dependency> <package name="python" version="2.7.10"> <repository name="package_python_2_7_10" owner="iuc" prior_installation_required="True" /> </package> </tool_dependency>
Please let me know if you have more questions.
M.
On Tue, Dec 1, 2015 at 10:54 AM Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
I have a Python script that I would like to publish to the tool shed. It has a couple dependencies, all Python modules (argparse, PyVCF, PyYAML). It also needs a Python version 2.7<=X<3, which is not available per default everywhere.
Firstly, how do I specify these dependencies in the tool <requirements> tag?
And secondly, once I have a correct set of tool definition files with tests and test data, how do I best get it onto the tool shed?
I had a look through the wiki and the planemo documentation. Unfortunately how to handle these dependencies did not become immediately obvious (to me).
Thanks for any pointers and sorry if this has been answered some place I did not look.
Cheers Ulf
-- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England 61 Colindale Ave, London NW9 5EQ ulf.schaefer@phe.gov.uk http://www.gov.uk/phe Protecting and improving the nation’s health
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ************************************************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks Martin and Bjoern for your advice. Bjoern, does this modification to the galaxy.ini need to be made on all Galaxy instances that want to install and run this tool once its available in the shed? I assume it has nothing to do with the ability to publish it in the first place. Thanks Ulf On 01/12/15 16:41, Björn Grüning wrote:
Hi Ulf,
just adding to Martins great explanation that you should set
enable_beta_tool_command_isolation = True
in your galaxy.ini file if you are using a strict Python dependency with package_python_2_7_10.
Cheers, Bjoern
Am 01.12.2015 um 17:07 schrieb Martin Čech:
Hi Ulf,
you most probably want your tool to depend on package_python_2_7_10 from the iuc (it will ensure your tool has access to 2.7.10 no matter what python the Galaxy executing your tool is using)
The package is being developed here https://github.com/galaxyproject/tools-iuc/tree/master/packages/package_pyth... and is available at the MTS here https://toolshed.g2.bx.psu.edu/view/iuc/package_python_2_7_10/73631e818d3d
You will need to add to your wrapper's xml something like:
<requirements> <requirement type="package" version="2.7.10">python</requirement> </requirements>
And you have to include tool_dependencies.xml file which will look something like:
<?xml version="1.0"?> <tool_dependency> <package name="python" version="2.7.10"> <repository name="package_python_2_7_10" owner="iuc" prior_installation_required="True" /> </package> </tool_dependency>
Please let me know if you have more questions.
M.
On Tue, Dec 1, 2015 at 10:54 AM Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
I have a Python script that I would like to publish to the tool shed. It has a couple dependencies, all Python modules (argparse, PyVCF, PyYAML). It also needs a Python version 2.7<=X<3, which is not available per default everywhere.
Firstly, how do I specify these dependencies in the tool <requirements> tag?
And secondly, once I have a correct set of tool definition files with tests and test data, how do I best get it onto the tool shed?
I had a look through the wiki and the planemo documentation. Unfortunately how to handle these dependencies did not become immediately obvious (to me).
Thanks for any pointers and sorry if this has been answered some place I did not look.
Cheers Ulf
-- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England 61 Colindale Ave, London NW9 5EQ ulf.schaefer@phe.gov.uk http://www.gov.uk/phe Protecting and improving the nation’s health
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ************************************************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
Hi,
Thanks Martin and Bjoern for your advice.
Bjoern, does this modification to the galaxy.ini need to be made on all Galaxy instances that want to install and run this tool once its available in the shed? I assume it has nothing to do with the ability to publish it in the first place.
It's only needed by Galaxy instance which runs these tools. The reason is that Galaxy's Python env and the Python env from the Tool needs to be completely separated. In theory it is now possible that you run tools against Python 3.5 in a Python 2.6 Galaxy installation. Cheers, Bjoern
Thanks Ulf
On 01/12/15 16:41, Björn Grüning wrote:
Hi Ulf,
just adding to Martins great explanation that you should set
enable_beta_tool_command_isolation = True
in your galaxy.ini file if you are using a strict Python dependency with package_python_2_7_10.
Cheers, Bjoern
Am 01.12.2015 um 17:07 schrieb Martin Čech:
Hi Ulf,
you most probably want your tool to depend on package_python_2_7_10 from the iuc (it will ensure your tool has access to 2.7.10 no matter what python the Galaxy executing your tool is using)
The package is being developed here https://github.com/galaxyproject/tools-iuc/tree/master/packages/package_pyth... and is available at the MTS here https://toolshed.g2.bx.psu.edu/view/iuc/package_python_2_7_10/73631e818d3d
You will need to add to your wrapper's xml something like:
<requirements> <requirement type="package" version="2.7.10">python</requirement> </requirements>
And you have to include tool_dependencies.xml file which will look something like:
<?xml version="1.0"?> <tool_dependency> <package name="python" version="2.7.10"> <repository name="package_python_2_7_10" owner="iuc" prior_installation_required="True" /> </package> </tool_dependency>
Please let me know if you have more questions.
M.
On Tue, Dec 1, 2015 at 10:54 AM Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
I have a Python script that I would like to publish to the tool shed. It has a couple dependencies, all Python modules (argparse, PyVCF, PyYAML). It also needs a Python version 2.7<=X<3, which is not available per default everywhere.
Firstly, how do I specify these dependencies in the tool <requirements> tag?
And secondly, once I have a correct set of tool definition files with tests and test data, how do I best get it onto the tool shed?
I had a look through the wiki and the planemo documentation. Unfortunately how to handle these dependencies did not become immediately obvious (to me).
Thanks for any pointers and sorry if this has been answered some place I did not look.
Cheers Ulf
-- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England 61 Colindale Ave, London NW9 5EQ ulf.schaefer@phe.gov.uk http://www.gov.uk/phe Protecting and improving the nation’s health
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ************************************************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
Hi again all So I uploaded a tool with a single dependency to python2.7.10 to the testtoolshed (https://testtoolshed.g2.bx.psu.edu/view/ulfschaefer/filter_vcf/d0afc9c121a5). Then I try to install it back on my test Galaxy instance. There are some error messages when trying to install the dependencies of package_python_2_7_10 (e.g. package readline_6_3 or package_gdbm_1_11) These seem to be resolved when I go into the admin interface and uninstall and reinstall these one by one. However, when I try to reinstall the package_python_2_7_10 itself after uninstalling it, I get this: Error - <type 'exceptions.TypeError'>: coercing to Unicode: need string or buffer, NoneType found URL: http://testgalaxy/admin_toolshed/manage_tool_dependencies?repository_id=d071e794759ab192&operation=install File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 151 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 133 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 191 in handle_request body = method( trans, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/decorators.py', line 92 in decorator return func( self, trans, *args, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 891 in manage_tool_dependencies status=status ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/decorators.py', line 92 in decorator return func( self, trans, *args, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 494 in initiate_tool_dependency_installation tool_shed_repository.repo_path( trans.app ) ) File '/home/galaxy/galaxy-dist/lib/tool_shed/util/hg_util.py', line 138 in get_config_from_disk for root, dirs, files in os.walk( relative_install_dir ): File '/usr/lib64/python2.6/os.py', line 276 in walk names = listdir(top) TypeError: coercing to Unicode: need string or buffer, NoneType found What have I done wrong so that relative_install_dir is not set. Wasn't really a problem when I installed the other dependencies. Any help would be appreciated. I tried switching if off and on again, but unfortunately it did not help. Thanks Ulf On 01/12/15 16:07, Martin Čech wrote:
Hi Ulf,
you most probably want your tool to depend on package_python_2_7_10 from the iuc (it will ensure your tool has access to 2.7.10 no matter what python the Galaxy executing your tool is using)
The package is being developed here https://github.com/galaxyproject/tools-iuc/tree/master/packages/package_pyth... and is available at the MTS here https://toolshed.g2.bx.psu.edu/view/iuc/package_python_2_7_10/73631e818d3d
You will need to add to your wrapper's xml something like:
<requirements> <requirement type="package" version="2.7.10">python</requirement> </requirements>
And you have to include tool_dependencies.xml file which will look something like:
<?xml version="1.0"?> <tool_dependency> <package name="python" version="2.7.10"> <repository name="package_python_2_7_10" owner="iuc" prior_installation_required="True" /> </package> </tool_dependency>
Please let me know if you have more questions.
M.
On Tue, Dec 1, 2015 at 10:54 AM Ulf Schaefer <Ulf.Schaefer@phe.gov.uk <mailto:Ulf.Schaefer@phe.gov.uk>> wrote:
Dear all
I have a Python script that I would like to publish to the tool shed. It has a couple dependencies, all Python modules (argparse, PyVCF, PyYAML). It also needs a Python version 2.7<=X<3, which is not available per default everywhere.
Firstly, how do I specify these dependencies in the tool <requirements> tag?
And secondly, once I have a correct set of tool definition files with tests and test data, how do I best get it onto the tool shed?
I had a look through the wiki and the planemo documentation. Unfortunately how to handle these dependencies did not become immediately obvious (to me).
Thanks for any pointers and sorry if this has been answered some place I did not look.
Cheers Ulf
-- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England 61 Colindale Ave, London NW9 5EQ ulf.schaefer@phe.gov.uk <mailto:ulf.schaefer@phe.gov.uk> http://www.gov.uk/phe Protecting and improving the nation’s health
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ************************************************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
participants (3)
-
Björn Grüning -
Martin Čech -
Ulf Schaefer