Dear Galaxy Team,
Two questions.
1) Is there a bare-bones version of Galaxy available somewhere? That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
2) Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base. I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
dear keith,
in our work on building the Linguistic Analysis Portal (LAP) at the University of Oslo, we have had to confront the same two issues you mention. our general approach has been to separate LAP-specific code (and specifications) from the Galaxy tree as much as possible.
for example, we keep our tool descriptions, tool implementations, job runner configurations, job resource specifications, and such in a separately versioned tree and use the various options in ‘galaxy.ini’ to point to our own variants, e.g.
tool_path = /home/laportal/tools tool_config_file = /home/laportal/tools/config.xml job_config_file = /home/laportal/tools/jobs.xml job_resource_params_file = /home/laportal/tools/resources.xml
for the upload tool, we modify the JavaScript files (‘upload-row.js’ and ‘upload-view.js’) to hide UI elements (notably the Genome selection) that do not apply to us, and we have added custom code to the actual tool implementation to post-process incoming data. as for the latter changes, our modified ‘upload.py’ and ‘upload.xml’ reside outside the Galaxy tree (with the other tools), so our only risk is that we need to validate compatibility (or import updates) when moving to newer Galaxy versions. for the JavaScript changes, we keep a separate copy of the handful of UI files (from below ‘client/galaxy/’) that we modify outside of Galaxy and use a modified ‘GruntFile.js’ (with different ‘dest’ targets) to make grunt(1) propagate our changes into the Galaxy tree.
maybe some of these techniques could work for you too? oe
On Thu, Mar 31, 2016 at 10:17 PM, Suderman Keith suderman@cs.vassar.edu wrote:
Dear Galaxy Team,
Two questions.
- Is there a bare-bones version of Galaxy available somewhere? That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
- Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base. I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
Cheers, Keith
Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Stephan,
Thank you for the pointers, they will definitely help. Would you be willing to share any of the code you have for the upload tool?
Will you be at the LREC conference in Portorož? It might be fun to arrange a meet up of all Galaxy users if we can.
Cheers, Keith
On Mar 31, 2016, at 5:12 PM, Stephan Oepen oe@ifi.uio.no wrote:
dear keith,
in our work on building the Linguistic Analysis Portal (LAP) at the University of Oslo, we have had to confront the same two issues you mention. our general approach has been to separate LAP-specific code (and specifications) from the Galaxy tree as much as possible.
for example, we keep our tool descriptions, tool implementations, job runner configurations, job resource specifications, and such in a separately versioned tree and use the various options in ‘galaxy.ini’ to point to our own variants, e.g.
tool_path = /home/laportal/tools tool_config_file = /home/laportal/tools/config.xml job_config_file = /home/laportal/tools/jobs.xml job_resource_params_file = /home/laportal/tools/resources.xml
for the upload tool, we modify the JavaScript files (‘upload-row.js’ and ‘upload-view.js’) to hide UI elements (notably the Genome selection) that do not apply to us, and we have added custom code to the actual tool implementation to post-process incoming data. as for the latter changes, our modified ‘upload.py’ and ‘upload.xml’ reside outside the Galaxy tree (with the other tools), so our only risk is that we need to validate compatibility (or import updates) when moving to newer Galaxy versions. for the JavaScript changes, we keep a separate copy of the handful of UI files (from below ‘client/galaxy/’) that we modify outside of Galaxy and use a modified ‘GruntFile.js’ (with different ‘dest’ targets) to make grunt(1) propagate our changes into the Galaxy tree.
maybe some of these techniques could work for you too? oe
On Thu, Mar 31, 2016 at 10:17 PM, Suderman Keith suderman@cs.vassar.edu wrote:
Dear Galaxy Team,
Two questions.
- Is there a bare-bones version of Galaxy available somewhere? That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
- Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base. I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
Cheers, Keith
Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Hi Keith,
with this PR merged https://github.com/bgruening/docker-galaxy-stable/pull/164
You can use the Galaxy Docker Image and pass in the -e BARE=True option. This will give you an production ready Galaxy Instance without tools, but with FTP Server, scheduler and activated TS.
You could even create your own flavour out of it. Have a look at the readme for more informations:
https://github.com/bgruening/docker-galaxy-stable/blob/master/README.md#exte...
Hope this helps, I will merge the PR over the weekend hopefully.
Cheers, Bjoern
Am 31.03.2016 um 22:17 schrieb Suderman Keith:
Dear Galaxy Team,
Two questions.
- Is there a bare-bones version of Galaxy available somewhere? That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
- Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base. I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
Cheers, Keith
Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Björn,
Have you pushed the new version to the Docker Hub? When I run the galaxy-stable image with the -e BARE=true option I still get a Galaxy instance with the default tools.
Also, does this really give me a “bare” Galaxy instance, or just a Galaxy instance with an empty tool_conf.xml file?
Cheers, Keith
On Apr 1, 2016, at 11:21 AM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Keith,
with this PR merged https://github.com/bgruening/docker-galaxy-stable/pull/164
You can use the Galaxy Docker Image and pass in the -e BARE=True option. This will give you an production ready Galaxy Instance without tools, but with FTP Server, scheduler and activated TS.
You could even create your own flavour out of it. Have a look at the readme for more informations:
https://github.com/bgruening/docker-galaxy-stable/blob/master/README.md#exte...
Hope this helps, I will merge the PR over the weekend hopefully.
Cheers, Bjoern
Am 31.03.2016 um 22:17 schrieb Suderman Keith:
Dear Galaxy Team,
Two questions.
- Is there a bare-bones version of Galaxy available somewhere? That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
- Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base. I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
Cheers, Keith
Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Hi Keith,
Hi Björn,
Have you pushed the new version to the Docker Hub? When I run the galaxy-stable image with the -e BARE=true option I still get a Galaxy instance with the default tools.
Yes it is, but you need the :dev version, aka:
docker run -i -t -p 8080:80 bgruening/galaxy-stable:dev
(hope it works, I'm currently not on my PC)
Also, does this really give me a “bare” Galaxy instance, or just a Galaxy instance with an empty tool_conf.xml file?
It only deactivates the tools in tool_conf.xml files. Is this is problem? These tools do not have any dependencies other than python and disc space should not be a problem as these are only a bunch of XML files.
The my comments inline.
Am 31.03.2016 um 22:17 schrieb Suderman Keith:
Dear Galaxy Team,
Two questions.
- Is there a bare-bones version of Galaxy available somewhere?
That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns? Disc-space? As a fist shoot I would try to hide genomic-specifc stuff from users. In a long run we can talk about this with the Galaxy team, isn't it. I know there is already a consensus of getting tools into the TS and have as less as possible tools in Galaxy by default. If you look 3 years back, Galaxy was full of tools, nowadays it's more or less text manipulation tools without dependencies. So plain XML files with a few scripts.
- Is is possible to either a) modify the existing upload tool,
or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Hi Björn,
Yes it is, but you need the :dev version,
DOH, I should have known…
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Currently I think I have just removed the datatypes. I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns?
Not really concerns, I just would like to remove as much unneeded code as possible. I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use. Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers. I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
I think you are correct; modifying the current upload tool is the way to go. A plugin mechanism for post-processing uploaded files sounds like a great idea!
Cheers, Keith
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
[Resending (again) as GMail keeps mangling my email address… sorry for triple posting...]
Hi Björn,
Yes it is, but you need the :dev version,
DOH, I should have known…
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Currently I think I have just removed the datatypes. I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns?
Not really concerns, I just would like to remove as much unneeded code as possible. I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use. Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers. I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
I think you are correct; modifying the current upload tool is the way to go. A plugin mechanism for post-processing uploaded files sounds like a great idea!
Cheers, Keith
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
On Apr 4, 2016, at 7:01 PM, Suderman Keith keithsuderman@gmail.com wrote:
Hi Björn,
Yes it is, but you need the :dev version,
DOH, I should have known…
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Currently I think I have just removed the datatypes. I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns?
Not really concerns, I just would like to remove as much unneeded code as possible. I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use. Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers. I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
I think you are correct; modifying the current upload tool is the way to go. A plugin mechanism for post-processing uploaded files sounds like a great idea!
Cheers, Keith
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Hi Keith,
Hi Björn,
Yes it is, but you need the :dev version,
DOH, I should have known…
:)
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Currently I think I have just removed the datatypes. I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.
We could think about removing datatypes as well from the BARE Galaxy Docker container, similar to what we have done now for tools, if you like.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns?
Not really concerns, I just would like to remove as much unneeded code as possible. I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use.
Mh, but I see a bigger problem here. If you have conflicting extensions, could you not prefix them? I have the small dream of having a university workbench with all tools. So offering your tools as well as genomics and in this case I will have the problems again. Removing genomic datatypes makes sense in your case and I'm happy to work on this. But it would be nice if we could also try to make these tools, datatypes work on every Galaxy instance.
Can we prepend them with 'nltk_'?
Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers. I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.
True, but if you deactivate datatypes the code should not be loaded and with every modification we do that is not tested we increase the attack vector again.
Let me know what you decide and if you like to use the Docker container and I will find time to get rid of datatypes in BARE mode.
Thanks, Bjoern
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
I think you are correct; modifying the current upload tool is the way to go. A plugin mechanism for post-processing uploaded files sounds like a great idea!
Cheers, Keith
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
[Resending because GMail keeps mangling my email address…]
Hi Björn,
Yes it is, but you need the :dev version,
DOH, I should have known…
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Currently I think I have just removed the datatypes. I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns?
Not really concerns, I just would like to remove as much unneeded code as possible. I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use. Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers. I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
I think you are correct; modifying the current upload tool is the way to go. A plugin mechanism for post-processing uploaded files sounds like a great idea!
Cheers, Keith
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
On Apr 3, 2016, at 5:05 AM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Keith,
Hi Björn,
Have you pushed the new version to the Docker Hub? When I run the galaxy-stable image with the -e BARE=true option I still get a Galaxy instance with the default tools.
Yes it is, but you need the :dev version, aka:
docker run -i -t -p 8080:80 bgruening/galaxy-stable:dev
(hope it works, I'm currently not on my PC)
Also, does this really give me a “bare” Galaxy instance, or just a Galaxy instance with an empty tool_conf.xml file?
It only deactivates the tools in tool_conf.xml files. Is this is problem? These tools do not have any dependencies other than python and disc space should not be a problem as these are only a bunch of XML files.
The my comments inline.
Am 31.03.2016 um 22:17 schrieb Suderman Keith:
Dear Galaxy Team,
Two questions.
- Is there a bare-bones version of Galaxy available somewhere?
That is, Galaxy with no tools pre-installed? We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed. Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
Can you explain what you delete? I do really support this idea as we are also developing cheminformatics or imaging flavours.
Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
I don't think so you are pioneering here! But I think the gain of this step is currently not much, isn't it? What are your concerns? Disc-space? As a fist shoot I would try to hide genomic-specifc stuff from users. In a long run we can talk about this with the Galaxy team, isn't it. I know there is already a consensus of getting tools into the TS and have as less as possible tools in Galaxy by default. If you look 3 years back, Galaxy was full of tools, nowadays it's more or less text manipulation tools without dependencies. So plain XML files with a few scripts.
- Is is possible to either a) modify the existing upload tool,
or b) implement a new upload tool? Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded. I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.
Yes it is, actually you need to have the upload tool, otherwise many things do not work. I would like to go with a)!
That is put big #if #else conditions in the JS and Galaxy code to display genomic-specific content ... we could have a galaxy.ini config to trigger this #if.
I do think we have a good chance to get this accepted.
Second step, and this will take for careful thoughts, is a plugin mechanism for post-upload modifications. The currently upload tool has some kind of these functionality, for example the "Space-To-Tab" converter. Ideally exposing tools (even from the TS)?
I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
Let's try to stay as upstream as possible. upload.xml is so tightly integrated in Galaxy that it is better to get code upstream I assume. And I know Galaxy devs like the idea of being less genomic centric and more a general framework. It's just that no one had time for this until know.
However, could I follow the approach suggested in that thread? I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool? How far back in time would I have to go to find an upload.py that was a standalone tool?
I think this will cause more trouble ... :( Let me know what you decide I will try to help you as much as I can.
Cheers, Bjoern
Cheers, Keith
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY
galaxy-dev@lists.galaxyproject.org
-
Bjoern Gruening -
Björn Grüning
-
Stephan Oepen -
Suderman Keith -
Suderman Keith