OK, thanks for the collection of resources, that's useful!
I’m posting to start a discussion about how galaxy is going to handle
interactions with Docker.
I know there are already some tools and resources out there that make use
of Docker. As I understood, we can already specify Docker as a job
destination in the tool xml, along with an image to use (See
https://github.com/apetkau/galaxy-hackathon-2014/tree/master/smalt for a
detailed example). This is cool, and allows for the controlling of
resources, but I think we could take the integration to the next step if we
allowed the generation of Docker images in galaxy.
That's pretty cool indeed, I didn't know this.
I was thinking if there is a trusted baseimage (ie without root access), we
could let users install
Sure. My idea was that the normal, non-admin galaxy user would be dropped into a more or less populated container, with normal user rights, so
that there shouldn't be a way to do much harm to the docker host. Then they could install their favorite R, python, perl, ... package (or import a toolshed tool ...).
When satisfied and all software is there, a commit is triggered, and the image would be stored for later use.
Or alternatively the steps necessary for installation would be transformed into a script that is added and executed during the Docker build process.
The advantage here would be that you don't need to be admin to do this.
I'll try to get the basic functionality working, a demo of this might clear things up.
all the packages they need, commit after the install, and let them use this
new docker image. This could be beneficial for generating new tools with a
sandboxed version of the Galaxy Tool Factory (
http://www.ncbi.nlm.nih.gov/pubmed/23024011).
I am currently working on this (
https://bitbucket.org/mvdbeek/dockertoolfactory), and I would like to have
your opinion on how to manage these user-generated Docker images.
Some questions that I came across:
Do we store the committed images in the user’s history (they could
potentially become very big!)?
How to display available images to the user? Something like a per-user
image registry?
How to identify available images (dataset_id as the tag?)
How to transfer user-generated images to the toolshed?
Is it a good idea at all to store user-generated images in the toolshed?
What do we do if the security of the baseimage is compromised? Obviously
we can blacklist execution of images, but what if somebody installed a
dangerous image from the toolshed, and is not aware of this?
For example I think most of the time an IPython instance would be sufficient. You will only store notebook files in your history (small) and execute them on new datasets.
Yes, especially since you can also interact with R and the shell inside IPython, but right now it can't be used inside a workflow, for example ... unless I missed something.
I would be really interested in your use-cases.
Well, in my experience, many biologists working with genomics data *can* put together a commandline, but few are going to use galaxy,
because they can't run the specific tool that is not available in the institute's instance, or they need a specific option not exposed ... . And so the students/ other people in their team are also not going to use galaxy.
I think the moment they had the possibility to run their code, I think a lot of them could be lured in the galaxy ecosystem. If you show them how easy it is to share their analyses with coworkers ... .
So you can let those people run any script. If it works as expected, it can be used in workflows, and/or transformed in a (local) toolshed tool, that would even allow version control.
Of course, a toolshed tool is not automatically a good tool, but it's something to start with!
You could even think about letting them import tools from the toolshed, and have them sandboxed transparently.
So that would be the main use case, basically the same target audience of the original toolfactory, but extended to non-admins.
It might not be a good idea for the main galaxy instance, but why not do this for the server at your local institute?
Imho, all this should be taken with care, in the end we would like to have proper tools and images in the toolshed and reusable for many others. I consider IPython Integration and the Toolfactory as bonus and tools mostly for developers. Nothing a biologist should deal with. And we should take care to not over complicate the tool setup/usage.
I agree, it has to stay simple (or become easier!), shareable and reproducible. But I think we can come up with ways to get more command-line people on board without increasing complexity.
I disagree on one part though, I think that biologists dealing with big data have to know at least one programming/scripting language, and why not having them learn inside galaxy and ipython, with their own data?
That would be much better than GFFs edited in excel ... .
Ultimately I think this could be a way to have advanced users run their own
scripts inside galaxy, to generate their own tools and tool-dependencies
inside galaxy, and why not, even have “user-space tools”.
It would bridge the gap between galaxy users and galaxy tool-developers,
so I’m curious what you’re thinking about this.
Bjoern
Sure, I'm looking forward to how this is going to develop!
More feedback welcome!
Cheers,
Marius
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/