Workflow improvement suggestion
We are finding galaxy really useful but with the implications of users not cleaning up after themselves when running long workflow is hitting on our storage. This could be easily fixed by allowing users to mark steps in a workflow as "KEPT" or "DELETED". Then whenever the workflow is run the files that are just created in passing to the next step and aren't required as output can be automatically marked as deleted and cleaned up as part of normal cleanup operations. Would you consider implementing this, or if not - which modules would be best to look at if we want to do it ourselves? Cheers Dennis
Hi Dennis, You may have seen that Dannon recently committed support for post-job- actions in worfklows that allow for some of the things you've been needing (e.g. change datatype). One of the actions is "hide dataset". One could definitely create a "delete dataset" action based on that, which marked the dataset as deleted so it can be picked up be the cleanup script later. It should be an easy change if you want to take a stab at it. Actions are implemented in the module "galaxy.jobs.actions.post" Thanks, James On Jun 14, 2010, at 6:03 PM, Dennis Gascoigne wrote:
We are finding galaxy really useful but with the implications of users not cleaning up after themselves when running long workflow is hitting on our storage.
This could be easily fixed by allowing users to mark steps in a workflow as "KEPT" or "DELETED". Then whenever the workflow is run the files that are just created in passing to the next step and aren't required as output can be automatically marked as deleted and cleaned up as part of normal cleanup operations.
Would you consider implementing this, or if not - which modules would be best to look at if we want to do it ourselves?
Hi all, where can I find more info on using post-job-actions in workflows, I ould like to implement this on my local install. Thanks Shaun Webb where can I find more info about Quoting James Taylor <james@bx.psu.edu>:
Hi Dennis,
You may have seen that Dannon recently committed support for post-job-actions in worfklows that allow for some of the things you've been needing (e.g. change datatype). One of the actions is "hide dataset". One could definitely create a "delete dataset" action based on that, which marked the dataset as deleted so it can be picked up be the cleanup script later.
It should be an easy change if you want to take a stab at it. Actions are implemented in the module "galaxy.jobs.actions.post"
Thanks, James
On Jun 14, 2010, at 6:03 PM, Dennis Gascoigne wrote:
We are finding galaxy really useful but with the implications of users not cleaning up after themselves when running long workflow is hitting on our storage.
This could be easily fixed by allowing users to mark steps in a workflow as "KEPT" or "DELETED". Then whenever the workflow is run the files that are just created in passing to the next step and aren't required as output can be automatically marked as deleted and cleaned up as part of normal cleanup operations.
Would you consider implementing this, or if not - which modules would be best to look at if we want to do it ourselves?
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi Shaun, I'm finalizing an update for the post-job-actions that will be available shortly, and it will include documentation on the available actions and some info on how to make your own. By default, actions will be available for renaming, hiding, deleting, changing the datatype, and setting metadata on output datasets in workflows. -Dannon On Jun 18, 2010, at 8:53 AM, SHAUN WEBB wrote:
Hi all, where can I find more info on using post-job-actions in workflows, I ould like to implement this on my local install.
Thanks Shaun Webb
where can I find more info about
Quoting James Taylor <james@bx.psu.edu>:
Hi Dennis,
You may have seen that Dannon recently committed support for post-job-actions in worfklows that allow for some of the things you've been needing (e.g. change datatype). One of the actions is "hide dataset". One could definitely create a "delete dataset" action based on that, which marked the dataset as deleted so it can be picked up be the cleanup script later.
It should be an easy change if you want to take a stab at it. Actions are implemented in the module "galaxy.jobs.actions.post"
Thanks, James
On Jun 14, 2010, at 6:03 PM, Dennis Gascoigne wrote:
We are finding galaxy really useful but with the implications of users not cleaning up after themselves when running long workflow is hitting on our storage.
This could be easily fixed by allowing users to mark steps in a workflow as "KEPT" or "DELETED". Then whenever the workflow is run the files that are just created in passing to the next step and aren't required as output can be automatically marked as deleted and cleaned up as part of normal cleanup operations.
Would you consider implementing this, or if not - which modules would be best to look at if we want to do it ourselves?
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
participants (4)
-
Dannon Baker -
Dennis Gascoigne -
James Taylor -
SHAUN WEBB