10:03 p.m.
We are finding galaxy really useful but with the implications of users not
cleaning up after themselves when running long workflow is hitting on our
storage.
This could be easily fixed by allowing users to mark steps in a workflow as
"KEPT" or "DELETED". Then whenever the workflow is run the files that
are
just created in passing to the next step and aren't required as output can
be automatically marked as deleted and cleaned up as part of normal cleanup
operations.
Would you consider implementing this, or if not - which modules would be
best to look at if we want to do it ourselves?
Cheers
Dennis
10:07 p.m.
Hi Dennis,
You may have seen that Dannon recently committed support for post-job-
actions in worfklows that allow for some of the things you've been
needing (e.g. change datatype). One of the actions is "hide dataset".
One could definitely create a "delete dataset" action based on that,
which marked the dataset as deleted so it can be picked up be the
cleanup script later.
It should be an easy change if you want to take a stab at it. Actions
are implemented in the module "galaxy.jobs.actions.post"
Thanks,
James
On Jun 14, 2010, at 6:03 PM, Dennis Gascoigne wrote:
We are finding galaxy really useful but with the implications of
users not cleaning up after themselves when running long workflow is
hitting on our storage.
This could be easily fixed by allowing users to mark steps in a
workflow as "KEPT" or "DELETED". Then whenever the workflow is run
the files that are just created in passing to the next step and
aren't required as output can be automatically marked as deleted and
cleaned up as part of normal cleanup operations.
Would you consider implementing this, or if not - which modules
would be best to look at if we want to do it ourselves?
12:53 p.m.
Hi all,
where can I find more info on using post-job-actions in workflows, I
ould like to implement this on my local install.
Thanks
Shaun Webb
where can I find more info about
Quoting James Taylor <james(a)bx.psu.edu>:
Hi Dennis,
You may have seen that Dannon recently committed support for
post-job-actions in worfklows that allow for some of the things you've
been needing (e.g. change datatype). One of the actions is "hide
dataset". One could definitely create a "delete dataset" action based
on that, which marked the dataset as deleted so it can be picked up be
the cleanup script later.
It should be an easy change if you want to take a stab at it. Actions
are implemented in the module "galaxy.jobs.actions.post"
Thanks,
James
On Jun 14, 2010, at 6:03 PM, Dennis Gascoigne wrote:
> We are finding galaxy really useful but with the implications of
> users not cleaning up after themselves when running long workflow
> is hitting on our storage.
>
> This could be easily fixed by allowing users to mark steps in a
> workflow as "KEPT" or "DELETED". Then whenever the workflow is
run
> the files that are just created in passing to the next step and
> aren't required as output can be automatically marked as deleted
> and cleaned up as part of normal cleanup operations.
>
> Would you consider implementing this, or if not - which modules
> would be best to look at if we want to do it ourselves?
_______________________________________________
galaxy-dev mailing list
galaxy-dev(a)lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-dev
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
1:22 p.m.
Hi Shaun,
I'm finalizing an update for the post-job-actions that will be available shortly, and
it will include documentation on the available actions and some info on how to make your
own. By default, actions will be available for renaming, hiding, deleting, changing the
datatype, and setting metadata on output datasets in workflows.
-Dannon
On Jun 18, 2010, at 8:53 AM, SHAUN WEBB wrote:
Hi all,
where can I find more info on using post-job-actions in workflows, I ould like to
implement this on my local install.
Thanks
Shaun Webb
where can I find more info about
Quoting James Taylor <james(a)bx.psu.edu>:
> Hi Dennis,
>
> You may have seen that Dannon recently committed support for
> post-job-actions in worfklows that allow for some of the things you've
> been needing (e.g. change datatype). One of the actions is "hide
> dataset". One could definitely create a "delete dataset" action based
> on that, which marked the dataset as deleted so it can be picked up be
> the cleanup script later.
>
> It should be an easy change if you want to take a stab at it. Actions
> are implemented in the module "galaxy.jobs.actions.post"
>
> Thanks,
> James
>
> On Jun 14, 2010, at 6:03 PM, Dennis Gascoigne wrote:
>
>> We are finding galaxy really useful but with the implications of users not
cleaning up after themselves when running long workflow is hitting on our storage.
>>
>> This could be easily fixed by allowing users to mark steps in a workflow as
"KEPT" or "DELETED". Then whenever the workflow is run the files that
are just created in passing to the next step and aren't required as output can be
automatically marked as deleted and cleaned up as part of normal cleanup operations.
>>
>> Would you consider implementing this, or if not - which modules would be best to
look at if we want to do it ourselves?
>
> _______________________________________________
> galaxy-dev mailing list
> galaxy-dev(a)lists.bx.psu.edu
> http://lists.bx.psu.edu/listinfo/galaxy-dev
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
_______________________________________________
galaxy-dev mailing list
galaxy-dev(a)lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-dev
4236
days inactive
4240
days old
galaxy-dev@lists.galaxyproject.org
3 comments
4 participants
participants (4)
-
Dannon Baker -
Dennis Gascoigne -
James Taylor -
SHAUN WEBB