Hi guys;
We have taken advantage of your recent excellent additions to the workflow and have a lot of histories where intermediate datasets are hidden. It is becoming quite critical to us to catch these hidden history items in the cleanup scripts and delete them as they consume a lot of room and we cannot identify another way of removing them.
The obvious place to remove hidden datasets appears to be with an option in the cleanup procedures -In previous discussions you have mentioned that this is something you propose doing and it would at first seem relatively quick. What is your timing on this, or has it already been done. If it is a while away, is there anything I should consider in writing/amedning a query to roll our own?
Cheers Dennis
Dennis,
I'm glad you're finding the actions useful. Regarding hidden datasets, we're still working out how to best handle displaying and them in the UI, but I would expect the ability to unhide and/or delete them from a particular history next week.
It does seem that given the way you're using the hide action it might be more ideal to write and utilize a deferred delete action. That is, outputs are marked for deletion but that only happens after the full workflow is complete and successful. This would be different from the current delete action that is only really useful for throwing away extra outputs from workflow steps that are not used subsequently.
Lastly, if you did want to look into cleaning it up yourself manually, you're looking for HistoryDatasetAssociation entries where visible is False. You could then set deleted to True and then the cleanup scripts would take care of the rest as usual.
-Dannon
On Jul 7, 2010, at 5:02 PM, Dennis Gascoigne wrote:
Hi guys;
We have taken advantage of your recent excellent additions to the workflow and have a lot of histories where intermediate datasets are hidden. It is becoming quite critical to us to catch these hidden history items in the cleanup scripts and delete them as they consume a lot of room and we cannot identify another way of removing them.
The obvious place to remove hidden datasets appears to be with an option in the cleanup procedures -In previous discussions you have mentioned that this is something you propose doing and it would at first seem relatively quick. What is your timing on this, or has it already been done. If it is a while away, is there anything I should consider in writing/amedning a query to roll our own?
Cheers Dennis _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Oh, so if I wrote a small script that just ran an update query with pseudo code like; update HistoryDatasetAssociation set deleted=true where visible=false
The cleanup scripts will catch them?
Regarding deferred deletions and changes to visibility etc. These indeed seem like a good solution and we are looking forward to them.
Cheers Dennis
On Sat, Jul 10, 2010 at 5:01 AM, Dannon Baker dannonbaker@me.com wrote:
Dennis,
I'm glad you're finding the actions useful. Regarding hidden datasets, we're still working out how to best handle displaying and them in the UI, but I would expect the ability to unhide and/or delete them from a particular history next week.
It does seem that given the way you're using the hide action it might be more ideal to write and utilize a deferred delete action. That is, outputs are marked for deletion but that only happens after the full workflow is complete and successful. This would be different from the current delete action that is only really useful for throwing away extra outputs from workflow steps that are not used subsequently.
Lastly, if you did want to look into cleaning it up yourself manually, you're looking for HistoryDatasetAssociation entries where visible is False. You could then set deleted to True and then the cleanup scripts would take care of the rest as usual.
-Dannon
On Jul 7, 2010, at 5:02 PM, Dennis Gascoigne wrote:
Hi guys;
We have taken advantage of your recent excellent additions to the
workflow and have a lot of histories where intermediate datasets are hidden. It is becoming quite critical to us to catch these hidden history items in the cleanup scripts and delete them as they consume a lot of room and we cannot identify another way of removing them.
The obvious place to remove hidden datasets appears to be with an option
in the cleanup procedures -In previous discussions you have mentioned that this is something you propose doing and it would at first seem relatively quick. What is your timing on this, or has it already been done. If it is a while away, is there anything I should consider in writing/amedning a query to roll our own?
Cheers Dennis _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Yes.
Also, I'm not sure what repository you track, but the I pushed the code for showing hidden items and unhiding them using the normal history panel to galaxy-central last night if you're interested in just using that.
-Dannon
On Jul 11, 2010, at 3:16 AM, Dennis Gascoigne wrote:
Oh, so if I wrote a small script that just ran an update query with pseudo code like; update HistoryDatasetAssociation set deleted=true where visible=false
The cleanup scripts will catch them?
Regarding deferred deletions and changes to visibility etc. These indeed seem like a good solution and we are looking forward to them.
Cheers Dennis
On Sat, Jul 10, 2010 at 5:01 AM, Dannon Baker dannonbaker@me.com wrote: Dennis,
I'm glad you're finding the actions useful. Regarding hidden datasets, we're still working out how to best handle displaying and them in the UI, but I would expect the ability to unhide and/or delete them from a particular history next week.
It does seem that given the way you're using the hide action it might be more ideal to write and utilize a deferred delete action. That is, outputs are marked for deletion but that only happens after the full workflow is complete and successful. This would be different from the current delete action that is only really useful for throwing away extra outputs from workflow steps that are not used subsequently.
Lastly, if you did want to look into cleaning it up yourself manually, you're looking for HistoryDatasetAssociation entries where visible is False. You could then set deleted to True and then the cleanup scripts would take care of the rest as usual.
-Dannon
On Jul 7, 2010, at 5:02 PM, Dennis Gascoigne wrote:
Hi guys;
We have taken advantage of your recent excellent additions to the workflow and have a lot of histories where intermediate datasets are hidden. It is becoming quite critical to us to catch these hidden history items in the cleanup scripts and delete them as they consume a lot of room and we cannot identify another way of removing them.
The obvious place to remove hidden datasets appears to be with an option in the cleanup procedures -In previous discussions you have mentioned that this is something you propose doing and it would at first seem relatively quick. What is your timing on this, or has it already been done. If it is a while away, is there anything I should consider in writing/amedning a query to roll our own?
Cheers Dennis _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
-- Dennis Gascoigne 0407 639 995 dennis.gascoigne@gmail.com
galaxy-dev@lists.galaxyproject.org
-
Dannon Baker -
Dennis Gascoigne