Re: [galaxy-dev] Error when running cleanup_datasets.py
Hi Nate, I'm back to trying to figure this out again as I am running out of disk space. I added the bit of code you suggested below, but I don't think it helped, I'm not so familiar with python. I'm now running Galaxy Reports and it tells me that I have 7479 datasets that were deleted but have not yet been purged. I get the error below when I run cleanup_datasets.py with both the -5 and -4 flag Marking as deleted: LibraryDatasetDatasetAssociation id 6907 (for dataset id 51991) Deleting dataset id 51991 Deleting library dataset id 7225 Traceback (most recent call last): File "scripts/cleanup_datasets/cleanup_datasets.py", line 526, in <module> if __name__ == "__main__": main() File "scripts/cleanup_datasets/cleanup_datasets.py", line 124, in main purge_folders( app, cutoff_time, options.remove_from_disk, info_only = options.info_only, force_retry = options.force_retry ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 247, in purge_folders _purge_folder( folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 499, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 499, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "scripts/cleanup_datasets/cleanup_datasets.py", line 373, in _purge_dataset_instance log.debug( '%s %s has None dataset' % ( type( dataset_instance ), dataset_instance.id ) ) AttributeError: 'NoneType' object has no attribute 'id' Thanks in advance for your help, Liisa From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 02/10/2012 10:50 AM Subject: Re: [galaxy-dev] Error when running cleanup_datasets.py On Oct 2, 2012, at 10:44 AM, Liisa Koski wrote:
Hi Nate, That select statement does not return anything :(
Could you add a bit of debugging to the script to see what the id is of the dataset_instance that has a None dataset? if dataset_instance is None: log.debug( '%s %s has None dataset' % ( type( dataset_instance ), dataset_instance.id ) ) Thanks, --nate
Thanks, Liisa
From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 01/10/2012 01:01 PM Subject: Re: [galaxy-dev] Error when running cleanup_datasets.py
On Sep 24, 2012, at 10:41 AM, Liisa Koski wrote:
Hello, I am trying to run the cleanup scripts on my local installation but
./scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 10
-5 -r
Deleting library dataset id 7225 Traceback (most recent call last): File "./scripts/cleanup_datasets/cleanup_datasets.py", line 524, in
<module>
if __name__ == "__main__": main() File "./scripts/cleanup_datasets/cleanup_datasets.py", line 124, in
purge_folders( app, cutoff_time, options.remove_from_disk,
info_only = options.info_only, force_retry = options.force_retry )
File "./scripts/cleanup_datasets/cleanup_datasets.py", line 247, in
_purge_folder( folder, app, remove_from_disk, info_only =
info_only )
File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 495, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "./scripts/cleanup_datasets/cleanup_datasets.py", line 376, in _purge_dataset_instance ( dataset_instance.__class__.__name__, dataset_instance.id, dataset_instance.dataset.id ) AttributeError: 'NoneType' object has no attribute 'id'
Hi Liisa,
It'd appear that you have a library_dataset_dataset_association in your dataset that lacks an associated dataset. Does 'select id from
get stuck when trying to run the following: main purge_folders library_dataset_dataset_association where dataset_id is null' in your database return anything?
--nate
Any help would be much appreciated.
Thanks, Liisa ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On Nov 8, 2012, at 8:55 PM, Liisa Koski wrote:
Hi Nate, I'm back to trying to figure this out again as I am running out of disk space. I added the bit of code you suggested below, but I don't think it helped, I'm not so familiar with python.
I'm now running Galaxy Reports and it tells me that I have 7479 datasets that were deleted but have not yet been purged.
I get the error below when I run cleanup_datasets.py with both the -5 and -4 flag
Marking as deleted: LibraryDatasetDatasetAssociation id 6907 (for dataset id 51991) Deleting dataset id 51991 Deleting library dataset id 7225 Traceback (most recent call last): File "scripts/cleanup_datasets/cleanup_datasets.py", line 526, in <module> if __name__ == "__main__": main() File "scripts/cleanup_datasets/cleanup_datasets.py", line 124, in main purge_folders( app, cutoff_time, options.remove_from_disk, info_only = options.info_only, force_retry = options.force_retry ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 247, in purge_folders _purge_folder( folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 499, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 499, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "scripts/cleanup_datasets/cleanup_datasets.py", line 373, in _purge_dataset_instance log.debug( '%s %s has None dataset' % ( type( dataset_instance ), dataset_instance.id ) ) AttributeError: 'NoneType' object has no attribute 'id'
Ah, I was looking at the wrong level. You have a library_dataset without an associated library_dataset_dataset_association. The following SQL should return it and any others: select id from library_dataset where library_dataset_dataset_association_id is null Although the output indicates that the offending library_dataset id should be 7225. The easiest way to solve this problem is probably to orphan the broken library dataset, e.g.: update library_dataset set folder_id = null where id=7225; --nate
Thanks in advance for your help, Liisa
From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 02/10/2012 10:50 AM Subject: Re: [galaxy-dev] Error when running cleanup_datasets.py
On Oct 2, 2012, at 10:44 AM, Liisa Koski wrote:
Hi Nate, That select statement does not return anything :(
Could you add a bit of debugging to the script to see what the id is of the dataset_instance that has a None dataset?
if dataset_instance is None: log.debug( '%s %s has None dataset' % ( type( dataset_instance ), dataset_instance.id ) )
Thanks, --nate
Thanks, Liisa
From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 01/10/2012 01:01 PM Subject: Re: [galaxy-dev] Error when running cleanup_datasets.py
On Sep 24, 2012, at 10:41 AM, Liisa Koski wrote:
Hello, I am trying to run the cleanup scripts on my local installation but get stuck when trying to run the following:
./scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 10 -5 -r
Deleting library dataset id 7225 Traceback (most recent call last): File "./scripts/cleanup_datasets/cleanup_datasets.py", line 524, in <module> if __name__ == "__main__": main() File "./scripts/cleanup_datasets/cleanup_datasets.py", line 124, in main purge_folders( app, cutoff_time, options.remove_from_disk, info_only = options.info_only, force_retry = options.force_retry ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 247, in purge_folders _purge_folder( folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 495, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "./scripts/cleanup_datasets/cleanup_datasets.py", line 376, in _purge_dataset_instance ( dataset_instance.__class__.__name__, dataset_instance.id, dataset_instance.dataset.id ) AttributeError: 'NoneType' object has no attribute 'id'
Hi Liisa,
It'd appear that you have a library_dataset_dataset_association in your dataset that lacks an associated dataset. Does 'select id from library_dataset_dataset_association where dataset_id is null' in your database return anything?
--nate
Any help would be much appreciated.
Thanks, Liisa ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Nate, I'm back to trying to figure this out again as I am running out of disk space. I added the bit of code you suggested below, but I don't think it helped, I'm not so familiar with python.
I'm now running Galaxy Reports and it tells me that I have 7479 datasets
That did it! Thanks Nate! From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 12/11/2012 12:13 PM Subject: Re: [galaxy-dev] Error when running cleanup_datasets.py On Nov 8, 2012, at 8:55 PM, Liisa Koski wrote: that were deleted but have not yet been purged.
I get the error below when I run cleanup_datasets.py with both the -5
and -4 flag
Marking as deleted: LibraryDatasetDatasetAssociation id 6907 (for
dataset id 51991)
Deleting dataset id 51991 Deleting library dataset id 7225 Traceback (most recent call last): File "scripts/cleanup_datasets/cleanup_datasets.py", line 526, in <module> if __name__ == "__main__": main() File "scripts/cleanup_datasets/cleanup_datasets.py", line 124, in main
purge_folders( app, cutoff_time, options.remove_from_disk, info_only
File "scripts/cleanup_datasets/cleanup_datasets.py", line 247, in
= options.info_only, force_retry = options.force_retry ) purge_folders
_purge_folder( folder, app, remove_from_disk, info_only = info_only
)
File "scripts/cleanup_datasets/cleanup_datasets.py", line 499, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 499, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "scripts/cleanup_datasets/cleanup_datasets.py", line 373, in _purge_dataset_instance log.debug( '%s %s has None dataset' % ( type( dataset_instance ), dataset_instance.id ) ) AttributeError: 'NoneType' object has no attribute 'id'
Ah, I was looking at the wrong level. You have a library_dataset without an associated library_dataset_dataset_association. The following SQL should return it and any others: select id from library_dataset where library_dataset_dataset_association_id is null Although the output indicates that the offending library_dataset id should be 7225. The easiest way to solve this problem is probably to orphan the broken library dataset, e.g.: update library_dataset set folder_id = null where id=7225; --nate
Thanks in advance for your help, Liisa
From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 02/10/2012 10:50 AM Subject: Re: [galaxy-dev] Error when running cleanup_datasets.py
On Oct 2, 2012, at 10:44 AM, Liisa Koski wrote:
Hi Nate, That select statement does not return anything :(
Could you add a bit of debugging to the script to see what the id is of
the dataset_instance that has a None dataset?
if dataset_instance is None: log.debug( '%s %s has None dataset' % ( type( dataset_instance ),
dataset_instance.id ) )
Thanks, --nate
Thanks, Liisa
From: Nate Coraor <nate@bx.psu.edu> To: Liisa Koski <liisa.koski@basf.com> Cc: galaxy-dev@lists.bx.psu.edu Date: 01/10/2012 01:01 PM Subject: Re: [galaxy-dev] Error when running
On Sep 24, 2012, at 10:41 AM, Liisa Koski wrote:
Hello, I am trying to run the cleanup scripts on my local installation but
get stuck when trying to run the following:
./scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d
10 -5 -r
Deleting library dataset id 7225 Traceback (most recent call last): File "./scripts/cleanup_datasets/cleanup_datasets.py", line 524,
in <module>
if __name__ == "__main__": main() File "./scripts/cleanup_datasets/cleanup_datasets.py", line 124,
in main
purge_folders( app, cutoff_time, options.remove_from_disk,
info_only = options.info_only, force_retry = options.force_retry )
File "./scripts/cleanup_datasets/cleanup_datasets.py", line 247, in purge_folders _purge_folder( folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 495, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "./scripts/cleanup_datasets/cleanup_datasets.py", line 376, in _purge_dataset_instance ( dataset_instance.__class__.__name__, dataset_instance.id, dataset_instance.dataset.id ) AttributeError: 'NoneType' object has no attribute 'id'
Hi Liisa,
It'd appear that you have a library_dataset_dataset_association in your dataset that lacks an associated dataset. Does 'select id from
cleanup_datasets.py library_dataset_dataset_association where dataset_id is null' in your database return anything?
--nate
Any help would be much appreciated.
Thanks, Liisa ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (2)
-
Liisa Koski -
Nate Coraor