Manage local data admin tool failed with dbkey KeyError
Hi guys, I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error: URL: http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data [^<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data> ] File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/controllers/data_admin.py', line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix' I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files? Thanks Derrick
Hi guys, I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux? Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by "deferred job queue": URL: http://129.94.136.11:8080/galaxy_dev/data_admin/download_build File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/controllers/data_admin.py', line 199 in download_build jobid = trans.app.job_manager.deferred_job_queue.plugins['GenomeTransferPlugin'].create_job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue' Anyone can give some hints about what was going on? Regards, Derrick On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com> wrote:
Hi guys,
I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error:
URL: http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data [^<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data> ] File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/controllers/data_admin.py', line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix'
I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files?
Thanks Derrick
Derrick, The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted? The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions. --Dave B. On 9/9/12 21:20:44.000, Derrick Lin wrote:
Hi guys,
I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux?
Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by "deferred job queue":
URL: http://129.94.136.11:8080/galaxy_dev/data_admin/download_build File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/controllers/data_admin.py', line 199 in download_build jobid = trans.app.job_manager.deferred_job_queue.plugins['GenomeTransferPlugin'].create_job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue'
Anyone can give some hints about what was going on?
Regards, Derrick
On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com <mailto:klin938@gmail.com>> wrote:
Hi guys,
I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error:
URL: http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data [^ <http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>] File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/galaxy/galaxy_dist/dev/lib/galaxy/web/controllers/data_admin.py', line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix'
I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files?
Thanks Derrick
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Dave, The enable_beta_job_managers is set to true in my universe ini file. Unless it can be override in somewhere else? My galaxy hasn't been patched to the Sep release, but will give it a try. I actually also downloaded the complete pre-built indexes from galaxy's recent released rsync server. I am wondering if the Manage data tool recognizes those downloaded indexes and avoid downloading again from uscs server? Derrick On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Derrick,
The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted?
The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions.
--Dave B.
On 9/9/12 21:20:44.000, Derrick Lin wrote:
Hi guys,
I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux?
Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by "deferred job queue":
URL: http://129.94.136.11:8080/**galaxy_dev/data_admin/**download_build<http://129.94.136.11:8080/galaxy_dev/data_admin/download_build> File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** WebError-0.8a-py2.6.egg/**weberror/evalexception/**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/__init__.**py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/controllers/data_**admin.py', line 199 in download_build jobid = trans.app.job_manager.**deferred_job_queue.plugins['** GenomeTransferPlugin'].create_**job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue'
Anyone can give some hints about what was going on?
Regards, Derrick
On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com <mailto:klin938@gmail.com>> wrote:
Hi guys,
I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error:
URL: http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/**manage_data<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>[^ <http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/**manage_data<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>
]
File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** WebError-0.8a-py2.6.egg/**weberror/evalexception/**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/__init__.**py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/controllers/data_**admin.py', line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix'
I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files?
Thanks Derrick
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Dave, I upgraded the galaxy to Sep 7 release. The KeyError error has gone as expected. But the deferred_job_queue error persists. We are running galaxy in the cluster (SGE) here. Is it possible that Manage local data need to be configured in the cluster env? Regards, Derrick On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Derrick,
The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted?
The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions.
--Dave B.
On 9/9/12 21:20:44.000, Derrick Lin wrote:
Hi guys,
I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux?
Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by "deferred job queue":
URL: http://129.94.136.11:8080/**galaxy_dev/data_admin/**download_build<http://129.94.136.11:8080/galaxy_dev/data_admin/download_build> File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** WebError-0.8a-py2.6.egg/**weberror/evalexception/**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/__init__.**py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/controllers/data_**admin.py', line 199 in download_build jobid = trans.app.job_manager.**deferred_job_queue.plugins['** GenomeTransferPlugin'].create_**job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue'
Anyone can give some hints about what was going on?
Regards, Derrick
On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com <mailto:klin938@gmail.com>> wrote:
Hi guys,
I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error:
URL: http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/**manage_data<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>[^ <http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/**manage_data<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>
]
File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** WebError-0.8a-py2.6.egg/**weberror/evalexception/**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/__init__.**py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/controllers/data_**admin.py', line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix'
I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files?
Thanks Derrick
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
______________________________**_____________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Derrick, I have not actually tested the local data manager on CloudBioLinux, but that sounds like something worth looking into. As for the rsync server mentioned in another email, the data manager checks our rsync server first, then tries generating indexes if the rsync server doesn't have them. However, it will always produce indexes, as I decided to allow for the option to regenerate indexes, if the local ones had somehow become corrupted. --Dave B. On 9/11/12 02:03:45.000, Derrick Lin wrote:
Dave,
I upgraded the galaxy to Sep 7 release. The KeyError error has gone as expected.
But the deferred_job_queue error persists. We are running galaxy in the cluster (SGE) here. Is it possible that Manage local data need to be configured in the cluster env?
Regards, Derrick
On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier <dave@bx.psu.edu <mailto:dave@bx.psu.edu>> wrote:
Derrick,
The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted?
The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions.
--Dave B.
On 9/9/12 21:20:44.000, Derrick Lin wrote:
Hi guys,
I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux?
Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by "deferred job queue":
URL: http://129.94.136.11:8080/__galaxy_dev/data_admin/__download_build <http://129.94.136.11:8080/galaxy_dev/data_admin/download_build> File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__WebError-0.8a-py2.6.egg/__weberror/evalexception/__middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/__init__.__py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/controllers/data___admin.py', line 199 in download_build jobid = trans.app.job_manager.__deferred_job_queue.plugins['__GenomeTransferPlugin'].create___job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue'
Anyone can give some hints about what was going on?
Regards, Derrick
On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com <mailto:klin938@gmail.com> <mailto:klin938@gmail.com <mailto:klin938@gmail.com>>> wrote:
Hi guys,
I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error:
URL: http://pwbc.garvan.unsw.edu.__au/galaxy_dev/data_admin/__manage_data <http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data> [^
<http://pwbc.garvan.unsw.edu.__au/galaxy_dev/data_admin/__manage_data <http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>>]
File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__WebError-0.8a-py2.6.egg/__weberror/evalexception/__middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__debug/prints.py', line 98 in __call__ environ, self.app) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__recursive.py', line 80 in __call__ return self.application(environ, start_response) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/__init__.__py', line 184 in decorator return func( self, trans, *args, **kwargs ) File
'/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/controllers/data___admin.py', line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix'
I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files?
Thanks Derrick
_____________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
_____________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Dave, Thanks for the clarification. Our testing shows that the local data manager works in the vanilla Galaxy (simply hg clone, then run.sh). It failed on our clustered instance that it has been configured to use multiple web threads, one job manager and two job handlers. Local data manager is a great idea, it would be even better if it can fetch the indexes from the local file system. Cheers, Derrick On Tue, Sep 11, 2012 at 10:31 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Derrick,
I have not actually tested the local data manager on CloudBioLinux, but that sounds like something worth looking into.
As for the rsync server mentioned in another email, the data manager checks our rsync server first, then tries generating indexes if the rsync server doesn't have them. However, it will always produce indexes, as I decided to allow for the option to regenerate indexes, if the local ones had somehow become corrupted.
--Dave B.
On 9/11/12 02:03:45.000, Derrick Lin wrote:
Dave,
I upgraded the galaxy to Sep 7 release. The KeyError error has gone as expected.
But the deferred_job_queue error persists. We are running galaxy in the cluster (SGE) here. Is it possible that Manage local data need to be configured in the cluster env?
Regards, Derrick
On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier <dave@bx.psu.edu <mailto:dave@bx.psu.edu>> wrote:
Derrick,
The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted?
The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions.
--Dave B.
On 9/9/12 21:20:44.000, Derrick Lin wrote:
Hi guys,
I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux?
Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by "deferred job queue":
URL: http://129.94.136.11:8080/__**galaxy_dev/data_admin/__** download_build<http://129.94.136.11:8080/__galaxy_dev/data_admin/__download_build> <http://129.94.136.11:8080/**galaxy_dev/data_admin/** download_build<http://129.94.136.11:8080/galaxy_dev/data_admin/download_build>
File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** WebError-0.8a-py2.6.egg/__**weberror/evalexception/__**middleware.py',
line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**debug/prints.py',
line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**wsgilib.py',
line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**recursive.py',
line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**httpexceptions.py',
line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/framework/base.py',
line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/framework/__init__.**__py',
line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/controllers/data___**admin.py',
line 199 in download_build jobid = trans.app.job_manager.__**deferred_job_queue.plugins['__** GenomeTransferPlugin'].create_**__job(
trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue'
Anyone can give some hints about what was going on?
Regards, Derrick
On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com <mailto:klin938@gmail.com> <mailto:klin938@gmail.com <mailto:klin938@gmail.com>>> wrote:
Hi guys,
I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error:
URL: http://pwbc.garvan.unsw.edu.__**au/galaxy_dev/data_admin/__** manage_data <http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/** manage_data<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>
[^
<http://pwbc.garvan.unsw.edu._**_au/galaxy_dev/data_admin/__** manage_data <http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/** manage_data<http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>
]
File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** WebError-0.8a-py2.6.egg/__**weberror/evalexception/__**middleware.py',
line 364 in respond app_iter = self.application(environ, detect_start_response) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**debug/prints.py',
line 98 in __call__ environ, self.app) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**wsgilib.py',
line 539 in intercept_output app_iter = application(environ, replacement_start_response) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**recursive.py',
line 80 in __call__ return self.application(environ, start_response) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**httpexceptions.py',
line 632 in __call__ return self.application(environ, start_response) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/framework/base.py',
line 160 in __call__ body = method( trans, **kwargs ) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/framework/__init__.**__py',
line 184 in decorator return func( self, trans, *args, **kwargs ) File
'/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/controllers/data___**admin.py',
line 47 in manage_data genomes[ dbkey ][ table ] = dict( state='Generated', style=self.jobstyles[ 'done' ] ) KeyError: 'phix'
I tried to fix it by commenting out all the phix entries in all .loc files. Then it gave the same error but this time caused by "hg18". I think it must have something to do with builts.txt, manual_builts.txt and the loc files?
Thanks Derrick
______________________________**______________________________**_
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
______________________________**______________________________**_
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (2)
-
Dave Bouvier -
Derrick Lin