Hi Dave,

Thanks for the clarification. Our testing shows that the local data manager works in the vanilla Galaxy (simply hg clone, then run.sh). It failed on our clustered instance that it has been configured to use multiple web threads, one job manager and two job handlers.

Local data manager is a great idea, it would be even better if it can fetch the indexes from the local file system.

Cheers,
Derrick

On Tue, Sep 11, 2012 at 10:31 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Derrick,

I have not actually tested the local data manager on CloudBioLinux, but that sounds like something worth looking into.

As for the rsync server mentioned in another email, the data manager checks our rsync server first, then tries generating indexes if the rsync server doesn't have them. However, it will always produce indexes, as I decided to allow for the option to regenerate indexes, if the local ones had somehow become corrupted.

   --Dave B.


On 9/11/12 02:03:45.000, Derrick Lin wrote:
Dave,

I upgraded the galaxy to Sep 7 release. The KeyError error has gone as
expected.

But the deferred_job_queue error persists. We are running galaxy in the
cluster (SGE) here. Is it possible that Manage local data need to be
configured in the cluster env?

Regards,
Derrick

On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier <dave@bx.psu.edu
<mailto:dave@bx.psu.edu>> wrote:

    Derrick,

    The deferred_job_queue instance in the job manager object is only
    instantiated when the enable_beta_job_managers variable is set to
    True. Is it possible that your previous setting of that variable was
    somehow reverted?

    The KeyError error you mentioned in your previous should be fixed in
    the latest -dist and -central revisions.

        --Dave B.


    On 9/9/12 21:20:44.000, Derrick Lin wrote:

        Hi guys,

        I found that the problematic loc files were created by
        CloudBioLinux and
        point to the reference indexes downloaded by CloudBioLinux script. I
        suspect that Manage Local Data cannot recognize the file structure
        created by CloudBioLinux?

        Anyhow, I removed all the loc files from CloudBioLinux, then Manage
        Local Data can run now. Soon after I tried to download the
        bowtie hg18
        from ucsc, I was given another error caused by "deferred job queue":

        URL:
        http://129.94.136.11:8080/__galaxy_dev/data_admin/__download_build
        <http://129.94.136.11:8080/galaxy_dev/data_admin/download_build>
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__WebError-0.8a-py2.6.egg/__weberror/evalexception/__middleware.py',

        line 364 in respond
            app_iter = self.application(environ, detect_start_response)
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__debug/prints.py',

        line 98 in __call__
            environ, self.app)
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__wsgilib.py',

        line 539 in intercept_output
            app_iter = application(environ, replacement_start_response)
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__recursive.py',

        line 80 in __call__
            return self.application(environ, start_response)
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__httpexceptions.py',

        line 632 in __call__
            return self.application(environ, start_response)
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/base.py',

        line 160 in __call__
            body = method( trans, **kwargs )
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/__init__.__py',

        line 184 in decorator
            return func( self, trans, *args, **kwargs )
        File
        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/controllers/data___admin.py',

        line 199 in download_build
            jobid =
        trans.app.job_manager.__deferred_job_queue.plugins['__GenomeTransferPlugin'].create___job(

        trans, url, dbkey, longname, indexers )
        AttributeError: 'JobManager' object has no attribute
        'deferred_job_queue'

        Anyone can give some hints about what was going on?

        Regards,
        Derrick

        On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin <klin938@gmail.com
        <mailto:klin938@gmail.com>
        <mailto:klin938@gmail.com <mailto:klin938@gmail.com>>> wrote:

             Hi guys,

             I enabled the tool with enable_beta_job_managers = True,
        server runs
             fine. But after I clicked on Manage local data, it gave the
             following error:

             URL:
        http://pwbc.garvan.unsw.edu.__au/galaxy_dev/data_admin/__manage_data
        <http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>
        [^

        <http://pwbc.garvan.unsw.edu.__au/galaxy_dev/data_admin/__manage_data
        <http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data>>]

             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__WebError-0.8a-py2.6.egg/__weberror/evalexception/__middleware.py',

             line 364 in respond
                app_iter = self.application(environ, detect_start_response)
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__debug/prints.py',

             line 98 in __call__
                environ, self.app)
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__wsgilib.py',

             line 539 in intercept_output
                app_iter = application(environ, replacement_start_response)
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__recursive.py',

             line 80 in __call__
                return self.application(environ, start_response)
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__httpexceptions.py',

             line 632 in __call__
                return self.application(environ, start_response)
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/base.py',

             line 160 in __call__
                body = method( trans, **kwargs )
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/__init__.__py',

             line 184 in decorator
                return func( self, trans, *args, **kwargs )
             File

        '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/controllers/data___admin.py',

             line 47 in manage_data
                genomes[ dbkey ][ table ] = dict( state='Generated',
             style=self.jobstyles[ 'done' ] )
             KeyError: 'phix'

             I tried to fix it by commenting out all the phix entries in
        all .loc
             files. Then it gave the same error but this time caused by
        "hg18". I
             think it must have something to do with builts.txt,
             manual_builts.txt and the loc files?

             Thanks
             Derrick




        _____________________________________________________________

        Please keep all replies on the list by using "reply all"
        in your mail client.  To manage your subscriptions to this
        and other Galaxy lists, please use the interface at:

        http://lists.bx.psu.edu/

    _____________________________________________________________

    Please keep all replies on the list by using "reply all"
    in your mail client.  To manage your subscriptions to this
    and other Galaxy lists, please use the interface at:

    http://lists.bx.psu.edu/