Our local instance currently uses the traditional directories under
`database/` for datasets, job working directories, and temporary
files. Ultimately we wish to transition to using our Swift object
store for storage. We've been doing some experimentation with Galaxy's
Swift backend and have run into a few issues.

The first major issue we came across was Swift's 5 GB segment size
limit, since the segmentation/multipart upload code is bypassed for
instances of SwiftObjectStore [1]. SwiftStack support provided a patch
enabling multipart uploads for Swift (PR #648) which has been working
well for us so far. (Thanks, Charles!)

The next issue is that the path attribute of the cache tag in
object_store_conf.xml appears to be ignored. The value does get stored
to self.cache_path in _parse_config_xml, but elsewhere in the file
self.staging_path is used instead.

Finally, adding extra_dir tags to the Swift object store config
doesn't appear to do anything. Here's my object_store_conf.xml:

<?xml version="1.0"?>
<object_store type="hierarchical">
    <backends>
       <object_store type="swift" id="primary" order="0">
            <auth access_key="..." secret_key="..."/>
            <bucket name="galaxy_store"/>
            <connection host="tin.fhcrc.org" port="443"/>
            <cache path="database/object_store_cache" size="1000"/>
            <extra_dir type="temp" path="database/tmp"/>
            <extra_dir type="job_work" path="database/job_working_directory"/>
        </object_store>
        <object_store type="disk" id="secondary" order="1">
            <files_dir path="database/files"/>
        </object_store>
    </backends>
</object_store>

The goal with the hierarchical setup above is for new datasets to be
created in the primary (Swift) object store, caching to
`database/object_store_cache`, while the job and temporary directories
remain at `database/job_working_directory` and `database/tmp`,
respectively. Existing (pre-Swift) datasets remain in `database/files`
and are handled by the secondary disk store.

What actually happens (after renaming self.cache_path to
self.staging_path in _parse_config_xml to get the cache path working)
is this:

galaxy.jobs DEBUG 2015-02-06 16:07:26,615 (1) Working directory for
job is: /home/bclaywel/workspace/galaxy-central/database/object_store_cache/000/1

That is, the job working directory is created directly under the cache
path's hash directories. I assume temp files would probably end up
there also.

We're quite excited to get Galaxy and Swift working well together, and
I'm more than happy to help debug and test!


Cheers,

Brian


[1] https://bitbucket.org/galaxy/galaxy-central/src/54ed3adb6575addba47d627944ebd72f7547082d/lib/galaxy/objectstore/s3.py?at=default#cl-331

--
Brian Claywell | programmer/analyst
Matsen Group   | http://matsen.fredhutch.org
Fred Hutchinson Cancer Research Center
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/