[galaxy-dev] Want faster dataset import

I want to frequently import many tens of thousands of datasets. The files are on the same sever as Galaxy. But the upload based mechanism is really really slow. It takes hours to load this many files, yet the data is not moving at all!

What is the best strategy to go about making a faster bulk import? I can imagine a tight loop that is

My datatypes are limited.

foreach folder new LibraryFolder

foreach file in each directory new LibraryDataset new LibraryDatasetDatasetAssociation

flush once at the end.

Thoughts?