On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker <dannonbaker@me.com> wrote:
Main still runs these jobs in the standard non-split fashion, and as a resource that is occasionally saturated (and thus doesn't necessarily have extra resources to parallelize to) will probably continue doing so as long as there's significant overhead involved in splitting the files. Fancy scheduling could minimize the issue, but as it is during heavy load you would actually have lower total throughput due to the splitting overhead.
Because the splitting (currently) happens on the main server?
Regarding the merging of the out, I see there is a default merge method in lib/galaxy/datatypes/data.py which just concatenates the files. I am surprised at that - it seems like a very bad idea in general - consider many binary files, or XML. Why not put this as the default for text and subclasses thereof?
I can't think of a better reasonable default behavior for "Data", though you're obviously right that each datatype subclass will need to define particular behaviors for merging files.
The default should raise an error (and better yet, refuse to do the split in the first place). Zen of Python: In the face of ambiguity, refuse the temptation to guess. Peter