Hi Jorrit,

Thank you for your explanation. Would you be able to give us an example of what do you mean by fractions and when the task_%d are being used to pick files. Just want to make sure we have  good understanding of the problem that you solved.

Also, I vaguely remember seeing ‘data parallelism” mentioned somewhere with relation to the m: data sets.  Do you currently support in any way automatic distribution of processing of such datasets to parallel environments (e.g. array jobs in sge or such?)

Cheers,

- Piotr

From: Jorrit Boekel [mailto:jorrit.boekel@scilifelab.se]
Sent: Wednesday, July 31, 2013 8:18 PM
To: Khassapov, Alex (CSIRO IM&T, Clayton)
Cc: p.j.a.cock@googlemail.com; jmchilton@gmail.com; galaxy-dev@lists.bx.psu.edu; Szul, Piotr (ICT Centre, Marsfield); Burdett, Neil (ICT Centre, Herston - RBWH)
Subject: Re: Appending _task_%d suffix to multi files

Hi Alex,

In our lab, files are often fractions of an experiments, but they are named by their creators in whatever way they like. I put that code in to standardize fraction naming, in case a tool needs input from two files that originate from the same fraction (but have been treated in different ways). In those cases, in my fork, Galaxy always picks the files with the same task_%d numbers.

I can't help you very much right now, as I'm currently away from work until October, but I hope this explains why its in there.

cheers,
jorrit

On 07/31/2013 04:15 AM, Alex.Khassapov@csiro.au wrote:

Hi guys,

We've been using Galaxy for a year now, we created our own Galaxy fork where we were making changes to adapt Galaxy to our requirements.  As we need "multiple file dataset" - we were using Johns' fork for that initially.

Now we are trying to use "The most updated version of the multiple file dataset stuff" https://bitbucket.org/msiappdev/galaxy-extras/ directly as we don't want to maintain our own version.

One of the problems we have - when we upload multiple files - their file names are changed (_task_%d suffix is added to their names).

On our branch we simply removed the code which does it, but now we wonder if it is possible to avoid this renaming somehow? I.e. make it configurable?

Is it really necessary to change the file names?

-Alex

-----Original Message-----
From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Jorrit Boekel
Sent: Thursday, 25 October 2012 8:35 PM
To: Peter Cock
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] the multi job splitter

I keep the files matched by keeping a _task_%d suffix to their names. So each task is matched with its correct counterpart with the same number.

cheers,

jorrit