Thanks John, We do have pretty old code, so I applied your patches. One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml? -Alex -----Original Message----- From: jmchilton@gmail.com [mailto:jmchilton@gmail.com] On Behalf Of John Chilton Sent: Friday, 31 May 2013 5:50 AM To: Khassapov, Alex (CSIRO IM&T, Clayton) Cc: Burdett, Neil (ICT Centre, Herston - RBWH); <galaxy-dev@bx.psu.edu> Subject: Re: Multi File upload api Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed): https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-librar...). Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point): 1) You will need to replace: data['upload_option'] = 'upload_paths' with data['upload_option'] = 'upload_paths_multifile' 2) You will need to specify a name (e.g. add something like: data['NAME'] = os.path.basename(fullpath)) 3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs). The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads). Hope this helps, -John On Thu, May 30, 2013 at 1:40 AM, <Alex.Khassapov@csiro.au> wrote:

Hi John,

Can you please have a look at Neil's question.

Thank you,

-Alex

From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Thursday, 30 May 2013 4:30 PM To: Khassapov, Alex (CSIRO IM&T, Clayton) Subject: Multi File upload api

Hi Alex,

The file "galaxy-dist/scripts/api/example_watch_folder.py" allows us to watch a folder then upload files that arrive in a specified input folder into the database and history.

Can you ask your friend (who implemented the multi file upload tool), what changes are necessary to this file so we can upload multiple files as we do from the GUI. I assume he would know quite quickly what to do and hopefully quite simple....

Thanks

Neil

This changeset is where I implemented collecting and assigning names for multiple file datasets: https://bitbucket.org/msiappdev/galaxy-extras/commits/bfb07a8c8b7b4c404339cd... -John On Fri, May 31, 2013 at 1:21 AM, <Alex.Khassapov@csiro.au> wrote:

Thanks John,

We do have pretty old code, so I applied your patches.

One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml?

-Alex

-----Original Message----- From: jmchilton@gmail.com [mailto:jmchilton@gmail.com] On Behalf Of John Chilton Sent: Friday, 31 May 2013 5:50 AM To: Khassapov, Alex (CSIRO IM&T, Clayton) Cc: Burdett, Neil (ICT Centre, Herston - RBWH); <galaxy-dev@bx.psu.edu> Subject: Re: Multi File upload api

Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed):

https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable

If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-librar...).

Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point):

1) You will need to replace:

data['upload_option'] = 'upload_paths'

with

data['upload_option'] = 'upload_paths_multifile'

2) You will need to specify a name (e.g. add something like: data['NAME'] = os.path.basename(fullpath))

3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs).

The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads).

Hope this helps, -John

On Thu, May 30, 2013 at 1:40 AM, <Alex.Khassapov@csiro.au> wrote:

...

Hi John,

Can you please have a look at Neil's question.

Thank you,

-Alex

From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Thursday, 30 May 2013 4:30 PM To: Khassapov, Alex (CSIRO IM&T, Clayton) Subject: Multi File upload api

Hi Alex,

The file "galaxy-dist/scripts/api/example_watch_folder.py" allows us to watch a folder then upload files that arrive in a specified input folder into the database and history.

Can you ask your friend (who implemented the multi file upload tool), what changes are necessary to this file so we can upload multiple files as we do from the GUI. I assume he would know quite quickly what to do and hopefully quite simple....

Thanks

Neil