In my case I ended modifying the API workflow_execute.py script and used it to create symlinks I could use to run processes outside of galaxy. In case the new processing machine doesn't have access to the directory tree and use rsync with '--copy-links' to download the datasets.

This is the important part of my script: r = get(api_key, galaxy_url) # iterate over each data set in the library or history for item in r: # check if the name matchs the regular expression if item['type'] == 'file' and match(regex, item['name']) is not None: # if so go and get the details for the data set item_details = get(api_key, galaxy_url + '/' + item['id']) # link to the data set real file name os.symlink(item_details['file_name'], item_details['name']) # if this is a bam file, also sym link to the bam index file if item_details['data_type'] == 'bam': os.symlink(item_details['metadata_bam_index'], item_details['name'] + '.bai')

A few advantages of this approach over simply "scpying". First you get the same name you are using in galaxy and not a generic dataset_000.dat. Also, you can copy multiple files matching a provided regular expression in their name, with scp you would have to copy one at a time.

If you are interested I can send you the whole script, which I'm sure you will find several ways to break it, but in general it does its job.

Regards, Carlos

On Wed, Jun 6, 2012 at 11:20 AM, Hans-Rudolf Hotz hrh@fmi.ch wrote:

Hi Jean-Francois

My random guess: this is a web browser issue, struggling to download big files? - although, 3000 lines is not big

If required, we use scp on the command line to get a copy of the dataset

Regards, Hans

On 06/06/2012 03:53 PM, Jean-Francois Payotte wrote:

...

Hi folks,

Some of our local Galaxy instance users seem to be experiencing some strange behaviour lately. I searched the mailing-list archive but I didn't found anything related, so I'd be interested to know if somebody already had the same issue.

The problem is that sometimes, when people are trying to download their datasets from their history, although the file seems to download successfully, it appears that the downloaded file is incomplete (for example a 3000 lines text file will show only maybe 2000 lines at the first download, 1600 lines at the second download, and so on... and eventually, the file will download completely.

This issue happened with more than one user and with different tools.

Does anybody ever had this kind of issue? Or does somebody would have an idea of where to look to solve this problem?

Best regards, Jean-François

---

Please keep all replies on the list by using "reply all" in your mail client.  To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

---

Please keep all replies on the list by using "reply all" in your mail client.  To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

Hi Jennifer,

Thank you for your answer. Although using "curl" command might be a good solution to overcome the incomplete downloads issue, this solution might not be enough user-oriented (as I don't think our users are familiar with using the command line). Anyway, we have developed a tool which allows the user to select one file from his history and to copy it to another user-given location on our server. So using this tool allows us to download the files correctly.

Also, this issue happened while trying to download a file of size about 250Mb.. Is this considered a large file?

And finally an other question comes to mind: What about people downloading their result files not knowing that the download was incomplete? If the user doesn't check if the number of lines in the downloaded file correspond to what is shown in Galaxy history, he won't be able to know that the download fails.

And this is why I was wondering if there was some way to prevent incomplete downloads (using the download button). Or if there was some way to be warned automatically when the download fails.

Once again, thanks for your help and for that incredible tool that is Galaxy.

Best regards, Jean-François

From: Jennifer Jackson jen@bx.psu.edu To: Jean-Francois Payotte jean-francois.payotte@dnalandmarks.ca Cc: galaxy-dev@lists.bx.psu.edu Date: 07/06/2012 10:49 AM Subject: Re: [galaxy-dev] Incomplete datasets when dowloaded from history

Hello Jean-François,

For larger files, or if complete downloads seem to be a problem, using curl at a command or Terminal prompt is a great option. The syntax is:

% curl -O '<copied-link-from-dataset-disc-icon>'

http://user.list.galaxyproject.org/Download-multiple-files-from-history-td43...

(wget is not currently supported, so please disregard those instructions in the post)

I used the 'mailing list' version of the Galaxy custom google searches to find this thread. http://galaxy.psu.edu/search/mailinglists/

Links to all of the Galaxy custom searches are in the wiki in the side bar and other places, but also at the top of the Support page for quick access: http://wiki.g2.bx.psu.edu/Support

Hopefully this helps,

Jen Galaxy team

On 6/6/12 6:53 AM, Jean-Francois Payotte wrote: Hi folks,

Some of our local Galaxy instance users seem to be experiencing some strange behaviour lately. I searched the mailing-list archive but I didn't found anything related, so I'd be interested to know if somebody already had the same issue.

The problem is that sometimes, when people are trying to download their datasets from their history, although the file seems to download successfully, it appears that the downloaded file is incomplete (for example a 3000 lines text file will show only maybe 2000 lines at the first download, 1600 lines at the second download, and so on... and eventually, the file will download completely.

This issue happened with more than one user and with different tools.

Does anybody ever had this kind of issue? Or does somebody would have an idea of where to look to solve this problem?

Best regards, Jean-François

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/