Hi Megan,

If some columns of data were empty, then there should be two adjacent tabs in your data, and Galaxy would leave a blank, empty value in that column. You will need to check how your application actually output the data, possibly pad the empty values, and consider to exporting directly as plain, unformulated text if at all possible.

You could also try padding these empty values with a null value. Commonly used values are a dot "." and a zero "0". Different tools will expect different null values, see the tool forms for expected input formats. Completely blank values will cause problems with many tools.

Probably also would want to avoid " Convert spaces to tabs:", unless you know this will produce the correct output.

Best,

Jen
Galaxy team

On 6/4/12 2:15 PM, Estorninho, Megan wrote:

Thank you Jennifer, this is helpful to know.  I am experiencing fewer delays of recent.

One further question, when uploading a .txt file and converting it to tabular format I sometimes find I my file format changes, I think this is due to blanks in some lines of the tab-delimited file, so that it shifts things out of place and items are no longer in the same columns.  Is there a way to avoid this shifting?

Thanks,
Megan

________________________________________
From: Jennifer Jackson [jen@bx.psu.edu]
Sent: 04 June 2012 22:02
To: Estorninho, Megan
Cc: galaxy-user
Subject: Re: [galaxy-user] (no subject)

Hi Megan,

I ran a few tests and found that changing the file suffix to .txt when
using the "autodetect" upload type function speed up the loading process
considerably. As the final result is an identical Galaxy dataset to what
is produced with using the existing suffix, this is something I would
recommend that you try next time.

For my test, I took one of your files and change the suffix directly, no
other changes were made to the content, as it was already a
tab-delimited text file. I didn't continue with the testing to specify
the datatype at upload (tabular would be the correct choice), but this
is a change that may also speed up import slightly, although the .txt
suffix change was dramatic alone and the upload was quick  (I ran a
side-by-side comparison of an original and .txt-suffix modified file).

The general reason behind this is that Galaxy will interpret data to
detect and confirm datatypes during upload to create associated metadata
needed for tool use. Detection is a convenience option that comes at a
cost (compute resource and time). If you can provide this information
instead, the detection portion of the process can be avoided,
confirmation and metadata creation can be started directly, and the
result is a quicker upload.

Hopefully this helps for next time,

Jen
Galaxy team

On 5/22/12 8:49 AM, Estorninho, Megan wrote:

Yes I am still experiencing problems. My files are only around 80-120MB and are taking hours to load if at all.
Thanks for your help,
Megan

Sent from my iPhone

On 22 May 2012, at 14:38, "Jennifer Jackson"<jen@bx.psu.edu>  wrote:

Hello Megan,

Are you still experiencing problems now? Galaxy may have been busy
immediately following the resolution of the cluster problem, although
your problem does appear to be unrelated.

It sounds like you are uploading file through a browser. A better choice
would be to use FTP. This is required for datasets approaching or
exceeding 2G in size.

Files that are<  2G, really any file over ~ 500MB, can also benefit from
FTP upload. An FTP client tracks the progress of an upload and can
resume an interrupted transfer.  http://wiki.g2.bx.psu.edu/FTPUpload

Hopefully this helps,

Jen
Galaxy team

On 5/21/12 10:21 AM, Estorninho, Megan wrote:

I have been unable to upload data files into Galaxy Main since Friday 18th May 2012. Today is my fourth day of attempting uploads. Refreshing and leaving the files to upload overnight does not work.
Although Jennifer has stated the bug has been fixed at 5.30pm today I am still unable to upload data files. I thought I may be exceeding maximum file capacity but I am well below at only 1.8Gb.
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

   http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

   http://lists.bx.psu.edu/

--
Jennifer Jackson
http://galaxyproject.org

--
Jennifer Jackson
http://galaxyproject.org

-- 
Jennifer Jackson
http://galaxyproject.org