Hi Rich, This is a good question! Maybe people have been asking about this. To download data with a unix line command method, please try wget or curl, for example: unix% wget 'url_for_the_dataset' or unix% wget 'url_for_the_history' To capture the url for a dataset, right click on the disk icon for a dataset and select "copy link location". To capture the url for an entire history, select "Options -> Export to File". The middle panel will display a link. A downloaded history can be loaded into a local Galaxy instance where the datasets can be managed (copy/rename) or the histories archived. Hopefully this helps you and others that are managing larger datasets & histories, Best, Jen Galaxy team On 11/7/11 7:50 AM, Richard Mark White wrote:

Hi, I am using the galaxy public server. Is there a way to access output files (via ftp, perhaps) so I can bulk download them to my computer? I am over my quota and want to get data off of Galaxy but prefer not to do this all one at a time. Similarly, is there a way to access a directory (via unix, ftp, etc) to rename files quickly while they are on Galaxy, since renaming each output file (i.e. the multiple ones output from cuffdiff) within galaxy is very inefficient and time consuming.

Thanks.

Rich

-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support

Hi Richard, Yes, it takes a short time for the UI counts to update. If you deleted permanently, then the result should be what you expected. Should the quota count remain high by tomorrow, that would point to an issue with lingering data counting in the quota. Places to search for unexplained disk use: 1 - Older pre-quota "deleted" datasets that were not permanently deleted. You can check for these in the View Histories -> advanced -> deleted set. The far right column "Status" will note deleted vs permanently deleted. 2 - Shared histories can count towards a quota. So, if not needed or only portions are, copy out of these what you want to use and ask the user that shared the data to "unshared" you, so you don't get stuck with the entire history in your quota. Shared histories/data and quotas are somewhat tricky to tune, and better solutions may be developed as the details are worked out, but this is the current implementation. A good feature to know about is that an imported dataset from a public Data Library never counts towards your quota (if left unmodified). You have probably seen this, but for others who may be reading the thread, this wiki has many details and tips for managing data: http://galaxyproject.org/wiki/Learn/Managing%20Datasets One last comment - it would be very helpful for us if questions were sent with the mailing list as a "to" recipient, so that our ticket tracker picks it up. Hopefully this helps! And please feel free to ask if you need more help or the disk size is not what you expect after the counts refresh. Best, Jen Galaxy team On 11/30/11 9:49 AM, Richard Mark White wrote:

Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?

richard

-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support

Hi Rich, This sounds correct. The quota is set on Main to be 250G. So anything at or over that amount will be 100% of use. http://galaxyproject.org/wiki/Main#User_data_and_job_quotas If I misunderstood your question, please provide more details. Thanks! Jen Galaxy team On 11/30/11 12:07 PM, Richard Mark White wrote:

Hi, Thanks for the info and I did what you suggested. But still no luck. I deleted everything, and when I add up the data totals in my active histories (I have nothing shared) it adds up to 376gb, but i am showing 100%. any ideas?

rich

------------------------------------------------------------------------ *From:* Jennifer Jackson <jen@bx.psu.edu> *To:* Richard Mark White <whiter3@yahoo.com>; "galaxy-user@bx.psu.edu" <galaxy-user@bx.psu.edu> *Cc:* closeticket@galaxyproject.org *Sent:* Wednesday, November 30, 2011 1:09 PM *Subject:* disk quota not updating

Hi Richard,

Yes, it takes a short time for the UI counts to update. If you deleted permanently, then the result should be what you expected. Should the quota count remain high by tomorrow, that would point to an issue with lingering data counting in the quota.

Places to search for unexplained disk use:

1 - Older pre-quota "deleted" datasets that were not permanently deleted. You can check for these in the View Histories -> advanced -> deleted set. The far right column "Status" will note deleted vs permanently deleted.

2 - Shared histories can count towards a quota. So, if not needed or only portions are, copy out of these what you want to use and ask the user that shared the data to "unshared" you, so you don't get stuck with the entire history in your quota. Shared histories/data and quotas are somewhat tricky to tune, and better solutions may be developed as the details are worked out, but this is the current implementation. A good feature to know about is that an imported dataset from a public Data Library never counts towards your quota (if left unmodified).

You have probably seen this, but for others who may be reading the thread, this wiki has many details and tips for managing data: http://galaxyproject.org/wiki/Learn/Managing%20Datasets

One last comment - it would be very helpful for us if questions were sent with the mailing list as a "to" recipient, so that our ticket tracker picks it up.

Hopefully this helps! And please feel free to ask if you need more help or the disk size is not what you expect after the counts refresh.

Best,

Jen Galaxy team

On 11/30/11 9:49 AM, Richard Mark White wrote:

...

Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?

richard

-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support

-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support

yup...took a while, but eventually resolved. thanks. rich ________________________________ From: Nate Coraor <nate@bx.psu.edu> To: Richard Mark White <whiter3@yahoo.com> Cc: Jennifer Jackson <jen@bx.psu.edu>; "galaxy-user@bx.psu.edu" <galaxy-user@bx.psu.edu> Sent: Monday, December 5, 2011 3:39 PM Subject: Re: [galaxy-user] disk quota not updating On Nov 30, 2011, at 12:49 PM, Richard Mark White wrote:

Hi,   I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently".  But my usage did not go down at all.  Is there a delay in this happening, or is there some way to purge the files?

Hi Richard, It can take a bit if you delete a large amount of data at once.  Did your usage eventually decrease? --nate

richard

___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org.  Please keep all replies on the list by using "reply all" in your mail client.  For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

I am unable to access for past several hours.  Are others having the same issue? rich

Hi,   I have generated a transcript file using cufflinks for the human (hg19) or zebrafish(zv9) assemblies.  When I try to display the cufflinks "assembled transcripts" in UCSC I get this error in the UCSC browser and it wont display the transcripts. human: GFF/GTF group NM_005638 on chrX+, this line is on chrY+, all group members must be on same seq and strand zebrafish: "GFF/GTF group vapb on chr6+, this line is on chr7-, all group members must be on same seq and strand" Any ideas? Rich

Hi, Is anyone else having trouble connecting to main.g2.bx.psu.edu for FTP uploads?  I cannot seem to connect since yesterday. Rich

Hi the file name is spr097786.fastq.bz2.After upload it showed spr097786.fastq. It showed it only contain around 5000 sequence reads. I also tried to upload through FTP. so i download the file to my computer and then upload to FTP in galaxy. the totlal 800M file was uploaded to the FTP successfully. But when i transfered the file to the history i met the same problem. only 5000 sequence reads was moved to history. I donnot whether it is because of the bz2 file extension. or i should try other compressed file extension. xaingmimg Quoting Jennifer Jackson <jen@bx.psu.edu>:

Hello Xiangmimg,

Data files can be loaded using a URL on the "Get Data => Upload" form. FTP and HTTP connections are supported. This is briefly described on that form.

If you are still having issues, there may be a problem with file compression or the connection. Downloading locally then using Galaxy's FTP upload function is certainly an option. http://wiki.g2.bx.psu.edu/Learn/Upload%20via%20FTP

Best,

Jen Galaxy team

On 11/6/11 8:54 PM, Xiangming Ding wrote:

...

Hi galaxy

I am a new user of galaxy. i met a problem and didnot find similar question in FAQ. I wanted to upload the data from DDBJ DRA dataset to galaxy through UTL method. The file is around 800M. However after uploading, the FASTQ file was just around 2M. So I wanted know whether it is possible to upload a large file to galaxy through URL method? or I should download the file to my pc and then uploading to galaxy through FTP method.

Thanks

xiangmimg ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:

http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support