question about uploading data through URL method
Hi galaxy I am a new user of galaxy. i met a problem and didnot find similar question in FAQ. I wanted to upload the data from DDBJ DRA dataset to galaxy through UTL method. The file is around 800M. However after uploading, the FASTQ file was just around 2M. So I wanted know whether it is possible to upload a large file to galaxy through URL method? or I should download the file to my pc and then uploading to galaxy through FTP method. Thanks xiangmimg
You can download DRA files directly by FTP to Galaxy . . Just paste the FTP address directly in the file box when using upload from my computer Best Simon On 7 November 2011 04:54, Xiangming Ding <dingxm@ucla.edu> wrote:
Hi galaxy
I am a new user of galaxy. i met a problem and didnot find similar question in FAQ. I wanted to upload the data from DDBJ DRA dataset to galaxy through UTL method. The file is around 800M. However after uploading, the FASTQ file was just around 2M. So I wanted know whether it is possible to upload a large file to galaxy through URL method? or I should download the file to my pc and then uploading to galaxy through FTP method.
Thanks
xiangmimg ______________________________**_____________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/**listinfo/galaxy-dev<http://lists.bx.psu.edu/listinfo/galaxy-dev>
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hello Xiangmimg, Data files can be loaded using a URL on the "Get Data => Upload" form. FTP and HTTP connections are supported. This is briefly described on that form. If you are still having issues, there may be a problem with file compression or the connection. Downloading locally then using Galaxy's FTP upload function is certainly an option. http://wiki.g2.bx.psu.edu/Learn/Upload%20via%20FTP Best, Jen Galaxy team On 11/6/11 8:54 PM, Xiangming Ding wrote:
Hi galaxy
I am a new user of galaxy. i met a problem and didnot find similar question in FAQ. I wanted to upload the data from DDBJ DRA dataset to galaxy through UTL method. The file is around 800M. However after uploading, the FASTQ file was just around 2M. So I wanted know whether it is possible to upload a large file to galaxy through URL method? or I should download the file to my pc and then uploading to galaxy through FTP method.
Thanks
xiangmimg ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
Hi, I am using the galaxy public server. Is there a way to access output files (via ftp, perhaps) so I can bulk download them to my computer? I am over my quota and want to get data off of Galaxy but prefer not to do this all one at a time. Similarly, is there a way to access a directory (via unix, ftp, etc) to rename files quickly while they are on Galaxy, since renaming each output file (i.e. the multiple ones output from cuffdiff) within galaxy is very inefficient and time consuming. Thanks. Rich
Hi Rich, This is a good question! Maybe people have been asking about this. To download data with a unix line command method, please try wget or curl, for example: unix% wget 'url_for_the_dataset' or unix% wget 'url_for_the_history' To capture the url for a dataset, right click on the disk icon for a dataset and select "copy link location". To capture the url for an entire history, select "Options -> Export to File". The middle panel will display a link. A downloaded history can be loaded into a local Galaxy instance where the datasets can be managed (copy/rename) or the histories archived. Hopefully this helps you and others that are managing larger datasets & histories, Best, Jen Galaxy team On 11/7/11 7:50 AM, Richard Mark White wrote:
Hi, I am using the galaxy public server. Is there a way to access output files (via ftp, perhaps) so I can bulk download them to my computer? I am over my quota and want to get data off of Galaxy but prefer not to do this all one at a time. Similarly, is there a way to access a directory (via unix, ftp, etc) to rename files quickly while they are on Galaxy, since renaming each output file (i.e. the multiple ones output from cuffdiff) within galaxy is very inefficient and time consuming.
Thanks.
Rich
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
Hi the file name is spr097786.fastq.bz2.After upload it showed spr097786.fastq. It showed it only contain around 5000 sequence reads. I also tried to upload through FTP. so i download the file to my computer and then upload to FTP in galaxy. the totlal 800M file was uploaded to the FTP successfully. But when i transfered the file to the history i met the same problem. only 5000 sequence reads was moved to history. I donnot whether it is because of the bz2 file extension. or i should try other compressed file extension. xaingmimg Quoting Jennifer Jackson <jen@bx.psu.edu>:
Hello Xiangmimg,
Data files can be loaded using a URL on the "Get Data => Upload" form. FTP and HTTP connections are supported. This is briefly described on that form.
If you are still having issues, there may be a problem with file compression or the connection. Downloading locally then using Galaxy's FTP upload function is certainly an option. http://wiki.g2.bx.psu.edu/Learn/Upload%20via%20FTP
Best,
Jen Galaxy team
On 11/6/11 8:54 PM, Xiangming Ding wrote:
Hi galaxy
I am a new user of galaxy. i met a problem and didnot find similar question in FAQ. I wanted to upload the data from DDBJ DRA dataset to galaxy through UTL method. The file is around 800M. However after uploading, the FASTQ file was just around 2M. So I wanted know whether it is possible to upload a large file to galaxy through URL method? or I should download the file to my pc and then uploading to galaxy through FTP method.
Thanks
xiangmimg ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
Hello Xaingmimg, When you uncompress the archive locally, does it contain a single file with more than 5000 reads? The consistent results and even number of reads (5000) may mean that the archive contains more than one file. Currently, Galaxy will only load the first file in an archive. Hopefully this helps or you have already found the solution, Take care, Jen Galaxy team On 11/7/11 10:35 PM, Xiangming Ding wrote:
Hi the file name is spr097786.fastq.bz2.After upload it showed spr097786.fastq. It showed it only contain around 5000 sequence reads. I also tried to upload through FTP. so i download the file to my computer and then upload to FTP in galaxy. the totlal 800M file was uploaded to the FTP successfully. But when i transfered the file to the history i met the same problem. only 5000 sequence reads was moved to history. I donnot whether it is because of the bz2 file extension. or i should try other compressed file extension.
xaingmimg
Quoting Jennifer Jackson <jen@bx.psu.edu>:
Hello Xiangmimg,
Data files can be loaded using a URL on the "Get Data => Upload" form. FTP and HTTP connections are supported. This is briefly described on that form.
If you are still having issues, there may be a problem with file compression or the connection. Downloading locally then using Galaxy's FTP upload function is certainly an option. http://wiki.g2.bx.psu.edu/Learn/Upload%20via%20FTP
Best,
Jen Galaxy team
On 11/6/11 8:54 PM, Xiangming Ding wrote:
Hi galaxy
I am a new user of galaxy. i met a problem and didnot find similar question in FAQ. I wanted to upload the data from DDBJ DRA dataset to galaxy through UTL method. The file is around 800M. However after uploading, the FASTQ file was just around 2M. So I wanted know whether it is possible to upload a large file to galaxy through URL method? or I should download the file to my pc and then uploading to galaxy through FTP method.
Thanks
xiangmimg ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files? richard
Hi Richard, Yes, it takes a short time for the UI counts to update. If you deleted permanently, then the result should be what you expected. Should the quota count remain high by tomorrow, that would point to an issue with lingering data counting in the quota. Places to search for unexplained disk use: 1 - Older pre-quota "deleted" datasets that were not permanently deleted. You can check for these in the View Histories -> advanced -> deleted set. The far right column "Status" will note deleted vs permanently deleted. 2 - Shared histories can count towards a quota. So, if not needed or only portions are, copy out of these what you want to use and ask the user that shared the data to "unshared" you, so you don't get stuck with the entire history in your quota. Shared histories/data and quotas are somewhat tricky to tune, and better solutions may be developed as the details are worked out, but this is the current implementation. A good feature to know about is that an imported dataset from a public Data Library never counts towards your quota (if left unmodified). You have probably seen this, but for others who may be reading the thread, this wiki has many details and tips for managing data: http://galaxyproject.org/wiki/Learn/Managing%20Datasets One last comment - it would be very helpful for us if questions were sent with the mailing list as a "to" recipient, so that our ticket tracker picks it up. Hopefully this helps! And please feel free to ask if you need more help or the disk size is not what you expect after the counts refresh. Best, Jen Galaxy team On 11/30/11 9:49 AM, Richard Mark White wrote:
Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?
richard
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
Hi, Thanks for the info and I did what you suggested. But still no luck. I deleted everything, and when I add up the data totals in my active histories (I have nothing shared) it adds up to 376gb, but i am showing 100%. any ideas? rich ________________________________ From: Jennifer Jackson <jen@bx.psu.edu> To: Richard Mark White <whiter3@yahoo.com>; "galaxy-user@bx.psu.edu" <galaxy-user@bx.psu.edu> Cc: closeticket@galaxyproject.org Sent: Wednesday, November 30, 2011 1:09 PM Subject: disk quota not updating Hi Richard, Yes, it takes a short time for the UI counts to update. If you deleted permanently, then the result should be what you expected. Should the quota count remain high by tomorrow, that would point to an issue with lingering data counting in the quota. Places to search for unexplained disk use: 1 - Older pre-quota "deleted" datasets that were not permanently deleted. You can check for these in the View Histories -> advanced -> deleted set. The far right column "Status" will note deleted vs permanently deleted. 2 - Shared histories can count towards a quota. So, if not needed or only portions are, copy out of these what you want to use and ask the user that shared the data to "unshared" you, so you don't get stuck with the entire history in your quota. Shared histories/data and quotas are somewhat tricky to tune, and better solutions may be developed as the details are worked out, but this is the current implementation. A good feature to know about is that an imported dataset from a public Data Library never counts towards your quota (if left unmodified). You have probably seen this, but for others who may be reading the thread, this wiki has many details and tips for managing data: http://galaxyproject.org/wiki/Learn/Managing%20Datasets One last comment - it would be very helpful for us if questions were sent with the mailing list as a "to" recipient, so that our ticket tracker picks it up. Hopefully this helps! And please feel free to ask if you need more help or the disk size is not what you expect after the counts refresh. Best, Jen Galaxy team On 11/30/11 9:49 AM, Richard Mark White wrote:
Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?
richard
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
Hi Rich, This sounds correct. The quota is set on Main to be 250G. So anything at or over that amount will be 100% of use. http://galaxyproject.org/wiki/Main#User_data_and_job_quotas If I misunderstood your question, please provide more details. Thanks! Jen Galaxy team On 11/30/11 12:07 PM, Richard Mark White wrote:
Hi, Thanks for the info and I did what you suggested. But still no luck. I deleted everything, and when I add up the data totals in my active histories (I have nothing shared) it adds up to 376gb, but i am showing 100%. any ideas?
rich
------------------------------------------------------------------------ *From:* Jennifer Jackson <jen@bx.psu.edu> *To:* Richard Mark White <whiter3@yahoo.com>; "galaxy-user@bx.psu.edu" <galaxy-user@bx.psu.edu> *Cc:* closeticket@galaxyproject.org *Sent:* Wednesday, November 30, 2011 1:09 PM *Subject:* disk quota not updating
Hi Richard,
Yes, it takes a short time for the UI counts to update. If you deleted permanently, then the result should be what you expected. Should the quota count remain high by tomorrow, that would point to an issue with lingering data counting in the quota.
Places to search for unexplained disk use:
1 - Older pre-quota "deleted" datasets that were not permanently deleted. You can check for these in the View Histories -> advanced -> deleted set. The far right column "Status" will note deleted vs permanently deleted.
2 - Shared histories can count towards a quota. So, if not needed or only portions are, copy out of these what you want to use and ask the user that shared the data to "unshared" you, so you don't get stuck with the entire history in your quota. Shared histories/data and quotas are somewhat tricky to tune, and better solutions may be developed as the details are worked out, but this is the current implementation. A good feature to know about is that an imported dataset from a public Data Library never counts towards your quota (if left unmodified).
You have probably seen this, but for others who may be reading the thread, this wiki has many details and tips for managing data: http://galaxyproject.org/wiki/Learn/Managing%20Datasets
One last comment - it would be very helpful for us if questions were sent with the mailing list as a "to" recipient, so that our ticket tracker picks it up.
Hopefully this helps! And please feel free to ask if you need more help or the disk size is not what you expect after the counts refresh.
Best,
Jen Galaxy team
On 11/30/11 9:49 AM, Richard Mark White wrote:
Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?
richard
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
-- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support
On Nov 30, 2011, at 12:49 PM, Richard Mark White wrote:
Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?
Hi Richard, It can take a bit if you delete a large amount of data at once. Did your usage eventually decrease? --nate
richard
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
yup...took a while, but eventually resolved. thanks. rich ________________________________ From: Nate Coraor <nate@bx.psu.edu> To: Richard Mark White <whiter3@yahoo.com> Cc: Jennifer Jackson <jen@bx.psu.edu>; "galaxy-user@bx.psu.edu" <galaxy-user@bx.psu.edu> Sent: Monday, December 5, 2011 3:39 PM Subject: Re: [galaxy-user] disk quota not updating On Nov 30, 2011, at 12:49 PM, Richard Mark White wrote:
Hi, I was nearing my disk quota (at 97%), so I deleted a large number of datasets using "delete permanently". But my usage did not go down at all. Is there a delay in this happening, or is there some way to purge the files?
Hi Richard, It can take a bit if you delete a large amount of data at once. Did your usage eventually decrease? --nate
richard
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi, My seq core returns FASTQ files to me in *.txt.tar.gz format. When I upload this to galaxy, it unzips, but it is apparently still TAR'd and cannot be read. Is it possible to upload this format, or do I need to untar and unzip it first (which is less than ideal)? Rich
I am unable to access for past several hours. Are others having the same issue? rich
On Dec 17, 2011, at 8:34 AM, Richard Mark White wrote:
I am unable to access for past several hours. Are others having the same issue?
Hi Rich, Our core router has crashed, we're working on the problem and hope to have it fixed within the next few hours. Sorry for the inconvenience. --nate
rich
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi, I have generated a transcript file using cufflinks for the human (hg19) or zebrafish(zv9) assemblies. When I try to display the cufflinks "assembled transcripts" in UCSC I get this error in the UCSC browser and it wont display the transcripts. human: GFF/GTF group NM_005638 on chrX+, this line is on chrY+, all group members must be on same seq and strand zebrafish: "GFF/GTF group vapb on chr6+, this line is on chr7-, all group members must be on same seq and strand" Any ideas? Rich
Hi, I have generated a transcript file using cufflinks for the human (hg19) or zebrafish(zv9) assemblies. When I try to display the cufflinks "assembled transcripts" in UCSC I get this error in the UCSC browser and it wont display the transcripts.
human: GFF/GTF group NM_005638 on chrX+, this line is on chrY+, all group members must be on same seq and strand zebrafish: "GFF/GTF group vapb on chr6+, this line is on chr7-, all group members must be on same seq and strand"
Any ideas?
Rich
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi, Is anyone else having trouble connecting to main.g2.bx.psu.edu for FTP uploads? I cannot seem to connect since yesterday. Rich
On Mar 20, 2012, at 8:13 AM, Richard Mark White wrote:
Hi, Is anyone else having trouble connecting to main.g2.bx.psu.edu for FTP uploads? I cannot seem to connect since yesterday.
Hi Rich, It's back up now. --nate
Rich
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (6)
-
Jennifer Jackson
-
Nate Coraor
-
Richard Mark White
-
Richard White
-
Simao Lee
-
Xiangming Ding