Galaxy on Cloud not recognizing .txt files as fastq
Hello, I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries. However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself. Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in the S3 bucket? thanks for any help! karl
Karl, For the URL you're pasting, is it the full path to a file in your s3 bucket, including the protocol (http://) ? For galaxy to recognize that it needs to pull from a URL rather than the contents of the box, you need to use the full string. -Dannon On Mar 14, 2011, at 4:40 PM, karlerhard@berkeley.edu wrote:
Hello,
I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries.
However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself.
Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in the S3 bucket?
thanks for any help!
karl
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
I have uploaded Illumina fastq file and use the Groomer
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Karl, As a quick check, are the permissions for the S3 file set so anyone can read the file? That's required before Galaxy's upload tool will be able to access the file. Alternatively, if you don't want to set such loose permissions, you should be able to get a a signed URL for the given file using a Firefox extension S3Fox. Enis On Mon, Mar 14, 2011 at 4:40 PM, <karlerhard@berkeley.edu> wrote:
Hello,
I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries.
However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself.
Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in the S3 bucket?
thanks for any help!
karl
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Ah, permissions, that may be the ticket. I will try again and see. thanks a lot! karl
Hi Karl, As a quick check, are the permissions for the S3 file set so anyone can read the file? That's required before Galaxy's upload tool will be able to access the file. Alternatively, if you don't want to set such loose permissions, you should be able to get a a signed URL for the given file using a Firefox extension S3Fox.
Enis
On Mon, Mar 14, 2011 at 4:40 PM, <karlerhard@berkeley.edu> wrote:
Hello,
I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries.
However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself.
Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in the S3 bucket?
thanks for any help!
karl
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
I changed the permission for the files to public (everyone can open/download), but this did not fix the problem. Also, to answer Dannon's question, I am entering the full path for these files, eg. https://s3.amazonaws.com/bucketname/xyz.txt
From the screencasts I have watched, I noticed that fastq files uploaded (from an S3 bucket or from elsewhere) always had a .fastq extension. Not sure whether this is an issue for galaxy or not.
Hi Karl, As a quick check, are the permissions for the S3 file set so anyone can read the file? That's required before Galaxy's upload tool will be able to access the file. Alternatively, if you don't want to set such loose permissions, you should be able to get a a signed URL for the given file using a Firefox extension S3Fox.
Enis
On Mon, Mar 14, 2011 at 4:40 PM, <karlerhard@berkeley.edu> wrote:
Hello,
I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries.
However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself.
Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in the S3 bucket?
thanks for any help!
karl
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Karl, Using the upload tool for https URLs was added in changeset 5220:425076fe5ea0. I am not sure when this will make it to the Cloud image, but meanwhile, if you can access the file without using ssl (just 'http'), you should be able to upload. If this is not the issue you are experiencing please let us know. Do not hesitate to contact us if you need further assistance. Thanks for using Galaxy, Dan On Mar 14, 2011, at 7:30 PM, karlerhard@berkeley.edu wrote:
I changed the permission for the files to public (everyone can open/download), but this did not fix the problem.
Also, to answer Dannon's question, I am entering the full path for these files, eg. https://s3.amazonaws.com/bucketname/xyz.txt
From the screencasts I have watched, I noticed that fastq files uploaded (from an S3 bucket or from elsewhere) always had a .fastq extension. Not sure whether this is an issue for galaxy or not.
Hi Karl, As a quick check, are the permissions for the S3 file set so anyone can read the file? That's required before Galaxy's upload tool will be able to access the file. Alternatively, if you don't want to set such loose permissions, you should be able to get a a signed URL for the given file using a Firefox extension S3Fox.
Enis
On Mon, Mar 14, 2011 at 4:40 PM, <karlerhard@berkeley.edu> wrote:
Hello,
I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries.
However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself.
Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in the S3 bucket?
thanks for any help!
karl
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Thanks Dan. The revision of Galaxy is the problem - the instance of Galaxy on the cloud is a month old now; it is at revision 5064 so. I'll release an update in the coming days along with an updated CloudMan. Enis On Mon, Mar 14, 2011 at 11:24 PM, Daniel Blankenberg <dan@bx.psu.edu> wrote:
Hi Karl,
Using the upload tool for https URLs was added in changeset 5220:425076fe5ea0. I am not sure when this will make it to the Cloud image, but meanwhile, if you can access the file without using ssl (just 'http'), you should be able to upload. If this is not the issue you are experiencing please let us know. Do not hesitate to contact us if you need further assistance.
Thanks for using Galaxy,
Dan
On Mar 14, 2011, at 7:30 PM, karlerhard@berkeley.edu wrote:
I changed the permission for the files to public (everyone can open/download), but this did not fix the problem.
Also, to answer Dannon's question, I am entering the full path for these files, eg. https://s3.amazonaws.com/bucketname/xyz.txt
From the screencasts I have watched, I noticed that fastq files uploaded (from an S3 bucket or from elsewhere) always had a .fastq extension. Not sure whether this is an issue for galaxy or not.
Hi Karl, As a quick check, are the permissions for the S3 file set so anyone can read the file? That's required before Galaxy's upload tool will be able to access the file. Alternatively, if you don't want to set such loose permissions, you
be able to get a a signed URL for the given file using a Firefox extension S3Fox.
Enis
On Mon, Mar 14, 2011 at 4:40 PM, <karlerhard@berkeley.edu> wrote:
Hello,
I have been able to get Galaxy to instantiate on our Cloud account, and would like to use the NGS tools to trim and map Illumina libraries.
However, when I tried to import .txt files (the data contained in these files are in fastq format) located in an S3 bucket, Galaxy did not recognize them as fastq data. I tried to import the files via the URL of the S3 bucket, and Galaxy just imported the actual name of the URL rather than the file itself.
Do I need to change the extension of these .txt files to .fastq? Or have I gotten something wrong with respect to the location of the data in
should the
S3 bucket?
thanks for any help!
karl
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (5)
-
Daniel Blankenberg -
Dannon Baker -
Enis Afgan -
karlerhard＠berkeley.edu -
Wong,Kwong K