Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). I am not sure what is the rationale behind that. Ideally I think there should be no need to upload such heavy files into the workspace. They could actually be used straight away by the path specified. Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
-----------------------------
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/
Abhishek:
Let talk. This is the area of active current development. We are looking at implementing a universal fastq-like format or supporting multiple formats. Perhaps we should join efforts in ironing out specifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). I am not sure what is the rationale behind that. Ideally I think there should be no need to upload such heavy files into the workspace. They could actually be used straight away by the path specified. Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
Hello Abhi,
Can you clarify the steps you took that produced the behavior? See my comments below.
Anton Nekrutenko wrote:
Abhishek:
Let talk. This is the area of active current development. We are looking at implementing a universal fastq-like format or supporting multiple formats. Perhaps we should join efforts in ironing out specifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB).
Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk?
I am not sure what is the rationale
behind that. Ideally I think there should be no need to upload such heavy files into the workspace.
A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. Also, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features.
They could actually be used straight
away by the path specified.
What do you mean by "the path specified"?
Also is there any way to access the
scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. However, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing.
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Hi All
@Greg : Please find my comments below.
On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kusterghv2@psu.edu wrote:
Hello Abhi,
Can you clarify the steps you took that produced the behavior? See my comments below.
Anton Nekrutenko wrote:
Abhishek:
Let talk. This is the area of active current development. We are looking at implementing a universal fastq-like format or supporting multiple formats. Perhaps we should join efforts in ironing out specifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB).
Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk?
Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale
behind that. Ideally I think there should be no need to upload such heavy files into the workspace.
A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. Also, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features.
They could actually be used straight
Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me.
A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge
away by the path specified.
What do you mean by "the path specified"?
Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed.
Also is there any way to access the
scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. However, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing.
Ok I was under the impression that running from SHELL will eliminate the step of uploading them into galaxy file space.
-Abhi
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great. I can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ). But uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All
@Greg : Please find my comments below.
On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kusterghv2@psu.edu wrote:
Hello Abhi,
Can you clarify the steps you took that produced the behavior? †See my comments below.
Anton Nekrutenko wrote:
Abhishek:
Let talk. This is the area of active current development. We are †looking at implementing a universal fastq-like format or supporting †multiple formats. Perhaps we should join efforts in ironing out †specifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB).
Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk?
Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale
behind that. Ideally I think there should be no need to upload such heavy files into the workspace.
A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. †Also, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features.
They could actually be used straight
Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me.
A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge
away by the path specified.
What do you mean by "the path specified"?
Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed.
Also is there any way to access the
scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. †However, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing.
Ok I was under the impression that running from SHELL will eliminate the step of uploading them into galaxy file space.
-Abhi
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Hi Greg, Anton and all Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great. I can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ). But uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All
@Greg : Please find my comments below.
On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kusterghv2@psu.edu wrote:
Hello Abhi,
Can you clarify the steps you took that produced the behavior? †See my comments below.
Anton Nekrutenko wrote:
Abhishek:
Let talk. This is the area of active current development. We are †looking at implementing a universal fastq-like format or supporting †multiple formats. Perhaps we should join efforts in ironing out †specifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB).
Are you using the Galaxy upload utility to create an item in your
history that points to the dataset file on disk?
Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale
behind that. Ideally I think there should be no need to upload such
heavy files into the workspace.
A data file that originates from a place external to Galaxy must be
uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. †Also, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features.
They could actually be used straight
Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me.
A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge
away by the path specified.
What do you mean by "the path specified"?
Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed.
Also is there any way to access the
scripts for analysis on the command line. I know this undermines the
main aim of working with galaxy but rite now I am concerned about the performance/time.
You should be able to run any Galaxy tool from the command line as long
as you have all of the tool's required binaries in your path. †However, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing.
Ok I was under the impression that running from SHELL will eliminate the step of uploading them into galaxy file space.
-Abhi
I will be happy to discuss more about this in case you have some
comments/questions for me.
Best, -Abhi
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
On Friday 25 September 2009 21:44:36 Abhishek Pratap wrote:
Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
If you need a more elaborate solution creating folders per group and user you could take a look at what I have written up here:
http://idotamir.blogspot.com/2009/08/adventures-in-galaxy-pt1.html http://idotamir.blogspot.com/2009/09/adventures-in-galaxy-pt2.html
The solution given at galaxy-central-importer should just work(tm) and show you what you can do.
But just don't merge it into the current version - this will not work.
best, ido
Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abishek, We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great.  I can't guarantee that all Galaxy features will function correctly if you do this though. Assaf, have you found that using your script breaks anything? Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ).  But uploading a file to a history will create a new copy of the file each time it is uploaded. Greg Von Kuster Galaxy Development Team Abhishek Pratap wrote: Hi All @Greg : Please find my comments below. On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu <mailto:ghv2@psu.edu>> wrote: Hello Abhi, Can you clarify the steps you took that produced the behavior? †See my comments below. Anton Nekrutenko wrote: Abhishek: Let talk. This is the area of active current development. We are †looking at implementing a universal fastq-like format or supporting †multiple formats. Perhaps we should join efforts in ironing out †specifications. anton galaxy team On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote: Hi All I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it. Few comments : I may have interepretted something described below in a wrong way. My apologies before hand. On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk? Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system. I am not sure what is the rationale behind that. Ideally I think there should be no need to upload such heavy files into the workspace. A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. †Also, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features. They could actually be used straight Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me. A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge away by the path specified. What do you mean by "the path specified"? Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed. Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time. You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. †However, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing. Ok I was under the impression that running from SHELL will eliminate the step of uploading them into galaxy file space. -Abhi I will be happy to discuss more about this in case you have some comments/questions for me. Best, -Abhi ----------------------------- Abhishek Pratap Bioinformatics Software Engineer Institute for Genome Sciences School of Medicine, Univ of Maryland 801, W. Baltimore Street, Baltimore, MD 21209 Ph: (+1)-410-706-2296 www.igs.umaryland.edu/ <http://www.igs.umaryland.edu/> _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Hi Guys!
Thanks for the reply - I wrote a simple .xml which allows the import of any file which is accessible by the galaxy process (read permission) and is mounted in the current file system of the galaxy server. in the web-interface you have to type in the absolute path+filename manually- It copies the file into galaxy (unix cp used). The uploaded file is assumed to be a text file and has to be set manually to the correct file format-
@nate: Thanks! ill have a look at that too-
Greetings,
mat
<tool id="importer_1" name="Cluster file importer"> <description>copies files from import directory into galaxy</description> <command> cp $source $target 2> $log_report </command>
<inputs> <param name="source" type="text" label="Absolute path+filename to file on cluster" optional="false" size="60"/> </inputs>
<outputs> <data name="target" format="text" label="Imported file" /> <data format="text" name="log_report" label="Detailed log report from importer"/> </outputs>
<help> **What it does**
This tool imports a file from the cluster into galaxy. The file will be copied into the galaxy environment, the original file remains on the cluster.
**!Important advice!**
Please make sure the uploaded file is set to the correct file format! Otherwise tools cannot access this file. This can be done by editing the properties of the uploaded file (pen symbol in the history). The default assumed file format is -text- which might be incorrect it most cases. Please keep although in mind that disk space on the galaxy server is limited.
**Example**
/home/mat/myfile.fa
**Support**
Feel free to contact us if you cant upload your files into galaxy. </help> </tool>
HI Greg Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto: ghv2@psu.edu> wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great.  I can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ).  But uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All @Greg : Please find my comments below. On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu <mailto:ghv2@psu.edu>> wrote: Hello Abhi, Can you clarify the steps you took that produced the behavior? †See my comments below. Anton Nekrutenko wrote: Abhishek: Let talk. This is the area of active current development. We are †looking at implementing a universal fastq-like format or supporting †multiple formats. Perhaps we should join efforts in ironing out †specifications. anton galaxy team On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote: Hi All I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it. Few comments : I may have interepretted something described below in a wrong way. My apologies before hand. On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk? Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system. I am not sure what is the rationale behind that. Ideally I think there should be no need to upload such heavy files into the workspace. A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. †Also, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features. They could actually be used straight Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me. A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge away by the path specified. What do you mean by "the path specified"? Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed. Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time. You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. †However, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing. Ok I was under the impression that running from SHELL willeliminate the step of uploading them into galaxy file space.
-Abhi I will be happy to discuss more about this in case you have some comments/questions for me. Best, -Abhi ----------------------------- Abhishek Pratap Bioinformatics Software Engineer Institute for Genome Sciences School of Medicine, Univ of Maryland 801, W. Baltimore Street, Baltimore, MD 21209 Ph: (+1)-410-706-2296 www.igs.umaryland.edu/ <http://www.igs.umaryland.edu/> _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu>http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu>
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu <mailto:ghv2@psu.edu
wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great.  I can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ).  But uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All @Greg : Please find my comments below. On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu <mailto:ghv2@psu.edu>> wrote: Hello Abhi, Can you clarify the steps you took that produced the behavior? †See my comments below. Anton Nekrutenko wrote: Abhishek: Let talk. This is the area of active current development. We are †looking at implementing a universal fastq-like format or supporting †multiple formats. Perhaps we should join efforts in ironing out †specifications. anton galaxy team On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote: Hi All I recently came to know about NGS analysis ongalaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand. On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). Are you using the Galaxy upload utility to create an itemin your history that points to the dataset file on disk?
Yes that is precisely correct, I am trying to upload a solexaFASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale behind that. Ideally I think there should be noneed to upload such heavy files into the workspace.
A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. †Also, when data is uploadedto
Galaxy ( either to a history or a library ), several database tablesettings are created that are used by various Galaxy features.
They could actually be used straight Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me. A. We already one instance of data saved on the local filesystem B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge
away by the path specified. What do you mean by "the path specified"? Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up fromthere rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed.
Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time. You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. †However, running a tool from within Galaxy should generally not be anyslower than running it outside of Galaxy, depending, of course, on what you are doing.
Ok I was under the impression that running from SHELL willeliminate the step of uploading them into galaxy file space.
-Abhi I will be happy to discuss more about this in case you have some comments/questions for me. Best, -Abhi ----------------------------- Abhishek Pratap Bioinformatics Software Engineer Institute for Genome Sciences School of Medicine, Univ of Maryland 801, W. Baltimore Street, Baltimore, MD 21209 Ph: (+1)-410-706-2296 www.igs.umaryland.edu/ <http://www.igs.umaryland.edu/_______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.eduhttp://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate Department of Biostatistics Associate Director Bioinformatics Core Harvard School of Public Health Skype: ohofmann Phone: +1 (617) 365 0984
Dear Oliver,
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
It is quite simple.
If you have 7 minutes do the following from the command line:
hg clone http://bitbucket.org/ido/galaxy-central-importer gci cd gci hg update -C importer sh setup.sh #sets up basic things ./run.sh #end with ctrl-c after startup completes (serving on ....) ./importer.sh #starts the importer wait for "done importing" ./run.sh
then open your browser at localhost:8080/galaxy You can log in with 3 different usernames/passwords: name1@/123456 #can only see group1 libs name2@123456 #can only see group2 libs tamir@/123456 #can see all libs
Of course only the file paths get inserted into the db.
more on: http://idotamir.blogspot.com
best wishes, ido
Change set 2812 will be included in a release to the distribution today - here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired). Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled. Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/'). This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE. This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing. Only the first symlink (the one in the import directory itself) is dereferenced; all others remain. See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x 2 nate nate 512 Oct 1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx 1 nate nate 71 Oct 1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx 1 nate nate 11 Oct 1 10:38 3.bed -> ../../3.bed lrwxrwxrwx 1 nate nate 35 Oct 1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx 1 nate nate 41 Oct 1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx 1 nate nate 44 Oct 1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
OliverOn 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great. ¨ÜI can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ). ¨ÜBut uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All @Greg : Please find my comments below. On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu <mailto:ghv2@psu.edu>> wrote: Hello Abhi, Can you clarify the steps you took that produced the behavior? ǃÜSee my comments below. Anton Nekrutenko wrote: Abhishek: Let talk. This is the area of active current development. We are ǃÜlooking at implementing a universal fastq-like format or supporting ǃÜmultiple formats. Perhaps we should join efforts in ironing out ǃÜspecifications. anton galaxy team On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote: Hi All I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it. Few comments : I may have interepretted something described below in a wrong way. My apologies before hand. On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk? Yes that is precisely correct, I am trying to upload a solexaFASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale behind that. Ideally I think there should be no need to upload such heavy files into the workspace. A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. ǃÜAlso, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features. They could actually be used straight Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me. A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge away by the path specified. What do you mean by "the path specified"? Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed. Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time. You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. ǃÜHowever, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you aredoing.
Ok I was under the impression that running from SHELL willeliminate the step of uploading them into galaxy file space.
-Abhi I will be happy to discuss more about this in case you have some comments/questions for me. Best, -Abhi ----------------------------- Abhishek Pratap Bioinformatics Software Engineer Institute for Genome Sciences School of Medicine, Univ of Maryland 801, W. Baltimore Street, Baltimore, MD 21209 Ph: (+1)-410-706-2296 www.igs.umaryland.edu/ <http://www.igs.umaryland.edu/> _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu>http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu>http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate Department of Biostatistics Associate Director Bioinformatics Core Harvard School of Public Health Skype: ohofmann Phone: +1 (617) 365 0984
Hi Greg
Many thanks for accommodating our requests in quick time. I will be testing it right away.
Have a good weekend.
Cheers, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today - here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired). Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled. Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/'). This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE. This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing. Only the first symlink (the one in the import directory itself) is dereferenced; all others remain. See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x 2 nate nate 512 Oct 1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx 1 nate nate 71 Oct 1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx 1 nate nate 11 Oct 1 10:38 3.bed -> ../../3.bed lrwxrwxrwx 1 nate nate 35 Oct 1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx 1 nate nate 41 Oct 1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx 1 nate nate 44 Oct 1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great. ¨ÜI can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ). ¨ÜBut uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All
@Greg : Please find my comments below.
On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abhi,
Can you clarify the steps you took that produced the behavior? ǃÜSee my
comments below.
Anton Nekrutenko wrote:
Abhishek:
Let talk. This is the area of active current development. We are ǃÜlooking
at implementing a universal fastq-like format or supporting ǃÜmultiple
formats. Perhaps we should join efforts in ironing out ǃÜspecifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB).
Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk?
Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale
behind that. Ideally I think there should be no need to upload such heavy files into the workspace.
A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. ǃÜAlso, when data is uploaded to
Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features.
They could actually be used straight
Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me.
A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge
away by the path specified.
What do you mean by "the path specified"?
Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed.
Also is there any way to access the
scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. ǃÜHowever, running
a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing.
Ok I was under the impression that running from SHELL will eliminate the step of uploading them into galaxy file space.
-Abhi
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
-----------------------------
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ http://www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
_______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate Department of Biostatistics Associate Director Bioinformatics Core Harvard School of Public Health Skype: ohofmann Phone: +1 (617) 365 0984
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today - here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired). Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled. Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/'). This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE. This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing. Only the first symlink (the one in the import directory itself) is dereferenced; all others remain. See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x 2 nate nate 512 Oct 1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx 1 nate nate 71 Oct 1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx 1 nate nate 11 Oct 1 10:38 3.bed -> ../../3.bed lrwxrwxrwx 1 nate nate 35 Oct 1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx 1 nate nate 41 Oct 1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx 1 nate nate 44 Oct 1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great. ¨ÜI can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ). ¨ÜBut uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All
@Greg : Please find my comments below.
On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abhi,
Can you clarify the steps you took that produced the behavior? ǃÜSee my
comments below.
Anton Nekrutenko wrote:
Abhishek:
Let talk. This is the area of active current development. We are ǃÜlooking
at implementing a universal fastq-like format or supporting ǃÜmultiple
formats. Perhaps we should join efforts in ironing out ǃÜspecifications.
anton galaxy team
On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
Hi All
I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it.
Few comments : I may have interepretted something described below in a wrong way. My apologies before hand.
On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB).
Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk?
Yes that is precisely correct, I am trying to upload a solexa FASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale
behind that. Ideally I think there should be no need to upload such heavy files into the workspace.
A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. ǃÜAlso, when data is uploaded to
Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features.
They could actually be used straight
Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me.
A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge
away by the path specified.
What do you mean by "the path specified"?
Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed.
Also is there any way to access the
scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time.
You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. ǃÜHowever, running
a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you are doing.
Ok I was under the impression that running from SHELL will eliminate the step of uploading them into galaxy file space.
-Abhi
I will be happy to discuss more about this in case you have some comments/questions for me.
Best, -Abhi
-----------------------------
Abhishek Pratap
Bioinformatics Software Engineer
Institute for Genome Sciences
School of Medicine, Univ of Maryland
801, W. Baltimore Street, Baltimore, MD 21209
Ph: (+1)-410-706-2296
www.igs.umaryland.edu/ http://www.igs.umaryland.edu/ _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
_______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate Department of Biostatistics Associate Director Bioinformatics Core Harvard School of Public Health Skype: ohofmann Phone: +1 (617) 365 0984
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today - here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired).  Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled.  Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/').  This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE.  This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing.  Only the first symlink (the one in the import directory itself) is dereferenced; all others remain.  See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x   2 nate     nate         512 Oct  1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx   1 nate     nate          71 Oct  1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx   1 nate     nate          11 Oct  1 10:38 3.bed -> ../../3.bed lrwxrwxrwx   1 nate     nate          35 Oct  1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx   1 nate     nate          41 Oct  1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx   1 nate     nate          44 Oct  1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
   Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location  but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files.  Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files.  The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)?  This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
  Hello Abishek,
  We are currently in the process of significantly enhancing the   current Galaxy upload utilities, and the new version should   eliminate the issue you've raised about the time needed to upload   large files via HTTP ( not for making an initial copy of the file in   the Galaxy environment ). However, it will probably not be ready for   release for a few more weeks, so if you can take advantage of   Assaf's script in the meantime, that's great. ¨ÜI can't guarantee   that all Galaxy features will function correctly if you do this though.
  Assaf, have you found that using your script breaks anything?
  Also, if you upload a file to a library rather than a history,   multiple users can "import" the library dataset into their history   for analysis, but there is only 1 file on disk ( users are pointing   to it from their histories ). ¨ÜBut uploading a file to a history   will create a new copy of the file each time it is uploaded.
  Greg Von Kuster   Galaxy Development Team
  Abhishek Pratap wrote:
      Hi All
      @Greg : Please find my comments below.
      On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu       mailto:ghv2@psu.edu> wrote:
          Hello Abhi,
          Can you clarify the steps you took that produced the           behavior? ǃÜSee my
          comments below.
          Anton Nekrutenko wrote:
              Abhishek:
              Let talk. This is the area of active current               development. We are ǃÜlooking
              at implementing a universal fastq-like format or               supporting ǃÜmultiple
              formats. Perhaps we should join efforts in ironing out               ǃÜspecifications.
              anton               galaxy team
              On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
                  Hi All
                  I recently came to know about NGS analysis on galaxy                   during ISMB.                   Getting excited I tried couple of things basically                   to play with it.
                  Few comments : I may have interepretted something                   described below in a                   wrong way. My apologies before hand.
                  On a standalone installation of galaxy while I was                   trying to explore                   one FASTQ(sequence) file. It takes considerable (>                   20 min) for a fastq                   file to get uploaded (2 GB).
          Are you using the Galaxy upload utility to create an item in           your history           that points to the dataset file on disk?
      Yes that is precisely correct, I am trying to upload a solexa FASTQ       file but on a standalone galaxy installation from my local file       system.
          I am not sure what is the rationale
                  behind that. Ideally I think there should be no need                   to upload such                   heavy files into the workspace.
          A data file that originates from a place external to Galaxy           must be uploaded           into Galaxy so that the disk file can be placed in the           location configured           in the Galaxy config file. ǃÜAlso, when data is uploaded to
          Galaxy ( either           to a history or a library ), several database table settings           are created           that are used by various Galaxy features.
          They could actually be used straight
      Thanks for the clarification but I am not sure this will help a       lot of       people who are interested to install and run galaxy locally       mainly for       the following reasons. May be it is just local to me.
      A. We already one instance of data saved on the local file system       B. Making another copy via galaxy will eat away a lot of space       in long run.       C. The time needed to import the files into galaxy space is huge
                  away by the path specified.
          What do you mean by "the path specified"?
      Well what I mean was a way to specify the path of the file/run       on the       lcoal file system and galaxy could directly pick it up from there       rather than uploading it into its own space. Now I understand this       might not work based on the way the system was designed.
          Also is there any way to access the
                  scripts for analysis on the command line. I know                   this undermines the                   main aim of working with galaxy but rite now I am                   concerned about the                   performance/time.
          You should be able to run any Galaxy tool from the command           line as long as           you have all of the tool's required binaries in your path.           ǃÜHowever, running
          a tool from within Galaxy should generally not be any slower           than running it           outside of Galaxy, depending, of course, on what you are doing.
      Ok I was under the impression that running from SHELL will eliminate       the step of uploading them into galaxy file space.
      -Abhi
                  I will be happy to discuss more about this in case                   you have some                   comments/questions for me.
                  Best,                   -Abhi
                  -----------------------------
                  Abhishek Pratap
                  Bioinformatics Software Engineer
                  Institute for Genome Sciences
                  School of Medicine, Univ of Maryland
                  801, W. Baltimore Street, Baltimore, MD 21209
                  Ph: (+1)-410-706-2296
                  www.igs.umaryland.edu/ http://www.igs.umaryland.edu/                   _______________________________________________                   galaxy-user mailing list                   galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
              Anton Nekrutenko               http://nekrut.bx.psu.edu               http://galaxyproject.org
              _______________________________________________               galaxy-user mailing list               galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
              http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate    Department of Biostatistics Associate Director    Bioinformatics Core                      Harvard School of Public Health Skype: ohofmann       Phone: +1 (617) 365 0984
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today - here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired).  Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled.  Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/').  This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE.  This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing.  Only the first symlink (the one in the import directory itself) is dereferenced; all others remain.  See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x   2 nate     nate         512 Oct  1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx   1 nate     nate          71 Oct  1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx   1 nate     nate          11 Oct  1 10:38 3.bed -> ../../3.bed lrwxrwxrwx   1 nate     nate          35 Oct  1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx   1 nate     nate          41 Oct  1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx   1 nate     nate          44 Oct  1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
   Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location  but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files.  Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files.  The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)?  This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
  Hello Abishek,
  We are currently in the process of significantly enhancing the   current Galaxy upload utilities, and the new version should   eliminate the issue you've raised about the time needed to upload   large files via HTTP ( not for making an initial copy of the file in   the Galaxy environment ). However, it will probably not be ready for   release for a few more weeks, so if you can take advantage of   Assaf's script in the meantime, that's great. ¨ÜI can't guarantee   that all Galaxy features will function correctly if you do this though.
  Assaf, have you found that using your script breaks anything?
  Also, if you upload a file to a library rather than a history,   multiple users can "import" the library dataset into their history   for analysis, but there is only 1 file on disk ( users are pointing   to it from their histories ). ¨ÜBut uploading a file to a history   will create a new copy of the file each time it is uploaded.
  Greg Von Kuster   Galaxy Development Team
  Abhishek Pratap wrote:
      Hi All
      @Greg : Please find my comments below.
      On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu       mailto:ghv2@psu.edu> wrote:
          Hello Abhi,
          Can you clarify the steps you took that produced the           behavior? ǃÜSee my
          comments below.
          Anton Nekrutenko wrote:
              Abhishek:
              Let talk. This is the area of active current               development. We are ǃÜlooking
              at implementing a universal fastq-like format or               supporting ǃÜmultiple
              formats. Perhaps we should join efforts in ironing out               ǃÜspecifications.
              anton               galaxy team
              On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
                  Hi All
                  I recently came to know about NGS analysis on galaxy                   during ISMB.                   Getting excited I tried couple of things basically                   to play with it.
                  Few comments : I may have interepretted something                   described below in a                   wrong way. My apologies before hand.
                  On a standalone installation of galaxy while I was                   trying to explore                   one FASTQ(sequence) file. It takes considerable (>                   20 min) for a fastq                   file to get uploaded (2 GB).
          Are you using the Galaxy upload utility to create an item in           your history           that points to the dataset file on disk?
      Yes that is precisely correct, I am trying to upload a solexa FASTQ       file but on a standalone galaxy installation from my local file       system.
          I am not sure what is the rationale
                  behind that. Ideally I think there should be no need                   to upload such                   heavy files into the workspace.
          A data file that originates from a place external to Galaxy           must be uploaded           into Galaxy so that the disk file can be placed in the           location configured           in the Galaxy config file. ǃÜAlso, when data is uploaded to
          Galaxy ( either           to a history or a library ), several database table settings           are created           that are used by various Galaxy features.
          They could actually be used straight
      Thanks for the clarification but I am not sure this will help a       lot of       people who are interested to install and run galaxy locally       mainly for       the following reasons. May be it is just local to me.
      A. We already one instance of data saved on the local file system       B. Making another copy via galaxy will eat away a lot of space       in long run.       C. The time needed to import the files into galaxy space is huge
                  away by the path specified.
          What do you mean by "the path specified"?
      Well what I mean was a way to specify the path of the file/run       on the       lcoal file system and galaxy could directly pick it up from there       rather than uploading it into its own space. Now I understand this       might not work based on the way the system was designed.
          Also is there any way to access the
                  scripts for analysis on the command line. I know                   this undermines the                   main aim of working with galaxy but rite now I am                   concerned about the                   performance/time.
          You should be able to run any Galaxy tool from the command           line as long as           you have all of the tool's required binaries in your path.           ǃÜHowever, running
          a tool from within Galaxy should generally not be any slower           than running it           outside of Galaxy, depending, of course, on what you are doing.
      Ok I was under the impression that running from SHELL will eliminate       the step of uploading them into galaxy file space.
      -Abhi
                  I will be happy to discuss more about this in case                   you have some                   comments/questions for me.
                  Best,                   -Abhi
                  -----------------------------
                  Abhishek Pratap
                  Bioinformatics Software Engineer
                  Institute for Genome Sciences
                  School of Medicine, Univ of Maryland
                  801, W. Baltimore Street, Baltimore, MD 21209
                  Ph: (+1)-410-706-2296
                  www.igs.umaryland.edu/ http://www.igs.umaryland.edu/                   _______________________________________________                   galaxy-user mailing list                   galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
              Anton Nekrutenko               http://nekrut.bx.psu.edu               http://galaxyproject.org
              _______________________________________________               galaxy-user mailing list               galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
              http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate    Department of Biostatistics Associate Director    Bioinformatics Core                      Harvard School of Public Health Skype: ohofmann       Phone: +1 (617) 365 0984
Please type the following in your galaxy install directory, and let me know what you get:
hg heads
Thanks
Abhishek Pratap wrote:
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets  ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today - here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired).  Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled.  Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/').  This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE.  This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing.  Only the first symlink (the one in the import directory itself) is dereferenced; all others remain.  See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x   2 nate     nate         512 Oct  1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx   1 nate     nate          71 Oct  1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx   1 nate     nate          11 Oct  1 10:38 3.bed -> ../../3.bed lrwxrwxrwx   1 nate     nate          35 Oct  1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx   1 nate     nate          41 Oct  1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx   1 nate     nate          44 Oct  1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
   Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location  but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files.  Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files.  The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)?  This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
  Hello Abishek,
  We are currently in the process of significantly enhancing the   current Galaxy upload utilities, and the new version should   eliminate the issue you've raised about the time needed to upload   large files via HTTP ( not for making an initial copy of the file in   the Galaxy environment ). However, it will probably not be ready for   release for a few more weeks, so if you can take advantage of   Assaf's script in the meantime, that's great. ¨ÜI can't guarantee   that all Galaxy features will function correctly if you do this though.
  Assaf, have you found that using your script breaks anything?
  Also, if you upload a file to a library rather than a history,   multiple users can "import" the library dataset into their history   for analysis, but there is only 1 file on disk ( users are pointing   to it from their histories ). ¨ÜBut uploading a file to a history   will create a new copy of the file each time it is uploaded.
  Greg Von Kuster   Galaxy Development Team
  Abhishek Pratap wrote:
      Hi All
      @Greg : Please find my comments below.
      On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu       mailto:ghv2@psu.edu> wrote:
          Hello Abhi,
          Can you clarify the steps you took that produced the           behavior? ǃÜSee my
          comments below.
          Anton Nekrutenko wrote:
              Abhishek:
              Let talk. This is the area of active current               development. We are ǃÜlooking
              at implementing a universal fastq-like format or               supporting ǃÜmultiple
              formats. Perhaps we should join efforts in ironing out               ǃÜspecifications.
              anton               galaxy team
              On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote:
                  Hi All
                  I recently came to know about NGS analysis on galaxy                   during ISMB.                   Getting excited I tried couple of things basically                   to play with it.
                  Few comments : I may have interepretted something                   described below in a                   wrong way. My apologies before hand.
                  On a standalone installation of galaxy while I was                   trying to explore                   one FASTQ(sequence) file. It takes considerable (>                   20 min) for a fastq                   file to get uploaded (2 GB).
          Are you using the Galaxy upload utility to create an item in           your history           that points to the dataset file on disk?
      Yes that is precisely correct, I am trying to upload a solexa FASTQ       file but on a standalone galaxy installation from my local file       system.
          I am not sure what is the rationale
                  behind that. Ideally I think there should be no need                   to upload such                   heavy files into the workspace.
          A data file that originates from a place external to Galaxy           must be uploaded           into Galaxy so that the disk file can be placed in the           location configured           in the Galaxy config file. ǃÜAlso, when data is uploaded to
          Galaxy ( either           to a history or a library ), several database table settings           are created           that are used by various Galaxy features.
          They could actually be used straight
      Thanks for the clarification but I am not sure this will help a       lot of       people who are interested to install and run galaxy locally       mainly for       the following reasons. May be it is just local to me.
      A. We already one instance of data saved on the local file system       B. Making another copy via galaxy will eat away a lot of space       in long run.       C. The time needed to import the files into galaxy space is huge
                  away by the path specified.
          What do you mean by "the path specified"?
      Well what I mean was a way to specify the path of the file/run       on the       lcoal file system and galaxy could directly pick it up from there       rather than uploading it into its own space. Now I understand this       might not work based on the way the system was designed.
          Also is there any way to access the
                  scripts for analysis on the command line. I know                   this undermines the                   main aim of working with galaxy but rite now I am                   concerned about the                   performance/time.
          You should be able to run any Galaxy tool from the command           line as long as           you have all of the tool's required binaries in your path.           ǃÜHowever, running
          a tool from within Galaxy should generally not be any slower           than running it           outside of Galaxy, depending, of course, on what you are doing.
      Ok I was under the impression that running from SHELL will eliminate       the step of uploading them into galaxy file space.
      -Abhi
                  I will be happy to discuss more about this in case                   you have some                   comments/questions for me.
                  Best,                   -Abhi
                  -----------------------------
                  Abhishek Pratap
                  Bioinformatics Software Engineer
                  Institute for Genome Sciences
                  School of Medicine, Univ of Maryland
                  801, W. Baltimore Street, Baltimore, MD 21209
                  Ph: (+1)-410-706-2296
                  www.igs.umaryland.edu/ http://www.igs.umaryland.edu/                   _______________________________________________                   galaxy-user mailing list                   galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
              Anton Nekrutenko               http://nekrut.bx.psu.edu               http://galaxyproject.org
              _______________________________________________               galaxy-user mailing list               galaxy-user@bx.psu.edu mailto:galaxy-user@bx.psu.edu
              http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate    Department of Biostatistics Associate Director    Bioinformatics Core                      Harvard School of Public Health Skype: ohofmann       Phone: +1 (617) 365 0984
Here it is :
hg heads
changeset: 2824:d97f4e86be45 tag: tip parent: 2823:2c0c81150dbd parent: 2821:04a753865407 user: Anton Nekrutenko anton@bx.psu.edu date: Fri Oct 02 11:02:14 2009 -0400 summary: merge
-Abhi
On Fri, Oct 2, 2009 at 3:30 PM, Greg Von Kuster ghv2@psu.edu wrote:
Please type the following in your galaxy install directory, and let me know what you get:
hg heads
Thanks
Abhishek Pratap wrote:
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets  ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today
here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired).  Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled.  Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/').  This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE.  This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing.  Only the first symlink (the one in the import directory itself) is dereferenced; all others remain.  See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x   2 nate     nate         512 Oct  1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx   1 nate     nate          71 Oct  1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx   1 nate     nate          11 Oct  1 10:38 3.bed -> ../../3.bed lrwxrwxrwx   1 nate     nate          35 Oct  1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx   1 nate     nate          41 Oct  1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx   1 nate     nate          44 Oct  1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
   Oliver
On 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
> HI Greg > > Thanks for a quick reply and making some requested changes. However I > am > not still sure if importing NGS data will help in long run. > > For Centers generating NGS data which could 2-3 T.B / week depending > on > no. of sequencers I think importing another copy of raw data into > galaxy > workspace will be asking for lot of disk space. I understand it is a > neat > way of doing things as it becomes agnostic of the raw data location >  but > might not be the best way for handling huge data in long run for > centers > like ours. > > Please correct me if I am wrong. I think we could also have a simple > option without having to import the data and just using it for > analysis > from > the current location, also storing results at the same location. That > way in > future even if the data set is moved analysis also stays with it. > > Let me know what you feel. I will be happy to know if there are any > other > smart reasons of importing the data in galaxy workspace just for > curiosity > sake. > > Thanks, > -Abhi > > On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu > wrote: > Hello Abhishek, > > The Galaxy distribution includes the enhancements to which I > previously > referred for uploading history files.  Uploading files to a > history > now > creates a Galaxy job just like any other tool, and can be run on a > cluster > node, allowing upload of very large files.  The initial pass of > this > work is > also completed for uploading to a Data Library, but this enhancement > is > still in test, so it should soon be available in the distribution. > > Do you want to avoid having to import at all (e.g. allow Galaxy to > refer > to datasets that live in their original locations)?  This is not > currently > possible, but if this is what you are looking for, we can consider > some > additional options on the current upload form, or possibly a new, > separate > form. > > > Greg Von Kuster > Galaxy Development Team > > > Abhishek Pratap wrote: > Hi Greg, Anton and all > > Just wondering if there has been any progress made on this end. I am > sorry I was not able to follow it up on Assaf's suggestion due to > other > things at work. > > I did try the latest version of galaxy and looks like the files are > still > transferred over HTTP before they could be used in the galaxy > workspace. > Also I would again like to highlight that many labs might want to use > the > local instance of galaxy and prefer to point to a local path where > the > file > is being stored. That way we will have both the benefits of using a > cool GUI > and process data stored locally. > > Let me know if you guys need some feedback or have more questions. I > will > be happy to discuss them. > > best, > -Abhi > > On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu > mailto:ghv2@psu.edu> wrote: > >   Hello Abishek, > >   We are currently in the process of significantly enhancing the >   current Galaxy upload utilities, and the new version should >   eliminate the issue you've raised about the time needed to > upload >   large files via HTTP ( not for making an initial copy of the > file in >   the Galaxy environment ). However, it will probably not be > ready for >   release for a few more weeks, so if you can take advantage of >   Assaf's script in the meantime, that's great. ¨ÜI can't > guarantee >   that all Galaxy features will function correctly if you do this > though. > >   Assaf, have you found that using your script breaks anything? > >   Also, if you upload a file to a library rather than a history, >   multiple users can "import" the library dataset into their > history >   for analysis, but there is only 1 file on disk ( users are > pointing >   to it from their histories ). ¨ÜBut uploading a file to > a history >   will create a new copy of the file each time it is uploaded. > >   Greg Von Kuster >   Galaxy Development Team > > > >   Abhishek Pratap wrote: > >       Hi All > >       @Greg : Please find my comments below. > >       On Tue, Jul 21, 2009 at 10:44 AM, Greg Von > Kuster<ghv2@psu.edu >       mailto:ghv2@psu.edu> wrote: > >           Hello Abhi, > >           Can you clarify the steps you took that > produced the >           behavior? ǃÜSee my > >           comments below. > >           Anton Nekrutenko wrote: > >               Abhishek: > >               Let talk. This is the area > of active current >               development. We are > ǃÜlooking > >               at implementing a universal > fastq-like format or >               supporting > ǃÜmultiple > >               formats. Perhaps we should > join efforts in ironing > out >               > ǃÜspecifications. > > >               anton >               galaxy team > > >               On Jul 20, 2009, at 5:18 > PM, Abhishek Pratap > wrote: > >                   Hi All > > >                   I recently came > to know about NGS analysis > on galaxy >                   during ISMB. >                   Getting excited > I tried couple of things > basically >                   to play with > it. > >                   Few comments : > I may have interepretted > something >                   described below > in a >                   wrong way. My > apologies before hand. > > > >                   On a standalone > installation of galaxy while > I was >                   trying to > explore >                   one > FASTQ(sequence) file. It takes > considerable (> >                   20 min) for a > fastq >                   file to get > uploaded (2 GB). > >           Are you using the Galaxy upload utility > to create an > item in >           your history >           that points to the dataset file on > disk? > > >       Yes that is precisely correct, I am trying to > upload a solexa > FASTQ >       file but on a standalone galaxy installation from > my local > file >       system. > >           I am not sure what is the rationale > >                   behind that. > Ideally I think there should be > no need >                   to upload such >                   heavy files > into the workspace. > >           A data file that originates from a > place external to > Galaxy >           must be uploaded >           into Galaxy so that the disk file can > be placed in the >           location configured >           in the Galaxy config file. > ǃÜAlso, when data is > uploaded to > >           Galaxy ( either >           to a history or a library ), several > database table > settings >           are created >           that are used by various Galaxy > features. > >           They could actually be used straight > > >       Thanks for the clarification but I am not sure this > will help > a >       lot of >       people who are interested to install and run galaxy > locally >       mainly for >       the following reasons. May be it is just local to > me. > >       A. We already one instance of data saved on the > local file > system >       B. Making another copy via galaxy will eat away a > lot of space >       in long run. >       C. The time needed to import the files into galaxy > space is > huge > >                   away by the > path specified. > >           What do you mean by "the path > specified"? > > > >       Well what I mean was a way to specify the path of > the file/run >       on the >       lcoal file system and galaxy could directly pick it > up from > there >       rather than uploading it into its own space. Now I > understand > this >       might not work based on the way the system was > designed. > > >           Also is there any way to access the > >                   scripts for > analysis on the command line. I > know >                   this undermines > the >                   main aim of > working with galaxy but rite now > I am >                   concerned about > the >                   > performance/time. > >           You should be able to run any Galaxy > tool from the > command >           line as long as >           you have all of the tool's required > binaries in your > path. >           ǃÜHowever, running > >           a tool from within Galaxy should > generally not be any > slower >           than running it >           outside of Galaxy, depending, of > course, on what you are > doing. > > > > >       Ok I was under the impression that running from > SHELL will > eliminate >       the step of uploading them into galaxy file space. > > >       -Abhi > >                   I will be happy > to discuss more about this > in case >                   you have some >                   > comments/questions for me. > > > >                   Best, >                   -Abhi > > > >                   > ----------------------------- > >                   Abhishek Pratap > >                   Bioinformatics > Software Engineer > >                   Institute for > Genome Sciences > >                   School of > Medicine, Univ of Maryland > >                   801, W. > Baltimore Street, Baltimore, MD > 21209 > >                   Ph: > (+1)-410-706-2296 > >                   > www.igs.umaryland.edu/ > http://www.igs.umaryland.edu/ >                   > _______________________________________________ >                   galaxy-user > mailing list >                   > galaxy-user@bx.psu.edu > mailto:galaxy-user@bx.psu.edu > > > http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user > >               Anton Nekrutenko >               http://nekrut.bx.psu.edu >               http://galaxyproject.org > >               > _______________________________________________ >               galaxy-user mailing list >               galaxy-user@bx.psu.edu > mailto:galaxy-user@bx.psu.edu > >               > http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user > > > > > > > > > _______________________________________________ > galaxy-user mailing list > galaxy-user@bx.psu.edu > http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate    Department of Biostatistics Associate Director    Bioinformatics Core                      Harvard School of Public Health Skype: ohofmann       Phone: +1 (617) 365 0984
Ok, you have the change set. Perhaps you are trying to upload into a history rather than a Data Library? This new feature is only available for uploading into a Data Library. Can you confirm? I'm not sure what else could be causing you to not see the 4 options on the drop-down menu for uploading dataset into a Data Library.
Greg
Abhishek Pratap wrote:
Here it is :
hg heads
changeset: 2824:d97f4e86be45 tag: tip parent: 2823:2c0c81150dbd parent: 2821:04a753865407 user: Anton Nekrutenko anton@bx.psu.edu date: Fri Oct 02 11:02:14 2009 -0400 summary: merge
-Abhi
On Fri, Oct 2, 2009 at 3:30 PM, Greg Von Kuster ghv2@psu.edu wrote:
Please type the following in your galaxy install directory, and let me know what you get:
hg heads
Thanks
Abhishek Pratap wrote:
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets  ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote:
Change set 2812 will be included in a release to the distribution today
here are details of a new option that we're hoping will provide what is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired).  Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled.  Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/').  This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE.  This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing.  Only the first symlink (the one in the import directory itself) is dereferenced; all others remain.  See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x   2 nate     nate         512 Oct  1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx   1 nate     nate          71 Oct  1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx   1 nate     nate          11 Oct  1 10:38 3.bed -> ../../3.bed lrwxrwxrwx   1 nate     nate          35 Oct  1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx   1 nate     nate          41 Oct  1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx   1 nate     nate          60 Oct  1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx   1 nate     nate          44 Oct  1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote: > Dear all, > > > to echo what Abhi said: we are also currently looking of ways to > automatically import data sets (libraries) into Galaxy without having > to > manually trigger the import via the administration interface, and > ideally > while keeping the data in the original place. The idea here is to have > multiple tools all point at the original 'source data' without having > to > replicate terabytes of data. > > Not quite sure how feasible this is in practice, but it certainly > would > be > incredibly helpful. > > Best, > >    Oliver > > > > > On 28 Sep 2009, at 14:24, Abhishek Pratap wrote: > >> HI Greg >> >> Thanks for a quick reply and making some requested changes. However I >> am >> not still sure if importing NGS data will help in long run. >> >> For Centers generating NGS data which could 2-3 T.B / week depending >> on >> no. of sequencers I think importing another copy of raw data into >> galaxy >> workspace will be asking for lot of disk space. I understand it is a >> neat >> way of doing things as it becomes agnostic of the raw data location >>  but >> might not be the best way for handling huge data in long run for >> centers >> like ours. >> >> Please correct me if I am wrong. I think we could also have a simple >> option without having to import the data and just using it for >> analysis >> from >> the current location, also storing results at the same location. That >> way in >> future even if the data set is moved analysis also stays with it. >> >> Let me know what you feel. I will be happy to know if there are any >> other >> smart reasons of importing the data in galaxy workspace just for >> curiosity >> sake. >> >> Thanks, >> -Abhi >> >> On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu >> wrote: >> Hello Abhishek, >> >> The Galaxy distribution includes the enhancements to which I >> previously >> referred for uploading history files.  Uploading files to a >> history >> now >> creates a Galaxy job just like any other tool, and can be run on a >> cluster >> node, allowing upload of very large files.  The initial pass of >> this >> work is >> also completed for uploading to a Data Library, but this enhancement >> is >> still in test, so it should soon be available in the distribution. >> >> Do you want to avoid having to import at all (e.g. allow Galaxy to >> refer >> to datasets that live in their original locations)?  This is not >> currently >> possible, but if this is what you are looking for, we can consider >> some >> additional options on the current upload form, or possibly a new, >> separate >> form. >> >> >> Greg Von Kuster >> Galaxy Development Team >> >> >> Abhishek Pratap wrote: >> Hi Greg, Anton and all >> >> Just wondering if there has been any progress made on this end. I am >> sorry I was not able to follow it up on Assaf's suggestion due to >> other >> things at work. >> >> I did try the latest version of galaxy and looks like the files are >> still >> transferred over HTTP before they could be used in the galaxy >> workspace. >> Also I would again like to highlight that many labs might want to use >> the >> local instance of galaxy and prefer to point to a local path where >> the >> file >> is being stored. That way we will have both the benefits of using a >> cool GUI >> and process data stored locally. >> >> Let me know if you guys need some feedback or have more questions. I >> will >> be happy to discuss them. >> >> best, >> -Abhi >> >> On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu >> mailto:ghv2@psu.edu> wrote: >> >>   Hello Abishek, >> >>   We are currently in the process of significantly enhancing the >>   current Galaxy upload utilities, and the new version should >>   eliminate the issue you've raised about the time needed to >> upload >>   large files via HTTP ( not for making an initial copy of the >> file in >>   the Galaxy environment ). However, it will probably not be >> ready for >>   release for a few more weeks, so if you can take advantage of >>   Assaf's script in the meantime, that's great. ¨ÜI can't >> guarantee >>   that all Galaxy features will function correctly if you do this >> though. >> >>   Assaf, have you found that using your script breaks anything? >> >>   Also, if you upload a file to a library rather than a history, >>   multiple users can "import" the library dataset into their >> history >>   for analysis, but there is only 1 file on disk ( users are >> pointing >>   to it from their histories ). ¨ÜBut uploading a file to >> a history >>   will create a new copy of the file each time it is uploaded. >> >>   Greg Von Kuster >>   Galaxy Development Team >> >> >> >>   Abhishek Pratap wrote: >> >>       Hi All >> >>       @Greg : Please find my comments below. >> >>       On Tue, Jul 21, 2009 at 10:44 AM, Greg Von >> Kuster<ghv2@psu.edu >>       mailto:ghv2@psu.edu> wrote: >> >>           Hello Abhi, >> >>           Can you clarify the steps you took that >> produced the >>           behavior? ǃÜSee my >> >>           comments below. >> >>           Anton Nekrutenko wrote: >> >>               Abhishek: >> >>               Let talk. This is the area >> of active current >>               development. We are >> ǃÜlooking >> >>               at implementing a universal >> fastq-like format or >>               supporting >> ǃÜmultiple >> >>               formats. Perhaps we should >> join efforts in ironing >> out >>               >> ǃÜspecifications. >> >> >>               anton >>               galaxy team >> >> >>               On Jul 20, 2009, at 5:18 >> PM, Abhishek Pratap >> wrote: >> >>                   Hi All >> >> >>                   I recently came >> to know about NGS analysis >> on galaxy >>                   during ISMB. >>                   Getting excited >> I tried couple of things >> basically >>                   to play with >> it. >> >>                   Few comments : >> I may have interepretted >> something >>                   described below >> in a >>                   wrong way. My >> apologies before hand. >> >> >> >>                   On a standalone >> installation of galaxy while >> I was >>                   trying to >> explore >>                   one >> FASTQ(sequence) file. It takes >> considerable (> >>                   20 min) for a >> fastq >>                   file to get >> uploaded (2 GB). >> >>           Are you using the Galaxy upload utility >> to create an >> item in >>           your history >>           that points to the dataset file on >> disk? >> >> >>       Yes that is precisely correct, I am trying to >> upload a solexa >> FASTQ >>       file but on a standalone galaxy installation from >> my local >> file >>       system. >> >>           I am not sure what is the rationale >> >>                   behind that. >> Ideally I think there should be >> no need >>                   to upload such >>                   heavy files >> into the workspace. >> >>           A data file that originates from a >> place external to >> Galaxy >>           must be uploaded >>           into Galaxy so that the disk file can >> be placed in the >>           location configured >>           in the Galaxy config file. >> ǃÜAlso, when data is >> uploaded to >> >>           Galaxy ( either >>           to a history or a library ), several >> database table >> settings >>           are created >>           that are used by various Galaxy >> features. >> >>           They could actually be used straight >> >> >>       Thanks for the clarification but I am not sure this >> will help >> a >>       lot of >>       people who are interested to install and run galaxy >> locally >>       mainly for >>       the following reasons. May be it is just local to >> me. >> >>       A. We already one instance of data saved on the >> local file >> system >>       B. Making another copy via galaxy will eat away a >> lot of space >>       in long run. >>       C. The time needed to import the files into galaxy >> space is >> huge >> >>                   away by the >> path specified. >> >>           What do you mean by "the path >> specified"? >> >> >> >>       Well what I mean was a way to specify the path of >> the file/run >>       on the >>       lcoal file system and galaxy could directly pick it >> up from >> there >>       rather than uploading it into its own space. Now I >> understand >> this >>       might not work based on the way the system was >> designed. >> >> >>           Also is there any way to access the >> >>                   scripts for >> analysis on the command line. I >> know >>                   this undermines >> the >>                   main aim of >> working with galaxy but rite now >> I am >>                   concerned about >> the >>                   >> performance/time. >> >>           You should be able to run any Galaxy >> tool from the >> command >>           line as long as >>           you have all of the tool's required >> binaries in your >> path. >>           ǃÜHowever, running >> >>           a tool from within Galaxy should >> generally not be any >> slower >>           than running it >>           outside of Galaxy, depending, of >> course, on what you are >> doing. >> >> >> >> >>       Ok I was under the impression that running from >> SHELL will >> eliminate >>       the step of uploading them into galaxy file space. >> >> >>       -Abhi >> >>                   I will be happy >> to discuss more about this >> in case >>                   you have some >>                   >> comments/questions for me. >> >> >> >>                   Best, >>                   -Abhi >> >> >> >>                   >> ----------------------------- >> >>                   Abhishek Pratap >> >>                   Bioinformatics >> Software Engineer >> >>                   Institute for >> Genome Sciences >> >>                   School of >> Medicine, Univ of Maryland >> >>                   801, W. >> Baltimore Street, Baltimore, MD >> 21209 >> >>                   Ph: >> (+1)-410-706-2296 >> >>                   >> www.igs.umaryland.edu/ >> http://www.igs.umaryland.edu/ >>                   >> _______________________________________________ >>                   galaxy-user >> mailing list >>                   >> galaxy-user@bx.psu.edu >> mailto:galaxy-user@bx.psu.edu >> >> >> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >> >>               Anton Nekrutenko >>               http://nekrut.bx.psu.edu >>               http://galaxyproject.org >> >>               >> _______________________________________________ >>               galaxy-user mailing list >>               galaxy-user@bx.psu.edu >> mailto:galaxy-user@bx.psu.edu >> >>               >> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >> >> >> >> >> >> >> >> >> _______________________________________________ >> galaxy-user mailing list >> galaxy-user@bx.psu.edu >> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user > -- > Research Associate    Department of Biostatistics > Associate Director    Bioinformatics Core >                     >  Harvard School of Public Health > Skype: ohofmann       Phone: +1 (617) 365 0984 > > >
I think that could be it. I am using the "Upload File " option under the "Get Data" from the left hand menu. Sorry for the confusion if I am doing something that I am not supposed to. How do we upload data to data library ?
-Abhi
On Fri, Oct 2, 2009 at 3:46 PM, Greg Von Kuster ghv2@psu.edu wrote:
Ok, you have the change set. Perhaps you are trying to upload into a history rather than a Data Library? This new feature is only available for uploading into a Data Library. Can you confirm? I'm not sure what else could be causing you to not see the 4 options on the drop-down menu for uploading dataset into a Data Library.
Greg
Abhishek Pratap wrote:
Here it is :
hg heads
changeset: 2824:d97f4e86be45 tag: tip parent: 2823:2c0c81150dbd parent: 2821:04a753865407 user: Anton Nekrutenko anton@bx.psu.edu date: Fri Oct 02 11:02:14 2009 -0400 summary: merge
-Abhi
On Fri, Oct 2, 2009 at 3:30 PM, Greg Von Kuster ghv2@psu.edu wrote:
Please type the following in your galaxy install directory, and let me know what you get:
hg heads
Thanks
Abhishek Pratap wrote:
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets  ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote: > > Change set 2812 will be included in a release to the distribution > today > - > here are details of a new option that we're hoping will provide what > is > needed for most labs. > > Add a new option, 'allow_library_path_paste' that adds a new upload > page > ("Upload files from file system paths") to the admin-side library > upload > pages. > This form contains a textarea that allows Galaxy admins to paste any > number > of > file system paths (files or directories) from which Galaxy will > import > library > datasets, saving the directory structure (if desired). >  Since such > ability > allows admins access to any file on the Galaxy server which is > readable > by > Galaxy's system user, this option is disabled by default, and system > administrators should take care in assigning Galaxy administrators > when > this > feature is enabled.  Controls on what files are accessible > to this > tool > based > on ownership or other properties can be added at a later date if > there > is > sufficient interest for such features. > > This commit also includes a checkbox on the "Upload directory of > files" > page > (as well as the new "Upload files from file system paths" page above) > that > will > prevent Galaxy from copying data to its files directory (by default, > 'database/files/').  This is useful for large library > datasets that > live > in > their own managed locations on the file system, this will prevent the > existence > of duplicate copies of datasets (but means administrators must take > care > to > manage data - moving or removing the data from its Galaxy-external > location > will render these datasets invalid within Galaxy). > > One unique feature to be aware of: when using the "Copy data into > Galaxy?" > checkbox on the "Upload directory of files" page, any symbolic links > encountered in the chosen import directory will be made absolute and > dereferenced ONCE.  This allows administrators to link > large > datasets to > the > import directory, rather than having to make full copies, while being > able > to > delete such links after importing.  Only the first symlink > (the one > in > the > import directory itself) is dereferenced; all others remain. >  See > the > following > for an example: > > library_import_dir = /galaxy/import > > % ls -lR /galaxy/import > /galaxy/import: > total 6 > drwxr-xr-x   2 nate     nate >         512 > Oct  1 11:31 link/ > > /galaxy/import/link: > total 10 > lrwxrwxrwx   1 nate     nate >         >  71 Oct  1 10:38 1.bed -> > ../../../home/nate/galaxy/test-data/1.bed > lrwxrwxrwx   1 nate     nate >         >  60 Oct  1 10:38 2.bed -> > /home/nate/galaxy/test-data/2.bed > lrwxrwxrwx   1 nate     nate >         >  11 Oct  1 10:38 3.bed -> > ../../3.bed > lrwxrwxrwx   1 nate     nate >         >  35 Oct  1 11:30 4.bed -> > ../../galaxy_symlink/test-data/4.bed > lrwxrwxrwx   1 nate     nate >         >  41 Oct  1 11:31 5.bed -> > /galaxy/galaxy_symlink/test-data/5.bed > > % ls -l /galaxy/3.bed > lrwxrwxrwx   1 nate     nate >         >  60 Oct  1 10:39 > /galaxy/3.bed -> > /home/nate/galaxy/test-data/3.bed > > % ls -l /galaxy/galaxy_symlink > lrwxrwxrwx   1 nate     nate >         >  44 Oct  1 11:30 > /galaxy/galaxy_symlink > -> /home/nate/galaxy/ > > In this example, > > 1.bed is a relative symbolic link to the real 1.bed. > > 2.bed is an absolute symlink to the real 2.bed. > > 3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which > itself > is > a symlink to the real 3.bed. > > 4.bed is a relative symlink which follows another symlink > (/galaxy/galaxy_symlink) to the real 4.bed. > > 5.bed is an absolute symlink in the same fashion as 4.bed > > If the 'link' server directory is chosen on the "Upload directory of > files" > page, and "Copy data into Galaxy?" is checked "No", the following > files > will > be > referenced by Galaxy: > > /home/nate/galaxy/test-data/1.bed > /home/nate/galaxy/test-data/2.bed > /galaxy/3.bed > /galaxy/galaxy_symlink/test-data/4.bed > /galaxy/galaxy_symlink/test-data/5.bed > > The Galaxy administrator may now safely delete /galaxy/import/link, > but > should > take care not to remove the referenced symbolic links (/galaxy/3.bed, > /galaxy/galaxy_symlink). > > Not all symbolic links are dereferenced because it is assumed that if > an > administrator links to a path in the import directory which itself is > (or > contains) links, that is the preferred path for accessing the data. > > > > Oliver Hofmann wrote: >> >> Dear all, >> >> >> to echo what Abhi said: we are also currently looking of ways to >> automatically import data sets (libraries) into Galaxy without >> having >> to >> manually trigger the import via the administration interface, and >> ideally >> while keeping the data in the original place. The idea here is to >> have >> multiple tools all point at the original 'source data' without >> having >> to >> replicate terabytes of data. >> >> Not quite sure how feasible this is in practice, but it certainly >> would >> be >> incredibly helpful. >> >> Best, >> >>    Oliver >> >> >> >> >> On 28 Sep 2009, at 14:24, Abhishek Pratap wrote: >> >>> HI Greg >>> >>> Thanks for a quick reply and making some requested changes. However >>> I >>> am >>> not still sure if importing NGS data will help in long run. >>> >>> For Centers generating NGS data which could 2-3 T.B / week >>> depending >>> on >>> no. of sequencers I think importing another copy of raw data into >>> galaxy >>> workspace will be asking for lot of disk space. I understand it is >>> a >>> neat >>> way of doing things as it becomes agnostic of the raw data location >>>  but >>> might not be the best way for handling huge data in long run for >>> centers >>> like ours. >>> >>> Please correct me if I am wrong. I think we could also have a >>> simple >>> option without having to import the data and just using it for >>> analysis >>> from >>> the current location, also storing results at the same location. >>> That >>> way in >>> future even if the data set is moved analysis also stays with it. >>> >>> Let me know what you feel. I will be happy to know if there are any >>> other >>> smart reasons of importing the data in galaxy workspace just for >>> curiosity >>> sake. >>> >>> Thanks, >>> -Abhi >>> >>> On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu >>> wrote: >>> Hello Abhishek, >>> >>> The Galaxy distribution includes the enhancements to which I >>> previously >>> referred for uploading history files.  Uploading files >>> to a >>> history >>> now >>> creates a Galaxy job just like any other tool, and can be run on a >>> cluster >>> node, allowing upload of very large files.  The initial >>> pass of >>> this >>> work is >>> also completed for uploading to a Data Library, but this >>> enhancement >>> is >>> still in test, so it should soon be available in the distribution. >>> >>> Do you want to avoid having to import at all (e.g. allow Galaxy to >>> refer >>> to datasets that live in their original locations)? >>>  This is not >>> currently >>> possible, but if this is what you are looking for, we can consider >>> some >>> additional options on the current upload form, or possibly a new, >>> separate >>> form. >>> >>> >>> Greg Von Kuster >>> Galaxy Development Team >>> >>> >>> Abhishek Pratap wrote: >>> Hi Greg, Anton and all >>> >>> Just wondering if there has been any progress made on this end. I >>> am >>> sorry I was not able to follow it up on Assaf's suggestion due to >>> other >>> things at work. >>> >>> I did try the latest version of galaxy and looks like the files are >>> still >>> transferred over HTTP before they could be used in the galaxy >>> workspace. >>> Also I would again like to highlight that many labs might want to >>> use >>> the >>> local instance of galaxy and prefer to point to a local path where >>> the >>> file >>> is being stored. That way we will have both the benefits of using a >>> cool GUI >>> and process data stored locally. >>> >>> Let me know if you guys need some feedback or have more questions. >>> I >>> will >>> be happy to discuss them. >>> >>> best, >>> -Abhi >>> >>> On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu >>> mailto:ghv2@psu.edu> wrote: >>> >>>   Hello Abishek, >>> >>>   We are currently in the process of significantly >>> enhancing the >>>   current Galaxy upload utilities, and the new version >>> should >>>   eliminate the issue you've raised about the time >>> needed to >>> upload >>>   large files via HTTP ( not for making an initial copy >>> of the >>> file in >>>   the Galaxy environment ). However, it will probably >>> not be >>> ready for >>>   release for a few more weeks, so if you can take >>> advantage of >>>   Assaf's script in the meantime, that's great. >>> ¨ÜI can't >>> guarantee >>>   that all Galaxy features will function correctly if >>> you do this >>> though. >>> >>>   Assaf, have you found that using your script breaks >>> anything? >>> >>>   Also, if you upload a file to a library rather than a >>> history, >>>   multiple users can "import" the library dataset into >>> their >>> history >>>   for analysis, but there is only 1 file on disk ( users >>> are >>> pointing >>>   to it from their histories ). ¨ÜBut >>> uploading a file to >>> a history >>>   will create a new copy of the file each time it is >>> uploaded. >>> >>>   Greg Von Kuster >>>   Galaxy Development Team >>> >>> >>> >>>   Abhishek Pratap wrote: >>> >>>       Hi All >>> >>>       @Greg : Please find my >>> comments below. >>> >>>       On Tue, Jul 21, 2009 at >>> 10:44 AM, Greg Von >>> Kuster<ghv2@psu.edu >>>       mailto:ghv2@psu.edu> >>> wrote: >>> >>>           >>> Hello Abhi, >>> >>>           >>> Can you clarify the steps you took that >>> produced the >>>           >>> behavior? ǃÜSee my >>> >>>           >>> comments below. >>> >>>           >>> Anton Nekrutenko wrote: >>> >>>           >>>     Abhishek: >>> >>>           >>>     Let talk. This is the area >>> of active current >>>           >>>     development. We are >>> ǃÜlooking >>> >>>           >>>     at implementing a universal >>> fastq-like format or >>>           >>>     supporting >>> ǃÜmultiple >>> >>>           >>>     formats. Perhaps we should >>> join efforts in ironing >>> out >>>           >>>     >>> ǃÜspecifications. >>> >>> >>>           >>>     anton >>>           >>>     galaxy team >>> >>> >>>           >>>     On Jul 20, 2009, at 5:18 >>> PM, Abhishek Pratap >>> wrote: >>> >>>           >>>         Hi All >>> >>> >>>           >>>         I recently came >>> to know about NGS analysis >>> on galaxy >>>           >>>         during ISMB. >>>           >>>         Getting excited >>> I tried couple of things >>> basically >>>           >>>         to play with >>> it. >>> >>>           >>>         Few comments : >>> I may have interepretted >>> something >>>           >>>         described below >>> in a >>>           >>>         wrong way. My >>> apologies before hand. >>> >>> >>> >>>           >>>         On a standalone >>> installation of galaxy while >>> I was >>>           >>>         trying to >>> explore >>>           >>>         one >>> FASTQ(sequence) file. It takes >>> considerable (> >>>           >>>         20 min) for a >>> fastq >>>           >>>         file to get >>> uploaded (2 GB). >>> >>>           >>> Are you using the Galaxy upload utility >>> to create an >>> item in >>>           >>> your history >>>           >>> that points to the dataset file on >>> disk? >>> >>> >>>       Yes that is precisely >>> correct, I am trying to >>> upload a solexa >>> FASTQ >>>       file but on a standalone >>> galaxy installation from >>> my local >>> file >>>       system. >>> >>>           I >>> am not sure what is the rationale >>> >>>           >>>         behind that. >>> Ideally I think there should be >>> no need >>>           >>>         to upload such >>>           >>>         heavy files >>> into the workspace. >>> >>>           A >>> data file that originates from a >>> place external to >>> Galaxy >>>           >>> must be uploaded >>>           >>> into Galaxy so that the disk file can >>> be placed in the >>>           >>> location configured >>>           in >>> the Galaxy config file. >>> ǃÜAlso, when data is >>> uploaded to >>> >>>           >>> Galaxy ( either >>>           to >>> a history or a library ), several >>> database table >>> settings >>>           >>> are created >>>           >>> that are used by various Galaxy >>> features. >>> >>>           >>> They could actually be used straight >>> >>> >>>       Thanks for the clarification >>> but I am not sure this >>> will help >>> a >>>       lot of >>>       people who are interested to >>> install and run galaxy >>> locally >>>       mainly for >>>       the following reasons. May >>> be it is just local to >>> me. >>> >>>       A. We already one instance >>> of data saved on the >>> local file >>> system >>>       B. Making another copy via >>> galaxy will eat away a >>> lot of space >>>       in long run. >>>       C. The time needed to import >>> the files into galaxy >>> space is >>> huge >>> >>>           >>>         away by the >>> path specified. >>> >>>           >>> What do you mean by "the path >>> specified"? >>> >>> >>> >>>       Well what I mean was a way >>> to specify the path of >>> the file/run >>>       on the >>>       lcoal file system and galaxy >>> could directly pick it >>> up from >>> there >>>       rather than uploading it >>> into its own space. Now I >>> understand >>> this >>>       might not work based on the >>> way the system was >>> designed. >>> >>> >>>           >>> Also is there any way to access the >>> >>>           >>>         scripts for >>> analysis on the command line. I >>> know >>>           >>>         this undermines >>> the >>>           >>>         main aim of >>> working with galaxy but rite now >>> I am >>>           >>>         concerned about >>> the >>>           >>>         >>> performance/time. >>> >>>           >>> You should be able to run any Galaxy >>> tool from the >>> command >>>           >>> line as long as >>>           >>> you have all of the tool's required >>> binaries in your >>> path. >>>           >>> ǃÜHowever, running >>> >>>           a >>> tool from within Galaxy should >>> generally not be any >>> slower >>>           >>> than running it >>>           >>> outside of Galaxy, depending, of >>> course, on what you are >>> doing. >>> >>> >>> >>> >>>       Ok I was under the >>> impression that running from >>> SHELL will >>> eliminate >>>       the step of uploading them >>> into galaxy file space. >>> >>> >>>       -Abhi >>> >>>           >>>         I will be happy >>> to discuss more about this >>> in case >>>           >>>         you have some >>>           >>>         >>> comments/questions for me. >>> >>> >>> >>>           >>>         Best, >>>           >>>         -Abhi >>> >>> >>> >>>           >>>         >>> ----------------------------- >>> >>>           >>>         Abhishek Pratap >>> >>>           >>>         Bioinformatics >>> Software Engineer >>> >>>           >>>         Institute for >>> Genome Sciences >>> >>>           >>>         School of >>> Medicine, Univ of Maryland >>> >>>           >>>         801, W. >>> Baltimore Street, Baltimore, MD >>> 21209 >>> >>>           >>>         Ph: >>> (+1)-410-706-2296 >>> >>>           >>>         >>> www.igs.umaryland.edu/ >>> http://www.igs.umaryland.edu/ >>>           >>>         >>> _______________________________________________ >>>           >>>         galaxy-user >>> mailing list >>>           >>>         >>> galaxy-user@bx.psu.edu >>> mailto:galaxy-user@bx.psu.edu >>> >>> >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> >>>           >>>     Anton Nekrutenko >>>           >>>     http://nekrut.bx.psu.edu >>>           >>>     http://galaxyproject.org >>> >>>           >>>     >>> _______________________________________________ >>>           >>>     galaxy-user mailing list >>>           >>>     galaxy-user@bx.psu.edu >>> mailto:galaxy-user@bx.psu.edu >>> >>>           >>>     >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> galaxy-user mailing list >>> galaxy-user@bx.psu.edu >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >> >> -- >> Research Associate    Department of >> Biostatistics >> Associate Director    Bioinformatics Core >>           >>           >>  Harvard School of Public Health >> Skype: ohofmann       Phone: +1 >> (617) 365 0984 >> >> >>
Set yourself up as a Galaxy admin, in universe_wsgi.ini, add your galaxy account ( email ) to the following:
# this should be a comma-separated list of valid Galaxy users admin_users = you@youremail.edu
You'll see an Admin link in the top Galaxy menu bar when you restart your Galaxy sever after making this change. Click on it, and you be presented with teh Galaxy Admin UI. Look for the "Manage data libraries" link in th left panel, create a new data library, and you can upload datasets to using 1 of 4 options, including this latest option just introduced.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
I think that could be it. I am using the "Upload File " option under the "Get Data" from the left hand menu. Sorry for the confusion if I am doing something that I am not supposed to. How do we upload data to data library ?
-Abhi
On Fri, Oct 2, 2009 at 3:46 PM, Greg Von Kuster ghv2@psu.edu wrote:
Ok, you have the change set.  Perhaps you are trying to upload into a history rather than a Data Library?  This new feature is only available for uploading into a Data Library.  Can you confirm?  I'm not sure what else could be causing you to not see the 4 options on the drop-down menu for uploading dataset into a Data Library.
Greg
Abhishek Pratap wrote:
Here it is :
hg heads
changeset:   2824:d97f4e86be45 tag:         tip parent:      2823:2c0c81150dbd parent:      2821:04a753865407 user:        Anton Nekrutenko anton@bx.psu.edu date:        Fri Oct 02 11:02:14 2009 -0400 summary:     merge
-Abhi
On Fri, Oct 2, 2009 at 3:30 PM, Greg Von Kuster ghv2@psu.edu wrote:
Please type the following in your galaxy install directory, and let me know what you get:
hg heads
Thanks
Abhishek Pratap wrote:
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets  ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: > Hi Greg > > I have updated my galaxy rep to changeset 2825. I dont see the > checkbox on the "Upload File" page. Am I missing something ? > > Thanks, > -Abhi > > On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster ghv2@psu.edu wrote: >> Change set 2812 will be included in a release to the distribution >> today >> - >> here are details of a new option that we're hoping will provide what >> is >> needed for most labs. >> >> Add a new option, 'allow_library_path_paste' that adds a new upload >> page >> ("Upload files from file system paths") to the admin-side library >> upload >> pages. >> This form contains a textarea that allows Galaxy admins to paste any >> number >> of >> file system paths (files or directories) from which Galaxy will >> import >> library >> datasets, saving the directory structure (if desired). >>  Since such >> ability >> allows admins access to any file on the Galaxy server which is >> readable >> by >> Galaxy's system user, this option is disabled by default, and system >> administrators should take care in assigning Galaxy administrators >> when >> this >> feature is enabled.  Controls on what files are accessible >> to this >> tool >> based >> on ownership or other properties can be added at a later date if >> there >> is >> sufficient interest for such features. >> >> This commit also includes a checkbox on the "Upload directory of >> files" >> page >> (as well as the new "Upload files from file system paths" page above) >> that >> will >> prevent Galaxy from copying data to its files directory (by default, >> 'database/files/').  This is useful for large library >> datasets that >> live >> in >> their own managed locations on the file system, this will prevent the >> existence >> of duplicate copies of datasets (but means administrators must take >> care >> to >> manage data - moving or removing the data from its Galaxy-external >> location >> will render these datasets invalid within Galaxy). >> >> One unique feature to be aware of: when using the "Copy data into >> Galaxy?" >> checkbox on the "Upload directory of files" page, any symbolic links >> encountered in the chosen import directory will be made absolute and >> dereferenced ONCE.  This allows administrators to link >> large >> datasets to >> the >> import directory, rather than having to make full copies, while being >> able >> to >> delete such links after importing.  Only the first symlink >> (the one >> in >> the >> import directory itself) is dereferenced; all others remain. >>  See >> the >> following >> for an example: >> >> library_import_dir = /galaxy/import >> >> % ls -lR /galaxy/import >> /galaxy/import: >> total 6 >> drwxr-xr-x   2 nate     nate >>         512 >> Oct  1 11:31 link/ >> >> /galaxy/import/link: >> total 10 >> lrwxrwxrwx   1 nate     nate >>         >>  71 Oct  1 10:38 1.bed -> >> ../../../home/nate/galaxy/test-data/1.bed >> lrwxrwxrwx   1 nate     nate >>         >>  60 Oct  1 10:38 2.bed -> >> /home/nate/galaxy/test-data/2.bed >> lrwxrwxrwx   1 nate     nate >>         >>  11 Oct  1 10:38 3.bed -> >> ../../3.bed >> lrwxrwxrwx   1 nate     nate >>         >>  35 Oct  1 11:30 4.bed -> >> ../../galaxy_symlink/test-data/4.bed >> lrwxrwxrwx   1 nate     nate >>         >>  41 Oct  1 11:31 5.bed -> >> /galaxy/galaxy_symlink/test-data/5.bed >> >> % ls -l /galaxy/3.bed >> lrwxrwxrwx   1 nate     nate >>         >>  60 Oct  1 10:39 >> /galaxy/3.bed -> >> /home/nate/galaxy/test-data/3.bed >> >> % ls -l /galaxy/galaxy_symlink >> lrwxrwxrwx   1 nate     nate >>         >>  44 Oct  1 11:30 >> /galaxy/galaxy_symlink >> -> /home/nate/galaxy/ >> >> In this example, >> >> 1.bed is a relative symbolic link to the real 1.bed. >> >> 2.bed is an absolute symlink to the real 2.bed. >> >> 3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which >> itself >> is >> a symlink to the real 3.bed. >> >> 4.bed is a relative symlink which follows another symlink >> (/galaxy/galaxy_symlink) to the real 4.bed. >> >> 5.bed is an absolute symlink in the same fashion as 4.bed >> >> If the 'link' server directory is chosen on the "Upload directory of >> files" >> page, and "Copy data into Galaxy?" is checked "No", the following >> files >> will >> be >> referenced by Galaxy: >> >> /home/nate/galaxy/test-data/1.bed >> /home/nate/galaxy/test-data/2.bed >> /galaxy/3.bed >> /galaxy/galaxy_symlink/test-data/4.bed >> /galaxy/galaxy_symlink/test-data/5.bed >> >> The Galaxy administrator may now safely delete /galaxy/import/link, >> but >> should >> take care not to remove the referenced symbolic links (/galaxy/3.bed, >> /galaxy/galaxy_symlink). >> >> Not all symbolic links are dereferenced because it is assumed that if >> an >> administrator links to a path in the import directory which itself is >> (or >> contains) links, that is the preferred path for accessing the data. >> >> >> >> Oliver Hofmann wrote: >>> Dear all, >>> >>> >>> to echo what Abhi said: we are also currently looking of ways to >>> automatically import data sets (libraries) into Galaxy without >>> having >>> to >>> manually trigger the import via the administration interface, and >>> ideally >>> while keeping the data in the original place. The idea here is to >>> have >>> multiple tools all point at the original 'source data' without >>> having >>> to >>> replicate terabytes of data. >>> >>> Not quite sure how feasible this is in practice, but it certainly >>> would >>> be >>> incredibly helpful. >>> >>> Best, >>> >>>    Oliver >>> >>> >>> >>> >>> On 28 Sep 2009, at 14:24, Abhishek Pratap wrote: >>> >>>> HI Greg >>>> >>>> Thanks for a quick reply and making some requested changes. However >>>> I >>>> am >>>> not still sure if importing NGS data will help in long run. >>>> >>>> For Centers generating NGS data which could 2-3 T.B / week >>>> depending >>>> on >>>> no. of sequencers I think importing another copy of raw data into >>>> galaxy >>>> workspace will be asking for lot of disk space. I understand it is >>>> a >>>> neat >>>> way of doing things as it becomes agnostic of the raw data location >>>>  but >>>> might not be the best way for handling huge data in long run for >>>> centers >>>> like ours. >>>> >>>> Please correct me if I am wrong. I think we could also have a >>>> simple >>>> option without having to import the data and just using it for >>>> analysis >>>> from >>>> the current location, also storing results at the same location. >>>> That >>>> way in >>>> future even if the data set is moved analysis also stays with it. >>>> >>>> Let me know what you feel. I will be happy to know if there are any >>>> other >>>> smart reasons of importing the data in galaxy workspace just for >>>> curiosity >>>> sake. >>>> >>>> Thanks, >>>> -Abhi >>>> >>>> On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu >>>> wrote: >>>> Hello Abhishek, >>>> >>>> The Galaxy distribution includes the enhancements to which I >>>> previously >>>> referred for uploading history files.  Uploading files >>>> to a >>>> history >>>> now >>>> creates a Galaxy job just like any other tool, and can be run on a >>>> cluster >>>> node, allowing upload of very large files.  The initial >>>> pass of >>>> this >>>> work is >>>> also completed for uploading to a Data Library, but this >>>> enhancement >>>> is >>>> still in test, so it should soon be available in the distribution. >>>> >>>> Do you want to avoid having to import at all (e.g. allow Galaxy to >>>> refer >>>> to datasets that live in their original locations)? >>>>  This is not >>>> currently >>>> possible, but if this is what you are looking for, we can consider >>>> some >>>> additional options on the current upload form, or possibly a new, >>>> separate >>>> form. >>>> >>>> >>>> Greg Von Kuster >>>> Galaxy Development Team >>>> >>>> >>>> Abhishek Pratap wrote: >>>> Hi Greg, Anton and all >>>> >>>> Just wondering if there has been any progress made on this end. I >>>> am >>>> sorry I was not able to follow it up on Assaf's suggestion due to >>>> other >>>> things at work. >>>> >>>> I did try the latest version of galaxy and looks like the files are >>>> still >>>> transferred over HTTP before they could be used in the galaxy >>>> workspace. >>>> Also I would again like to highlight that many labs might want to >>>> use >>>> the >>>> local instance of galaxy and prefer to point to a local path where >>>> the >>>> file >>>> is being stored. That way we will have both the benefits of using a >>>> cool GUI >>>> and process data stored locally. >>>> >>>> Let me know if you guys need some feedback or have more questions. >>>> I >>>> will >>>> be happy to discuss them. >>>> >>>> best, >>>> -Abhi >>>> >>>> On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu >>>> mailto:ghv2@psu.edu> wrote: >>>> >>>>   Hello Abishek, >>>> >>>>   We are currently in the process of significantly >>>> enhancing the >>>>   current Galaxy upload utilities, and the new version >>>> should >>>>   eliminate the issue you've raised about the time >>>> needed to >>>> upload >>>>   large files via HTTP ( not for making an initial copy >>>> of the >>>> file in >>>>   the Galaxy environment ). However, it will probably >>>> not be >>>> ready for >>>>   release for a few more weeks, so if you can take >>>> advantage of >>>>   Assaf's script in the meantime, that's great. >>>> ¨ÜI can't >>>> guarantee >>>>   that all Galaxy features will function correctly if >>>> you do this >>>> though. >>>> >>>>   Assaf, have you found that using your script breaks >>>> anything? >>>> >>>>   Also, if you upload a file to a library rather than a >>>> history, >>>>   multiple users can "import" the library dataset into >>>> their >>>> history >>>>   for analysis, but there is only 1 file on disk ( users >>>> are >>>> pointing >>>>   to it from their histories ). ¨ÜBut >>>> uploading a file to >>>> a history >>>>   will create a new copy of the file each time it is >>>> uploaded. >>>> >>>>   Greg Von Kuster >>>>   Galaxy Development Team >>>> >>>> >>>> >>>>   Abhishek Pratap wrote: >>>> >>>>       Hi All >>>> >>>>       @Greg : Please find my >>>> comments below. >>>> >>>>       On Tue, Jul 21, 2009 at >>>> 10:44 AM, Greg Von >>>> Kuster<ghv2@psu.edu >>>>       mailto:ghv2@psu.edu> >>>> wrote: >>>> >>>>           >>>> Hello Abhi, >>>> >>>>           >>>> Can you clarify the steps you took that >>>> produced the >>>>           >>>> behavior? ǃÜSee my >>>> >>>>           >>>> comments below. >>>> >>>>           >>>> Anton Nekrutenko wrote: >>>> >>>>           >>>>     Abhishek: >>>> >>>>           >>>>     Let talk. This is the area >>>> of active current >>>>           >>>>     development. We are >>>> ǃÜlooking >>>> >>>>           >>>>     at implementing a universal >>>> fastq-like format or >>>>           >>>>     supporting >>>> ǃÜmultiple >>>> >>>>           >>>>     formats. Perhaps we should >>>> join efforts in ironing >>>> out >>>>           >>>>     >>>> ǃÜspecifications. >>>> >>>> >>>>           >>>>     anton >>>>           >>>>     galaxy team >>>> >>>> >>>>           >>>>     On Jul 20, 2009, at 5:18 >>>> PM, Abhishek Pratap >>>> wrote: >>>> >>>>           >>>>         Hi All >>>> >>>> >>>>           >>>>         I recently came >>>> to know about NGS analysis >>>> on galaxy >>>>           >>>>         during ISMB. >>>>           >>>>         Getting excited >>>> I tried couple of things >>>> basically >>>>           >>>>         to play with >>>> it. >>>> >>>>           >>>>         Few comments : >>>> I may have interepretted >>>> something >>>>           >>>>         described below >>>> in a >>>>           >>>>         wrong way. My >>>> apologies before hand. >>>> >>>> >>>> >>>>           >>>>         On a standalone >>>> installation of galaxy while >>>> I was >>>>           >>>>         trying to >>>> explore >>>>           >>>>         one >>>> FASTQ(sequence) file. It takes >>>> considerable (> >>>>           >>>>         20 min) for a >>>> fastq >>>>           >>>>         file to get >>>> uploaded (2 GB). >>>> >>>>           >>>> Are you using the Galaxy upload utility >>>> to create an >>>> item in >>>>           >>>> your history >>>>           >>>> that points to the dataset file on >>>> disk? >>>> >>>> >>>>       Yes that is precisely >>>> correct, I am trying to >>>> upload a solexa >>>> FASTQ >>>>       file but on a standalone >>>> galaxy installation from >>>> my local >>>> file >>>>       system. >>>> >>>>           I >>>> am not sure what is the rationale >>>> >>>>           >>>>         behind that. >>>> Ideally I think there should be >>>> no need >>>>           >>>>         to upload such >>>>           >>>>         heavy files >>>> into the workspace. >>>> >>>>           A >>>> data file that originates from a >>>> place external to >>>> Galaxy >>>>           >>>> must be uploaded >>>>           >>>> into Galaxy so that the disk file can >>>> be placed in the >>>>           >>>> location configured >>>>           in >>>> the Galaxy config file. >>>> ǃÜAlso, when data is >>>> uploaded to >>>> >>>>           >>>> Galaxy ( either >>>>           to >>>> a history or a library ), several >>>> database table >>>> settings >>>>           >>>> are created >>>>           >>>> that are used by various Galaxy >>>> features. >>>> >>>>           >>>> They could actually be used straight >>>> >>>> >>>>       Thanks for the clarification >>>> but I am not sure this >>>> will help >>>> a >>>>       lot of >>>>       people who are interested to >>>> install and run galaxy >>>> locally >>>>       mainly for >>>>       the following reasons. May >>>> be it is just local to >>>> me. >>>> >>>>       A. We already one instance >>>> of data saved on the >>>> local file >>>> system >>>>       B. Making another copy via >>>> galaxy will eat away a >>>> lot of space >>>>       in long run. >>>>       C. The time needed to import >>>> the files into galaxy >>>> space is >>>> huge >>>> >>>>           >>>>         away by the >>>> path specified. >>>> >>>>           >>>> What do you mean by "the path >>>> specified"? >>>> >>>> >>>> >>>>       Well what I mean was a way >>>> to specify the path of >>>> the file/run >>>>       on the >>>>       lcoal file system and galaxy >>>> could directly pick it >>>> up from >>>> there >>>>       rather than uploading it >>>> into its own space. Now I >>>> understand >>>> this >>>>       might not work based on the >>>> way the system was >>>> designed. >>>> >>>> >>>>           >>>> Also is there any way to access the >>>> >>>>           >>>>         scripts for >>>> analysis on the command line. I >>>> know >>>>           >>>>         this undermines >>>> the >>>>           >>>>         main aim of >>>> working with galaxy but rite now >>>> I am >>>>           >>>>         concerned about >>>> the >>>>           >>>>         >>>> performance/time. >>>> >>>>           >>>> You should be able to run any Galaxy >>>> tool from the >>>> command >>>>           >>>> line as long as >>>>           >>>> you have all of the tool's required >>>> binaries in your >>>> path. >>>>           >>>> ǃÜHowever, running >>>> >>>>           a >>>> tool from within Galaxy should >>>> generally not be any >>>> slower >>>>           >>>> than running it >>>>           >>>> outside of Galaxy, depending, of >>>> course, on what you are >>>> doing. >>>> >>>> >>>> >>>> >>>>       Ok I was under the >>>> impression that running from >>>> SHELL will >>>> eliminate >>>>       the step of uploading them >>>> into galaxy file space. >>>> >>>> >>>>       -Abhi >>>> >>>>           >>>>         I will be happy >>>> to discuss more about this >>>> in case >>>>           >>>>         you have some >>>>           >>>>         >>>> comments/questions for me. >>>> >>>> >>>> >>>>           >>>>         Best, >>>>           >>>>         -Abhi >>>> >>>> >>>> >>>>           >>>>         >>>> ----------------------------- >>>> >>>>           >>>>         Abhishek Pratap >>>> >>>>           >>>>         Bioinformatics >>>> Software Engineer >>>> >>>>           >>>>         Institute for >>>> Genome Sciences >>>> >>>>           >>>>         School of >>>> Medicine, Univ of Maryland >>>> >>>>           >>>>         801, W. >>>> Baltimore Street, Baltimore, MD >>>> 21209 >>>> >>>>           >>>>         Ph: >>>> (+1)-410-706-2296 >>>> >>>>           >>>>         >>>> www.igs.umaryland.edu/ >>>> http://www.igs.umaryland.edu/ >>>>           >>>>         >>>> _______________________________________________ >>>>           >>>>         galaxy-user >>>> mailing list >>>>           >>>>         >>>> galaxy-user@bx.psu.edu >>>> mailto:galaxy-user@bx.psu.edu >>>> >>>> >>>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>>> >>>>           >>>>     Anton Nekrutenko >>>>           >>>>     http://nekrut.bx.psu.edu >>>>           >>>>     http://galaxyproject.org >>>> >>>>           >>>>     >>>> _______________________________________________ >>>>           >>>>     galaxy-user mailing list >>>>           >>>>     galaxy-user@bx.psu.edu >>>> mailto:galaxy-user@bx.psu.edu >>>> >>>>           >>>>     >>>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> galaxy-user mailing list >>>> galaxy-user@bx.psu.edu >>>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> -- >>> Research Associate    Department of >>> Biostatistics >>> Associate Director    Bioinformatics Core >>>           >>>           >>>  Harvard School of Public Health >>> Skype: ohofmann       Phone: +1 >>> (617) 365 0984 >>> >>> >>>
Hi!
One more question according change set 2812:
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages.
I am not sure where to add this option- i added to universe_wsgi.ini:
allow_library_path_paste = True allow_library_path_paste = /somepath/
both didnt work- how can i enable the "allow_library_path_paste" option?
thanks!
mat
Greg Von Kuster schrieb:
Change set 2812 will be included in a release to the distribution today
- here are details of a new option that we're hoping will provide what
is needed for most labs.
Add a new option, 'allow_library_path_paste' that adds a new upload page ("Upload files from file system paths") to the admin-side library upload pages. This form contains a textarea that allows Galaxy admins to paste any number of file system paths (files or directories) from which Galaxy will import library datasets, saving the directory structure (if desired). Since such ability allows admins access to any file on the Galaxy server which is readable by Galaxy's system user, this option is disabled by default, and system administrators should take care in assigning Galaxy administrators when this feature is enabled. Controls on what files are accessible to this tool based on ownership or other properties can be added at a later date if there is sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page (as well as the new "Upload files from file system paths" page above) that will prevent Galaxy from copying data to its files directory (by default, 'database/files/'). This is useful for large library datasets that live in their own managed locations on the file system, this will prevent the existence of duplicate copies of datasets (but means administrators must take care to manage data - moving or removing the data from its Galaxy-external location will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?" checkbox on the "Upload directory of files" page, any symbolic links encountered in the chosen import directory will be made absolute and dereferenced ONCE. This allows administrators to link large datasets to the import directory, rather than having to make full copies, while being able to delete such links after importing. Only the first symlink (the one in the import directory itself) is dereferenced; all others remain. See the following for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import /galaxy/import: total 6 drwxr-xr-x 2 nate nate 512 Oct 1 11:31 link/
/galaxy/import/link: total 10 lrwxrwxrwx 1 nate nate 71 Oct 1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed lrwxrwxrwx 1 nate nate 11 Oct 1 10:38 3.bed -> ../../3.bed lrwxrwxrwx 1 nate nate 35 Oct 1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed lrwxrwxrwx 1 nate nate 41 Oct 1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed lrwxrwxrwx 1 nate nate 60 Oct 1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink lrwxrwxrwx 1 nate nate 44 Oct 1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink (/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files" page, and "Copy data into Galaxy?" is checked "No", the following files will be referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed /home/nate/galaxy/test-data/2.bed /galaxy/3.bed /galaxy/galaxy_symlink/test-data/4.bed /galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should take care not to remove the referenced symbolic links (/galaxy/3.bed, /galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an administrator links to a path in the import directory which itself is (or contains) links, that is the preferred path for accessing the data.
Oliver Hofmann wrote:
Dear all,
to echo what Abhi said: we are also currently looking of ways to automatically import data sets (libraries) into Galaxy without having to manually trigger the import via the administration interface, and ideally while keeping the data in the original place. The idea here is to have multiple tools all point at the original 'source data' without having to replicate terabytes of data.
Not quite sure how feasible this is in practice, but it certainly would be incredibly helpful.
Best,
OliverOn 28 Sep 2009, at 14:24, Abhishek Pratap wrote:
HI Greg
Thanks for a quick reply and making some requested changes. However I am not still sure if importing NGS data will help in long run.
For Centers generating NGS data which could 2-3 T.B / week depending on no. of sequencers I think importing another copy of raw data into galaxy workspace will be asking for lot of disk space. I understand it is a neat way of doing things as it becomes agnostic of the raw data location but might not be the best way for handling huge data in long run for centers like ours.
Please correct me if I am wrong. I think we could also have a simple option without having to import the data and just using it for analysis from the current location, also storing results at the same location. That way in future even if the data set is moved analysis also stays with it.
Let me know what you feel. I will be happy to know if there are any other smart reasons of importing the data in galaxy workspace just for curiosity sake.
Thanks, -Abhi
On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster ghv2@psu.edu wrote: Hello Abhishek,
The Galaxy distribution includes the enhancements to which I previously referred for uploading history files. Uploading files to a history now creates a Galaxy job just like any other tool, and can be run on a cluster node, allowing upload of very large files. The initial pass of this work is also completed for uploading to a Data Library, but this enhancement is still in test, so it should soon be available in the distribution.
Do you want to avoid having to import at all (e.g. allow Galaxy to refer to datasets that live in their original locations)? This is not currently possible, but if this is what you are looking for, we can consider some additional options on the current upload form, or possibly a new, separate form.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote: Hi Greg, Anton and all
Just wondering if there has been any progress made on this end. I am sorry I was not able to follow it up on Assaf's suggestion due to other things at work.
I did try the latest version of galaxy and looks like the files are still transferred over HTTP before they could be used in the galaxy workspace. Also I would again like to highlight that many labs might want to use the local instance of galaxy and prefer to point to a local path where the file is being stored. That way we will have both the benefits of using a cool GUI and process data stored locally.
Let me know if you guys need some feedback or have more questions. I will be happy to discuss them.
best, -Abhi
On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu mailto:ghv2@psu.edu> wrote:
Hello Abishek,
We are currently in the process of significantly enhancing the current Galaxy upload utilities, and the new version should eliminate the issue you've raised about the time needed to upload large files via HTTP ( not for making an initial copy of the file in the Galaxy environment ). However, it will probably not be ready for release for a few more weeks, so if you can take advantage of Assaf's script in the meantime, that's great. ¨ÜI can't guarantee that all Galaxy features will function correctly if you do this though.
Assaf, have you found that using your script breaks anything?
Also, if you upload a file to a library rather than a history, multiple users can "import" the library dataset into their history for analysis, but there is only 1 file on disk ( users are pointing to it from their histories ). ¨ÜBut uploading a file to a history will create a new copy of the file each time it is uploaded.
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi All @Greg : Please find my comments below. On Tue, Jul 21, 2009 at 10:44 AM, Greg Von Kuster<ghv2@psu.edu <mailto:ghv2@psu.edu>> wrote: Hello Abhi, Can you clarify the steps you took that produced the behavior? ǃÜSee my comments below. Anton Nekrutenko wrote: Abhishek: Let talk. This is the area of active current development. We are ǃÜlooking at implementing a universal fastq-like format or supporting ǃÜmultiple formats. Perhaps we should join efforts in ironing out ǃÜspecifications. anton galaxy team On Jul 20, 2009, at 5:18 PM, Abhishek Pratap wrote: Hi All I recently came to know about NGS analysis on galaxy during ISMB. Getting excited I tried couple of things basically to play with it. Few comments : I may have interepretted something described below in a wrong way. My apologies before hand. On a standalone installation of galaxy while I was trying to explore one FASTQ(sequence) file. It takes considerable (> 20 min) for a fastq file to get uploaded (2 GB). Are you using the Galaxy upload utility to create an item in your history that points to the dataset file on disk? Yes that is precisely correct, I am trying to upload a solexaFASTQ file but on a standalone galaxy installation from my local file system.
I am not sure what is the rationale behind that. Ideally I think there should be no need to upload such heavy files into the workspace. A data file that originates from a place external to Galaxy must be uploaded into Galaxy so that the disk file can be placed in the location configured in the Galaxy config file. ǃÜAlso, when data is uploaded to Galaxy ( either to a history or a library ), several database table settings are created that are used by various Galaxy features. They could actually be used straight Thanks for the clarification but I am not sure this will help a lot of people who are interested to install and run galaxy locally mainly for the following reasons. May be it is just local to me. A. We already one instance of data saved on the local file system B. Making another copy via galaxy will eat away a lot of space in long run. C. The time needed to import the files into galaxy space is huge away by the path specified. What do you mean by "the path specified"? Well what I mean was a way to specify the path of the file/run on the lcoal file system and galaxy could directly pick it up from there rather than uploading it into its own space. Now I understand this might not work based on the way the system was designed. Also is there any way to access the scripts for analysis on the command line. I know this undermines the main aim of working with galaxy but rite now I am concerned about the performance/time. You should be able to run any Galaxy tool from the command line as long as you have all of the tool's required binaries in your path. ǃÜHowever, running a tool from within Galaxy should generally not be any slower than running it outside of Galaxy, depending, of course, on what you aredoing.
Ok I was under the impression that running from SHELL willeliminate the step of uploading them into galaxy file space.
-Abhi I will be happy to discuss more about this in case you have some comments/questions for me. Best, -Abhi ----------------------------- Abhishek Pratap Bioinformatics Software Engineer Institute for Genome Sciences School of Medicine, Univ of Maryland 801, W. Baltimore Street, Baltimore, MD 21209 Ph: (+1)-410-706-2296 www.igs.umaryland.edu/ <http://www.igs.umaryland.edu/> _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu>http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu <mailto:galaxy-user@bx.psu.edu>http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- Research Associate Department of Biostatistics Associate Director Bioinformatics Core Harvard School of Public Health Skype: ohofmann Phone: +1 (617) 365 0984
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Good Afternoon Anton:
I've written a fastq parser here. As I did that I discovered that existing fastq formats are ambiguous and not really identifiable from their data. Are you thinking of making a new fastq specification ?
--Hiram - UCSC genome browser
Anton Nekrutenko wrote:
Let talk. This is the area of active current development. We are looking at implementing a universal fastq-like format or supporting multiple formats. Perhaps we should join efforts in ironing out specifications.
anton galaxy team
Hiram:
Can you share the code? Which baseQ scaling are you using (Sanger| Solexa)? SOliD support?
Thanks,
anton
On Jul 21, 2009, at 4:35 PM, Hiram Clawson wrote:
I've written a fastq parser here.
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
kent source tree directories: src/utils/fastqToFa/ and src/utils/faToFastq/
I've never used the faToFastq program. The fastqToFa takes arguments to decide the baseQ score scheme. See usage message below. I haven't tested the -solexa scoring method option.
--Hiram
fastqToFa - Convert from fastq to fasta format. usage: fastqToFa [options] in.fastq out.fa options: -nameVerify='string' - for multi-line fastq files, 'string' must match somewhere in the sequence names in order to correctly identify the next sequence block (e.g.: -nameVerify='Supercontig_') -qual=file.qual.fa - output quality scores to specifed file (default: quality scores are ignored) -qualSizes=qual.sizes - write sizes file for the quality scores -noErrors - warn only on problems, do not error out (specify -verbose=3 to see warnings -solexa - use Solexa/Illumina quality score algorithm (instead of Phread quality) -verbose=2 - set warning level to get some stats output during processing
Anton Nekrutenko wrote:
Hiram:
Can you share the code? Which baseQ scaling are you using (Sanger|Solexa)? SOliD support?
Thanks,
anton
On Jul 21, 2009, at 4:35 PM, Hiram Clawson wrote:
I've written a fastq parser here.
Dr. Nekrutenko,
I've found the fq_all2std.pl script distributed with Maq to be very helpful for converting file formats to a standardized Fastq. Because it's so small (10K) I'll attach it to this email for anyone interested. Also, I'll copy its usage notes here so you can see the formats it converts:
Usage: fq_all2std.pl <command> <in.txt>
Command: scarf2std Convert SCARF format to the standard/Sanger FASTQ fqint2std Convert FASTQ-int format to the standard/Sanger F ASTQ sol2std Convert Solexa/Illumina FASTQ to the standard FAS TQ std2sol Convert standard FASTQ to Solexa/Illumina FASTQ ( simplified) fa2std Convert FASTA to the standard FASTQ seqprb2std Convert .seq and .prb files to the standard FASTQ fq2fa Convert various FASTQ-like format to FASTA export2sol Convert Solexa export format to Solexa FASTQ export2std Convert Solexa export format to Sanger FASTQ csfa2std Convert AB SOLiD read format to Sanger FASTQ std2qual Convert standard FASTQ to .seq+.qual instruction Explanation to different format example Show examples of various formats
Note: Read/quality sequences MUST be presented in one line.
John Obenauer
On Tuesday 21 July 2009 03:39:16 pm Anton Nekrutenko wrote:
Hiram:
Can you share the code? Which baseQ scaling are you using (Sanger| Solexa)? SOliD support?
Thanks,
anton
On Jul 21, 2009, at 4:35 PM, Hiram Clawson wrote:
I've written a fastq parser here.
Anton Nekrutenko http://nekrut.bx.psu.edu http://galaxyproject.org
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
-- John Obenauer, Ph.D. Bioinformatics Group Leader Information Sciences Department St. Jude Children's Research Hospital 262 Danny Thomas Place Mail Stop 312 Memphis, TN 38105 (901) 595-3188 john.obenauer@stjude.org
Email Disclaimer: www.stjude.org/emaildisclaimer
galaxy-user@lists.galaxyproject.org
-
Abhishek Pratap -
Anton Nekrutenko -
Greg Von Kuster -
Hiram Clawson -
Ido M. Tamir -
John Obenauer -
Matthias Dodt -
Oliver Hofmann