I think that could be it. I am using the "Upload File " option under the "Get Data" from the left hand menu. Sorry for the confusion if I am doing something that I am not supposed to. How do we upload data to data library ? -Abhi On Fri, Oct 2, 2009 at 3:46 PM, Greg Von Kuster <ghv2@psu.edu> wrote:
Ok, you have the change set. Perhaps you are trying to upload into a history rather than a Data Library? This new feature is only available for uploading into a Data Library. Can you confirm? I'm not sure what else could be causing you to not see the 4 options on the drop-down menu for uploading dataset into a Data Library.
Greg
Abhishek Pratap wrote:
Here it is :
hg heads
changeset: 2824:d97f4e86be45 tag: tip parent: 2823:2c0c81150dbd parent: 2821:04a753865407 user: Anton Nekrutenko <anton@bx.psu.edu> date: Fri Oct 02 11:02:14 2009 -0400 summary: merge
-Abhi
On Fri, Oct 2, 2009 at 3:30 PM, Greg Von Kuster <ghv2@psu.edu> wrote:
Please type the following in your galaxy install directory, and let me know what you get:
hg heads
Thanks
Abhishek Pratap wrote:
Hi Greg
Unfortunately it is not working for me. I made sure I cleared my browser cache before re-viewing it.
I have set the option as suggested by you in the universe_wsgi.ini file.
-Abhi
On Fri, Oct 2, 2009 at 2:53 PM, Greg Von Kuster <ghv2@psu.edu> wrote:
Hello Abhishek,
Add this to your universe_wsgi.ini file:
allow_library_path_paste = True
Then, clicking the down-arrow on the upload form
Create new data library datasets  ▼
will give you 4 options, 1 of which is:
Upload files from file system paths
Greg Von Kuster Galaxy Development Team
Abhishek Pratap wrote:
Hi Greg
I have updated my galaxy rep to changeset 2825. I dont see the checkbox on the "Upload File" page. Am I missing something ?
Thanks, -Abhi
On Fri, Oct 2, 2009 at 10:21 AM, Greg Von Kuster <ghv2@psu.edu> wrote: > > Change set 2812 will be included in a release to the distribution > today > - > here are details of a new option that we're hoping will provide what > is > needed for most labs. > > Add a new option, 'allow_library_path_paste' that adds a new upload > page > ("Upload files from file system paths") to the admin-side library > upload > pages. > This form contains a textarea that allows Galaxy admins to paste any > number > of > file system paths (files or directories) from which Galaxy will > import > library > datasets, saving the directory structure (if desired). >  Since such > ability > allows admins access to any file on the Galaxy server which is > readable > by > Galaxy's system user, this option is disabled by default, and system > administrators should take care in assigning Galaxy administrators > when > this > feature is enabled.  Controls on what files are accessible > to this > tool > based > on ownership or other properties can be added at a later date if > there > is > sufficient interest for such features. > > This commit also includes a checkbox on the "Upload directory of > files" > page > (as well as the new "Upload files from file system paths" page above) > that > will > prevent Galaxy from copying data to its files directory (by default, > 'database/files/').  This is useful for large library > datasets that > live > in > their own managed locations on the file system, this will prevent the > existence > of duplicate copies of datasets (but means administrators must take > care > to > manage data - moving or removing the data from its Galaxy-external > location > will render these datasets invalid within Galaxy). > > One unique feature to be aware of: when using the "Copy data into > Galaxy?" > checkbox on the "Upload directory of files" page, any symbolic links > encountered in the chosen import directory will be made absolute and > dereferenced ONCE.  This allows administrators to link > large > datasets to > the > import directory, rather than having to make full copies, while being > able > to > delete such links after importing.  Only the first symlink > (the one > in > the > import directory itself) is dereferenced; all others remain. >  See > the > following > for an example: > > library_import_dir = /galaxy/import > > % ls -lR /galaxy/import > /galaxy/import: > total 6 > drwxr-xr-x   2 nate     nate >         512 > Oct  1 11:31 link/ > > /galaxy/import/link: > total 10 > lrwxrwxrwx   1 nate     nate >         >  71 Oct  1 10:38 1.bed -> > ../../../home/nate/galaxy/test-data/1.bed > lrwxrwxrwx   1 nate     nate >         >  60 Oct  1 10:38 2.bed -> > /home/nate/galaxy/test-data/2.bed > lrwxrwxrwx   1 nate     nate >         >  11 Oct  1 10:38 3.bed -> > ../../3.bed > lrwxrwxrwx   1 nate     nate >         >  35 Oct  1 11:30 4.bed -> > ../../galaxy_symlink/test-data/4.bed > lrwxrwxrwx   1 nate     nate >         >  41 Oct  1 11:31 5.bed -> > /galaxy/galaxy_symlink/test-data/5.bed > > % ls -l /galaxy/3.bed > lrwxrwxrwx   1 nate     nate >         >  60 Oct  1 10:39 > /galaxy/3.bed -> > /home/nate/galaxy/test-data/3.bed > > % ls -l /galaxy/galaxy_symlink > lrwxrwxrwx   1 nate     nate >         >  44 Oct  1 11:30 > /galaxy/galaxy_symlink > -> /home/nate/galaxy/ > > In this example, > > 1.bed is a relative symbolic link to the real 1.bed. > > 2.bed is an absolute symlink to the real 2.bed. > > 3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which > itself > is > a symlink to the real 3.bed. > > 4.bed is a relative symlink which follows another symlink > (/galaxy/galaxy_symlink) to the real 4.bed. > > 5.bed is an absolute symlink in the same fashion as 4.bed > > If the 'link' server directory is chosen on the "Upload directory of > files" > page, and "Copy data into Galaxy?" is checked "No", the following > files > will > be > referenced by Galaxy: > > /home/nate/galaxy/test-data/1.bed > /home/nate/galaxy/test-data/2.bed > /galaxy/3.bed > /galaxy/galaxy_symlink/test-data/4.bed > /galaxy/galaxy_symlink/test-data/5.bed > > The Galaxy administrator may now safely delete /galaxy/import/link, > but > should > take care not to remove the referenced symbolic links (/galaxy/3.bed, > /galaxy/galaxy_symlink). > > Not all symbolic links are dereferenced because it is assumed that if > an > administrator links to a path in the import directory which itself is > (or > contains) links, that is the preferred path for accessing the data. > > > > Oliver Hofmann wrote: >> >> Dear all, >> >> >> to echo what Abhi said: we are also currently looking of ways to >> automatically import data sets (libraries) into Galaxy without >> having >> to >> manually trigger the import via the administration interface, and >> ideally >> while keeping the data in the original place. The idea here is to >> have >> multiple tools all point at the original 'source data' without >> having >> to >> replicate terabytes of data. >> >> Not quite sure how feasible this is in practice, but it certainly >> would >> be >> incredibly helpful. >> >> Best, >> >>    Oliver >> >> >> >> >> On 28 Sep 2009, at 14:24, Abhishek Pratap wrote: >> >>> HI Greg >>> >>> Thanks for a quick reply and making some requested changes. However >>> I >>> am >>> not still sure if importing NGS data will help in long run. >>> >>> For Centers generating NGS data which could 2-3 T.B / week >>> depending >>> on >>> no. of sequencers I think importing another copy of raw data into >>> galaxy >>> workspace will be asking for lot of disk space. I understand it is >>> a >>> neat >>> way of doing things as it becomes agnostic of the raw data location >>>  but >>> might not be the best way for handling huge data in long run for >>> centers >>> like ours. >>> >>> Please correct me if I am wrong. I think we could also have a >>> simple >>> option without having to import the data and just using it for >>> analysis >>> from >>> the current location, also storing results at the same location. >>> That >>> way in >>> future even if the data set is moved analysis also stays with it. >>> >>> Let me know what you feel. I will be happy to know if there are any >>> other >>> smart reasons of importing the data in galaxy workspace just for >>> curiosity >>> sake. >>> >>> Thanks, >>> -Abhi >>> >>> On Mon, Sep 28, 2009 at 9:28 AM, Greg Von Kuster <ghv2@psu.edu> >>> wrote: >>> Hello Abhishek, >>> >>> The Galaxy distribution includes the enhancements to which I >>> previously >>> referred for uploading history files.  Uploading files >>> to a >>> history >>> now >>> creates a Galaxy job just like any other tool, and can be run on a >>> cluster >>> node, allowing upload of very large files.  The initial >>> pass of >>> this >>> work is >>> also completed for uploading to a Data Library, but this >>> enhancement >>> is >>> still in test, so it should soon be available in the distribution. >>> >>> Do you want to avoid having to import at all (e.g. allow Galaxy to >>> refer >>> to datasets that live in their original locations)? >>>  This is not >>> currently >>> possible, but if this is what you are looking for, we can consider >>> some >>> additional options on the current upload form, or possibly a new, >>> separate >>> form. >>> >>> >>> Greg Von Kuster >>> Galaxy Development Team >>> >>> >>> Abhishek Pratap wrote: >>> Hi Greg, Anton and all >>> >>> Just wondering if there has been any progress made on this end. I >>> am >>> sorry I was not able to follow it up on Assaf's suggestion due to >>> other >>> things at work. >>> >>> I did try the latest version of galaxy and looks like the files are >>> still >>> transferred over HTTP before they could be used in the galaxy >>> workspace. >>> Also I would again like to highlight that many labs might want to >>> use >>> the >>> local instance of galaxy and prefer to point to a local path where >>> the >>> file >>> is being stored. That way we will have both the benefits of using a >>> cool GUI >>> and process data stored locally. >>> >>> Let me know if you guys need some feedback or have more questions. >>> I >>> will >>> be happy to discuss them. >>> >>> best, >>> -Abhi >>> >>> On Tue, Jul 21, 2009 at 4:26 PM, Greg Von Kuster <ghv2@psu.edu >>> <mailto:ghv2@psu.edu>> wrote: >>> >>>   Hello Abishek, >>> >>>   We are currently in the process of significantly >>> enhancing the >>>   current Galaxy upload utilities, and the new version >>> should >>>   eliminate the issue you've raised about the time >>> needed to >>> upload >>>   large files via HTTP ( not for making an initial copy >>> of the >>> file in >>>   the Galaxy environment ). However, it will probably >>> not be >>> ready for >>>   release for a few more weeks, so if you can take >>> advantage of >>>   Assaf's script in the meantime, that's great. >>> ¨ÜI can't >>> guarantee >>>   that all Galaxy features will function correctly if >>> you do this >>> though. >>> >>>   Assaf, have you found that using your script breaks >>> anything? >>> >>>   Also, if you upload a file to a library rather than a >>> history, >>>   multiple users can "import" the library dataset into >>> their >>> history >>>   for analysis, but there is only 1 file on disk ( users >>> are >>> pointing >>>   to it from their histories ). ¨ÜBut >>> uploading a file to >>> a history >>>   will create a new copy of the file each time it is >>> uploaded. >>> >>>   Greg Von Kuster >>>   Galaxy Development Team >>> >>> >>> >>>   Abhishek Pratap wrote: >>> >>>       Hi All >>> >>>       @Greg : Please find my >>> comments below. >>> >>>       On Tue, Jul 21, 2009 at >>> 10:44 AM, Greg Von >>> Kuster<ghv2@psu.edu >>>       <mailto:ghv2@psu.edu>> >>> wrote: >>> >>>           >>> Hello Abhi, >>> >>>           >>> Can you clarify the steps you took that >>> produced the >>>           >>> behavior? ǃÜSee my >>> >>>           >>> comments below. >>> >>>           >>> Anton Nekrutenko wrote: >>> >>>           >>>     Abhishek: >>> >>>           >>>     Let talk. This is the area >>> of active current >>>           >>>     development. We are >>> ǃÜlooking >>> >>>           >>>     at implementing a universal >>> fastq-like format or >>>           >>>     supporting >>> ǃÜmultiple >>> >>>           >>>     formats. Perhaps we should >>> join efforts in ironing >>> out >>>           >>>     >>> ǃÜspecifications. >>> >>> >>>           >>>     anton >>>           >>>     galaxy team >>> >>> >>>           >>>     On Jul 20, 2009, at 5:18 >>> PM, Abhishek Pratap >>> wrote: >>> >>>           >>>         Hi All >>> >>> >>>           >>>         I recently came >>> to know about NGS analysis >>> on galaxy >>>           >>>         during ISMB. >>>           >>>         Getting excited >>> I tried couple of things >>> basically >>>           >>>         to play with >>> it. >>> >>>           >>>         Few comments : >>> I may have interepretted >>> something >>>           >>>         described below >>> in a >>>           >>>         wrong way. My >>> apologies before hand. >>> >>> >>> >>>           >>>         On a standalone >>> installation of galaxy while >>> I was >>>           >>>         trying to >>> explore >>>           >>>         one >>> FASTQ(sequence) file. It takes >>> considerable (> >>>           >>>         20 min) for a >>> fastq >>>           >>>         file to get >>> uploaded (2 GB). >>> >>>           >>> Are you using the Galaxy upload utility >>> to create an >>> item in >>>           >>> your history >>>           >>> that points to the dataset file on >>> disk? >>> >>> >>>       Yes that is precisely >>> correct, I am trying to >>> upload a solexa >>> FASTQ >>>       file but on a standalone >>> galaxy installation from >>> my local >>> file >>>       system. >>> >>>           I >>> am not sure what is the rationale >>> >>>           >>>         behind that. >>> Ideally I think there should be >>> no need >>>           >>>         to upload such >>>           >>>         heavy files >>> into the workspace. >>> >>>           A >>> data file that originates from a >>> place external to >>> Galaxy >>>           >>> must be uploaded >>>           >>> into Galaxy so that the disk file can >>> be placed in the >>>           >>> location configured >>>           in >>> the Galaxy config file. >>> ǃÜAlso, when data is >>> uploaded to >>> >>>           >>> Galaxy ( either >>>           to >>> a history or a library ), several >>> database table >>> settings >>>           >>> are created >>>           >>> that are used by various Galaxy >>> features. >>> >>>           >>> They could actually be used straight >>> >>> >>>       Thanks for the clarification >>> but I am not sure this >>> will help >>> a >>>       lot of >>>       people who are interested to >>> install and run galaxy >>> locally >>>       mainly for >>>       the following reasons. May >>> be it is just local to >>> me. >>> >>>       A. We already one instance >>> of data saved on the >>> local file >>> system >>>       B. Making another copy via >>> galaxy will eat away a >>> lot of space >>>       in long run. >>>       C. The time needed to import >>> the files into galaxy >>> space is >>> huge >>> >>>           >>>         away by the >>> path specified. >>> >>>           >>> What do you mean by "the path >>> specified"? >>> >>> >>> >>>       Well what I mean was a way >>> to specify the path of >>> the file/run >>>       on the >>>       lcoal file system and galaxy >>> could directly pick it >>> up from >>> there >>>       rather than uploading it >>> into its own space. Now I >>> understand >>> this >>>       might not work based on the >>> way the system was >>> designed. >>> >>> >>>           >>> Also is there any way to access the >>> >>>           >>>         scripts for >>> analysis on the command line. I >>> know >>>           >>>         this undermines >>> the >>>           >>>         main aim of >>> working with galaxy but rite now >>> I am >>>           >>>         concerned about >>> the >>>           >>>         >>> performance/time. >>> >>>           >>> You should be able to run any Galaxy >>> tool from the >>> command >>>           >>> line as long as >>>           >>> you have all of the tool's required >>> binaries in your >>> path. >>>           >>> ǃÜHowever, running >>> >>>           a >>> tool from within Galaxy should >>> generally not be any >>> slower >>>           >>> than running it >>>           >>> outside of Galaxy, depending, of >>> course, on what you are >>> doing. >>> >>> >>> >>> >>>       Ok I was under the >>> impression that running from >>> SHELL will >>> eliminate >>>       the step of uploading them >>> into galaxy file space. >>> >>> >>>       -Abhi >>> >>>           >>>         I will be happy >>> to discuss more about this >>> in case >>>           >>>         you have some >>>           >>>         >>> comments/questions for me. >>> >>> >>> >>>           >>>         Best, >>>           >>>         -Abhi >>> >>> >>> >>>           >>>         >>> ----------------------------- >>> >>>           >>>         Abhishek Pratap >>> >>>           >>>         Bioinformatics >>> Software Engineer >>> >>>           >>>         Institute for >>> Genome Sciences >>> >>>           >>>         School of >>> Medicine, Univ of Maryland >>> >>>           >>>         801, W. >>> Baltimore Street, Baltimore, MD >>> 21209 >>> >>>           >>>         Ph: >>> (+1)-410-706-2296 >>> >>>           >>>         >>> www.igs.umaryland.edu/ >>> <http://www.igs.umaryland.edu/> >>>           >>>         >>> _______________________________________________ >>>           >>>         galaxy-user >>> mailing list >>>           >>>         >>> galaxy-user@bx.psu.edu >>> <mailto:galaxy-user@bx.psu.edu> >>> >>> >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> >>>           >>>     Anton Nekrutenko >>>           >>>     http://nekrut.bx.psu.edu >>>           >>>     http://galaxyproject.org >>> >>>           >>>     >>> _______________________________________________ >>>           >>>     galaxy-user mailing list >>>           >>>     galaxy-user@bx.psu.edu >>> <mailto:galaxy-user@bx.psu.edu> >>> >>>           >>>     >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> galaxy-user mailing list >>> galaxy-user@bx.psu.edu >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >> >> -- >> Research Associate    Department of >> Biostatistics >> Associate Director    Bioinformatics Core >>           >>           >>  Harvard School of Public Health >> Skype: ohofmann       Phone: +1 >> (617) 365 0984 >> >> >>