Data Library - upload files from filesystem paths
Hi As an admin I have created a data library on our local instance of galaxy, created folders in that library and added datasets using 'upload files from filesystem paths' option, choosing not to copy the data into Galaxy, which then loads all the files in the given directory into the data library. Is there a way to refresh this list automatically? so that if a new file is added into that directory it will appear in the data library? or do I have to add new files in that directory manually, giving the path to the new file? I am also having a problem with creating sub folders - I choose to create sub folder from the drop down list and it reports that the sub folder has been created but I cant see it, and I cant click on the blue triangles to the left of the folders either. Does anyone else have this problem? Any help would be appreciated, Thanks, Nicki --------------------- Nicki Gray MRC Molecular Haematology Unit 01865 222434
Hello Nicki, On Jan 28, 2011, at 1:13 PM, Nicki Gray wrote:
Hi
As an admin I have created a data library on our local instance of galaxy, created folders in that library and added datasets using 'upload files from filesystem paths' option, choosing not to copy the data into Galaxy, which then loads all the files in the given directory into the data library.
Is there a way to refresh this list automatically? so that if a new file is added into that directory it will appear in the data library? or do I have to add new files in that directory manually, giving the path to the new file?
You could write a script that polls the desired directory and uses Galaxy's API to upload new files. See <galaxy install dir>/scripts/api/README for some information. The distribution includes a script in this same directory to upload library datasets ( library_upload_from_import_dir.py ).
I am also having a problem with creating sub folders - I choose to create sub folder from the drop down list and it reports that the sub folder has been created but I cant see it, and I cant click on the blue triangles to the left of the folders either. Does anyone else have this problem?
Assuming you are adding sub-folders from the Shared Data -> Data Libraries view ( and not the Admin - Manage data libraries view ), this may be a result of permissions that have been placed on the library or parent folder in which you are adding sub-folders. The behavior you see could be a bug, but if so, I'll need some help tracking it down. Can you confirm from what view you are adding hte sub-folders, and what the permissions are on the parent folder and library? Thanks! Greg Von Kuster
Any help would be appreciated, Thanks, Nicki
--------------------- Nicki Gray MRC Molecular Haematology Unit 01865 222434
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
Hi Greg thanks for your response, I have some questions about your reply. On 1 Feb 2011, at 16:14, Greg Von Kuster wrote:
Hello Nicki, On Jan 28, 2011, at 1:13 PM, Nicki Gray wrote:
Hi
As an admin I have created a data library on our local instance of galaxy, created folders in that library and added datasets using 'upload files from filesystem paths' option, choosing not to copy the data into Galaxy, which then loads all the files in the given directory into the data library.
Is there a way to refresh this list automatically? so that if a new file is added into that directory it will appear in the data library? or do I have to add new files in that directory manually, giving the path to the new file?
You could write a script that polls the desired directory and uses Galaxy's API to upload new files. See <galaxy install dir>/scripts/ api/README for some information. The distribution includes a script in this same directory to upload library datasets ( library_upload_from_import_dir.py ).
I have had a quick look at this script and I'm not sure this will do what we want. I see it says I should " browse to that user's preferences and generate a new API Key" - will it need to be done for each user? What we would like is to initially upload a directory path for each user, and then the user can refresh this "library list" (or it happen automatically say, every 10 secs) I notice that under the 'admin-manage data libraries' there is an 'Upload a new version of this dataset' option which replaces existing datasets - how easy would it be to create something similar for a directory? an "Upload a new version of this file system directory" and could the user be able to do that themselves?
I am also having a problem with creating sub folders - I choose to create sub folder from the drop down list and it reports that the sub folder has been created but I cant see it, and I cant click on the blue triangles to the left of the folders either. Does anyone else have this problem?
Assuming you are adding sub-folders from the Shared Data -> Data Libraries view ( and not the Admin - Manage data libraries view ), this may be a result of permissions that have been placed on the library or parent folder in which you are adding sub-folders. The behavior you see could be a bug, but if so, I'll need some help tracking it down. Can you confirm from what view you are adding hte sub-folders, and what the permissions are on the parent folder and library?
I was creating sub folders from the Admin - Manage data libraries view. I click on the arrow next the folder name, choose Add sub- folder, give it a name and get "The new folder named 'xxx' has been added to the data library." but I cant see it Thanks for your help, Nicki
Thanks!
Greg Von Kuster
Any help would be appreciated, Thanks, Nicki
--------------------- Nicki Gray MRC Molecular Haematology Unit 01865 222434
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
On Feb 1, 2011, at 12:04 PM, Nicki Gray wrote:
Hi Greg thanks for your response, I have some questions about your reply.
On 1 Feb 2011, at 16:14, Greg Von Kuster wrote:
Hello Nicki, On Jan 28, 2011, at 1:13 PM, Nicki Gray wrote:
Hi
As an admin I have created a data library on our local instance of galaxy, created folders in that library and added datasets using 'upload files from filesystem paths' option, choosing not to copy the data into Galaxy, which then loads all the files in the given directory into the data library.
Is there a way to refresh this list automatically? so that if a new file is added into that directory it will appear in the data library? or do I have to add new files in that directory manually, giving the path to the new file?
You could write a script that polls the desired directory and uses Galaxy's API to upload new files. See <galaxy install dir>/scripts/api/README for some information. The distribution includes a script in this same directory to upload library datasets ( library_upload_from_import_dir.py ).
I have had a quick look at this script and I'm not sure this will do what we want. I see it says I should " browse to that user's preferences and generate a new API Key" - will it need to be done for each user?
Any user that uses the API will need an API key. They'll need to go to User -> Preferences -> Manage your API keys to generate one if it is not set for them. The API key only needs to be set once.
What we would like is to initially upload a directory path for each user, and then the user can refresh this "library list" (or it happen automatically say, every 10 secs)
I notice that under the 'admin-manage data libraries' there is an 'Upload a new version of this dataset' option which replaces existing datasets - how easy would it be to create something similar for a directory? an "Upload a new version of this file system directory" and could the user be able to do that themselves?
The best approach for doing this would be to upload each new directory "version" to a different folder or data library. When you upload a directory of files, whether copying files into the Galaxy files location or using file system paths, Galaxy creates pointers to those files, enabling the dataset objects associated with them to be versioned. Versions of local subdirectories cannot be done in Galaxy data libraries because the structure of the data library hierarchy is "data library / folder / datasets", and subdirectories themselves ( or information about them ) are not stored.
I am also having a problem with creating sub folders - I choose to create sub folder from the drop down list and it reports that the sub folder has been created but I cant see it, and I cant click on the blue triangles to the left of the folders either. Does anyone else have this problem?
Assuming you are adding sub-folders from the Shared Data -> Data Libraries view ( and not the Admin - Manage data libraries view ), this may be a result of permissions that have been placed on the library or parent folder in which you are adding sub-folders. The behavior you see could be a bug, but if so, I'll need some help tracking it down. Can you confirm from what view you are adding hte sub-folders, and what the permissions are on the parent folder and library?
I was creating sub folders from the Admin - Manage data libraries view. I click on the arrow next the folder name, choose Add sub-folder, give it a name and get "The new folder named 'xxx' has been added to the data library." but I cant see it
What browser are you using? Safari, chrome or firefox are recommended as IE tends to have many more behavioral problems like this.
Thanks for your help, Nicki
Thanks!
Greg Von Kuster
Any help would be appreciated, Thanks, Nicki
--------------------- Nicki Gray MRC Molecular Haematology Unit 01865 222434
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
Hi Greg
As an admin I have created a data library on our local instance of galaxy, created folders in that library and added datasets using 'upload files from filesystem paths' option, choosing not to copy the data into Galaxy, which then loads all the files in the given directory into the data library.
Is there a way to refresh this list automatically? so that if a new file is added into that directory it will appear in the data library? or do I have to add new files in that directory manually, giving the path to the new file?
You could write a script that polls the desired directory and uses Galaxy's API to upload new files. See <galaxy install dir>/ scripts/api/README for some information. The distribution includes a script in this same directory to upload library datasets ( library_upload_from_import_dir.py ).
I have had a quick look at this script and I'm not sure this will do what we want. I see it says I should " browse to that user's preferences and generate a new API Key" - will it need to be done for each user?
Any user that uses the API will need an API key. They'll need to go to User -> Preferences -> Manage your API keys to generate one if it is not set for them. The API key only needs to be set once.
thanks, I will try this and see if this will do what we require and let you know.
What we would like is to initially upload a directory path for each user, and then the user can refresh this "library list" (or it happen automatically say, every 10 secs)
I notice that under the 'admin-manage data libraries' there is an 'Upload a new version of this dataset' option which replaces existing datasets - how easy would it be to create something similar for a directory? an "Upload a new version of this file system directory" and could the user be able to do that themselves?
The best approach for doing this would be to upload each new directory "version" to a different folder or data library.
When you upload a directory of files, whether copying files into the Galaxy files location or using file system paths, Galaxy creates pointers to those files, enabling the dataset objects associated with them to be versioned.
Versions of local subdirectories cannot be done in Galaxy data libraries because the structure of the data library hierarchy is "data library / folder / datasets", and subdirectories themselves ( or information about them ) are not stored.
I am also having a problem with creating sub folders - I choose to create sub folder from the drop down list and it reports that the sub folder has been created but I cant see it, and I cant click on the blue triangles to the left of the folders either. Does anyone else have this problem?
Assuming you are adding sub-folders from the Shared Data -> Data Libraries view ( and not the Admin - Manage data libraries view ), this may be a result of permissions that have been placed on the library or parent folder in which you are adding sub-folders. The behavior you see could be a bug, but if so, I'll need some help tracking it down. Can you confirm from what view you are adding hte sub-folders, and what the permissions are on the parent folder and library?
I was creating sub folders from the Admin - Manage data libraries view. I click on the arrow next the folder name, choose Add sub- folder, give it a name and get "The new folder named 'xxx' has been added to the data library." but I cant see it
What browser are you using? Safari, chrome or firefox are recommended as IE tends to have many more behavioral problems like this.
I was using Camino - I have just tried with Firefox and I can see all the sub folders now. Thanks for your help! Nicki
participants (2)
-
Greg Von Kuster -
Nicki Gray