Hi, I noticed the recent update to Galaxy that allowed users to put data on the galaxy FTP server and then import it into their history. Would it be possible to use a similar mechanism to import data from the galaxy host file system but not use an FTP daemon? In our case we we would like to mount a volume to the galaxy file server and allow people to cp or mv data into it and then be able to import the data into their history. For example, In the context of the example at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP could you set up a directory called '/home/nate/galaxy_dist/database/localfs/' instead of '/home/nate/galaxy_dist/database/ftp/' which contained email named directories and the files in these in these would be available to import via the upload UI? Or is it all tied into FTP? Thanks for any advice, Steve
On 11/26/2010 12:29 PM, Steve Taylor wrote:
Hi,
I noticed the recent update to Galaxy that allowed users to put data on the galaxy FTP server and then import it into their history.
Would it be possible to use a similar mechanism to import data from the galaxy host file system but not use an FTP daemon? In our case we we would like to mount a volume to the galaxy file server and allow people to cp or mv data into it and then be able to import the data into their history.
Have you looked into "Data Libraries" ? There is also an option to allow uploading files by a non-admin user. For details see the "# -- Data Libraries" section in the 'universe_wsgi.ini' file. Regards, Hans
For example, In the context of the example at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP could you set up a directory called '/home/nate/galaxy_dist/database/localfs/' instead of '/home/nate/galaxy_dist/database/ftp/' which contained email named directories and the files in these in these would be available to import via the upload UI? Or is it all tied into FTP?
Thanks for any advice,
Steve _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Hi Hans, We did have a look at this but couldn't get it to work. But will try again. Thanks for the pointer, Steve
I noticed the recent update to Galaxy that allowed users to put data on the galaxy FTP server and then import it into their history.
Would it be possible to use a similar mechanism to import data from the galaxy host file system but not use an FTP daemon? In our case we we would like to mount a volume to the galaxy file server and allow people to cp or mv data into it and then be able to import the data into their history.
Have you looked into "Data Libraries" ? There is also an option to allow uploading files by a non-admin user.
For details see the "# -- Data Libraries" section in the 'universe_wsgi.ini' file.
Regards, Hans
For example, In the context of the example at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP could you set up a directory called '/home/nate/galaxy_dist/database/localfs/' instead of '/home/nate/galaxy_dist/database/ftp/' which contained email named directories and the files in these in these would be available to import via the upload UI? Or is it all tied into FTP?
Thanks for any advice,
Steve _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Steve Taylor wrote:
Hi Hans,
We did have a look at this but couldn't get it to work. But will try again.
Hi Steve, If there are problems with this method please do let us know so we can look in to it. Your thoughts about using the FTP upload as you describe should work fine. Unfortunately it is not written so generically, so you'll need to set an FTP server name in the appropriate option in the config file despite not having one, which will be displayed in the help text on the upload form. However, this server name is not verified in any way and can be any string. --nate
Thanks for the pointer,
Steve
I noticed the recent update to Galaxy that allowed users to put data on the galaxy FTP server and then import it into their history.
Would it be possible to use a similar mechanism to import data from the galaxy host file system but not use an FTP daemon? In our case we we would like to mount a volume to the galaxy file server and allow people to cp or mv data into it and then be able to import the data into their history.
Have you looked into "Data Libraries" ? There is also an option to allow uploading files by a non-admin user.
For details see the "# -- Data Libraries" section in the 'universe_wsgi.ini' file.
Regards, Hans
For example, In the context of the example at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP could you set up a directory called '/home/nate/galaxy_dist/database/localfs/' instead of '/home/nate/galaxy_dist/database/ftp/' which contained email named directories and the files in these in these would be available to import via the upload UI? Or is it all tied into FTP?
Thanks for any advice,
Steve _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
On Nov 28, 2010, at 3:25 PM, Nate Coraor wrote:
Steve Taylor wrote:
Hi Hans,
We did have a look at this but couldn't get it to work. But will try again.
Hi Steve,
If there are problems with this method please do let us know so we can look in to it.
I've seen a few issues related to adding datasets to libraries as a non admin: from file system or directory of files (anonymous bug report): https://bitbucket.org/galaxy/galaxy-central/issue/413/500-server-error-impor... from history (bug reported by me): http://bitbucket.org/galaxy/galaxy-central/issue/414/unable-to-add-dataset-t... -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153
Hi Nate,
If there are problems with this method please do let us know so we can look in to it.
Your thoughts about using the FTP upload as you describe should work fine. Unfortunately it is not written so generically, so you'll need to set an FTP server name in the appropriate option in the config file despite not having one, which will be displayed in the help text on the upload form. However, this server name is not verified in any way and can be any string.
I'm trying to set up a directory using FTP Upload (without an FTP server :-)) as discussed above. I added a test directory for the a user 'auser' ftp_upload_dir = /data/galaxy/ ftp_upload_site = localhost to the universe.ini We are using LDAP, so I set up a test directory called /data/galaxy/auser@molbiol.ox.ac.uk but having added a file in can't see anything in the FTP upload section or the 'Files uploaded via FTP' section even when 'auser' is logged in. Is there something else I need to do? Thanks, Steve
Stephen Taylor wrote:
Hi Nate,
If there are problems with this method please do let us know so we can look in to it.
Your thoughts about using the FTP upload as you describe should work fine. Unfortunately it is not written so generically, so you'll need to set an FTP server name in the appropriate option in the config file despite not having one, which will be displayed in the help text on the upload form. However, this server name is not verified in any way and can be any string.
I'm trying to set up a directory using FTP Upload (without an FTP server :-)) as discussed above.
I added a test directory for the a user 'auser'
ftp_upload_dir = /data/galaxy/ ftp_upload_site = localhost
to the universe.ini
We are using LDAP, so I set up a test directory called
/data/galaxy/auser@molbiol.ox.ac.uk
but having added a file in can't see anything in the FTP upload section or the 'Files uploaded via FTP' section even when 'auser' is logged in.
Is there something else I need to do?
This should be it - are you certain the email in Galaxy exactly matches the email used in the directory name (it's case-sensitive)? And this may be a silly question, but are you running a new enough revision that has the FTP Upload code? --nate
Thanks,
Steve
Hi Nate,
If there are problems with this method please do let us know so we can look in to it.
Your thoughts about using the FTP upload as you describe should work fine. Unfortunately it is not written so generically, so you'll need to set an FTP server name in the appropriate option in the config file despite not having one, which will be displayed in the help text on the upload form. However, this server name is not verified in any way and can be any string.
I'm trying to set up a directory using FTP Upload (without an FTP server :-)) as discussed above.
I added a test directory for the a user 'auser'
ftp_upload_dir = /data/galaxy/ ftp_upload_site = localhost
to the universe.ini
We are using LDAP, so I set up a test directory called
/data/galaxy/auser@molbiol.ox.ac.uk
but having added a file in can't see anything in the FTP upload section or the 'Files uploaded via FTP' section even when 'auser' is logged in.
Is there something else I need to do?
This should be it - are you certain the email in Galaxy exactly matches the email used in the directory name (it's case-sensitive)?
Yes.
And this may be a silly question, but are you running a new enough revision that has the FTP Upload code?
I think so. Just to double check is there a definitive way I can tell? In ./tools/data_source/upload.xml <param name="ftp_files" type="ftpfile" label="Files uploaded via FTP"/> Thanks, Steve
Hi,
If there are problems with this method please do let us know so we can look in to it.
Your thoughts about using the FTP upload as you describe should work fine. Unfortunately it is not written so generically, so you'll need to set an FTP server name in the appropriate option in the config file despite not having one, which will be displayed in the help text on the upload form. However, this server name is not verified in any way and can be any string.
I'm trying to set up a directory using FTP Upload (without an FTP server :-)) as discussed above.
I added a test directory for the a user 'auser'
ftp_upload_dir = /data/galaxy/ ftp_upload_site = localhost
to the universe.ini
We are using LDAP, so I set up a test directory called
/data/galaxy/auser@molbiol.ox.ac.uk
but having added a file in can't see anything in the FTP upload section or the 'Files uploaded via FTP' section even when 'auser' is logged in.
Is there something else I need to do?
This should be it - are you certain the email in Galaxy exactly matches the email used in the directory name (it's case-sensitive)?
Yes.
And this may be a silly question, but are you running a new enough revision that has the FTP Upload code?
I think so. Just to double check is there a definitive way I can tell? In ./tools/data_source/upload.xml
<param name="ftp_files" type="ftpfile" label="Files uploaded via FTP"/>
So I got this to work by hacking basic.py. def visible( self ): self.tool.app.config.ftp_upload_dir = '/data/galaxy' self.tool.app.config.ftp_upload_site = 'localhost' I can now see my files in my user directory. What is the best part of the code to use log.warning() to see what's going on? How can tell if it is reading the ini file ok? My next question is based on permissions. It looks like galaxy has to own the file (as nobody in your example) to upload and remove it from the directory. I'd like to change it to be more forceful, which would allow galaxy to remove the file if group owned by nobody. What module/lines do this and what impact would such a change have? Thanks for any help, Steve
Steve Taylor wrote:
I think so. Just to double check is there a definitive way I can tell? In ./tools/data_source/upload.xml
<param name="ftp_files" type="ftpfile" label="Files uploaded via FTP"/>
So I got this to work by hacking basic.py.
def visible( self ): self.tool.app.config.ftp_upload_dir = '/data/galaxy' self.tool.app.config.ftp_upload_site = 'localhost'
I can now see my files in my user directory. What is the best part of the code to use log.warning() to see what's going on? How can tell if it is reading the ini file ok?
Hi Steve, sorry for not getting back to you. I hadn't realized there was a question here when I originally skimmed the email. This can be done in lib/galaxy/config.py
My next question is based on permissions. It looks like galaxy has to own the file (as nobody in your example) to upload and remove it from the directory. I'd like to change it to be more forceful, which would allow galaxy to remove the file if group owned by nobody. What module/lines do this and what impact would such a change have?
This would likely have to be done in the tools/data_source/upload.py file. We are going to be working on more general methods to allow for local file uploads in typical lab environments like yours where Galaxy users have access to (at least part of) the local filesystem, which would hopefully make this less of a hack for you. --nate
Thanks for any help,
Steve
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
On 21/02/2011 14:56, Nate Coraor wrote:
Steve Taylor wrote:
I think so. Just to double check is there a definitive way I can tell? In ./tools/data_source/upload.xml
<param name="ftp_files" type="ftpfile" label="Files uploaded via FTP"/>
So I got this to work by hacking basic.py.
def visible( self ): self.tool.app.config.ftp_upload_dir = '/data/galaxy' self.tool.app.config.ftp_upload_site = 'localhost'
I can now see my files in my user directory. What is the best part of the code to use log.warning() to see what's going on? How can tell if it is reading the ini file ok?
Hi Steve, sorry for not getting back to you. I hadn't realized there was a question here when I originally skimmed the email.
This can be done in lib/galaxy/config.py
My next question is based on permissions. It looks like galaxy has to own the file (as nobody in your example) to upload and remove it from the directory. I'd like to change it to be more forceful, which would allow galaxy to remove the file if group owned by nobody. What module/lines do this and what impact would such a change have?
This would likely have to be done in the tools/data_source/upload.py file.
Thanks, I'll have a look.
We are going to be working on more general methods to allow for local file uploads in typical lab environments like yours where Galaxy users have access to (at least part of) the local filesystem, which would hopefully make this less of a hack for you.
That sounds very good. Basically it would be good to have file browser type functionality. We are planning on having designated areas that users can put files (or even sym links) in and see their data without uploading it. There are obviously permission issues involved in this that need some management, so I'd be interested in hearing the ideas planned. Steve
participants (5)
-
Glen Beane -
Hans-Rudolf Hotz -
Nate Coraor -
Stephen Taylor -
Steve Taylor