Upload file size and user authorization
Two questions, either of which may belong more on the developers list. But lets try here first: 1. As we migrate services to Galaxy, a stakeholder expressed concern at uploading large datafiles of "several gigabytes". Given that we can require users to upload from an ftp site or something similar, does anyone have experience with how well this works with gigabyte-size files? I figure I can always write a tool to upload masses of data via a slow but robust method, but it would be good to know if this is required. 2. Similarly, someone else wanted to know if Galaxy authorization could be plugged into another system, such as Active Directory. Any pointers on this, or will I have to cut some code? thanks ---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency ----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
Hi Paul, On Mar 22, 2011, at 11:59 AM, Paul-Michael Agapow wrote:
Two questions, either of which may belong more on the developers list. But lets try here first:
1. As we migrate services to Galaxy, a stakeholder expressed concern at uploading large datafiles of “several gigabytes”. Given that we can require users to upload from an ftp site or something similar, does anyone have experience with how well this works with gigabyte-size files? I figure I can always write a tool to upload masses of data via a slow but robust method, but it would be good to know if this is required.
Large files are no problem and you may use an FTP site. Depending how much bandwidth you have it may take some time though to move large files around. Please see: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer for details on how to optimize uploads. Downloads are are different thing though as right now you can only get them as an HTTP download.
2. Similarly, someone else wanted to know if Galaxy authorization could be plugged into another system, such as Active Directory. Any pointers on this, or will I have to cut some code?
Yes, using "external authentication" via an Apache proxy as described here: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ApacheProxy I'm using this to link my Galaxy to our University's LDAP and that works like a charm :). Cheers, Pi
thanks
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving.HTTP://www.HPA.org.uk **************************************************************************
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
------------------------------------------------------------- mobile: +31 6 143 66 783 e-mail: pieter.neerincx@gmail.com skype: pieter.online -------------------------------------------------------------
Peter, Damn - looked through the documentation and missed that authentication bit. Good to know that the large files are no problem - I'm sure the people around here are going to push it to the limit! Thanks again p ---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency -----Original Message----- From: Pieter Neerincx [mailto:pieter.neerincx@gmail.com] Sent: 22 March 2011 12:07 To: Paul-Michael Agapow Cc: galaxy-user@lists.bx.psu.edu Subject: Re: [galaxy-user] Upload file size and user authorization Hi Paul, On Mar 22, 2011, at 11:59 AM, Paul-Michael Agapow wrote:
Two questions, either of which may belong more on the developers list. But lets try here first:
1. As we migrate services to Galaxy, a stakeholder expressed concern at uploading large datafiles of "several gigabytes". Given that we can require users to upload from an ftp site or something similar, does anyone have experience with how well this works with gigabyte-size files? I figure I can always write a tool to upload masses of data via a slow but robust method, but it would be good to know if this is required.
Large files are no problem and you may use an FTP site. Depending how much bandwidth you have it may take some time though to move large files around. Please see: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer for details on how to optimize uploads. Downloads are are different thing though as right now you can only get them as an HTTP download.
2. Similarly, someone else wanted to know if Galaxy authorization could be plugged into another system, such as Active Directory. Any pointers on this, or will I have to cut some code?
Yes, using "external authentication" via an Apache proxy as described here: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ApacheProxy I'm using this to link my Galaxy to our University's LDAP and that works like a charm :). Cheers, Pi
thanks
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
************************************************************************ ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving.HTTP://www.HPA.org.uk ************************************************************************ **
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
------------------------------------------------------------- mobile: +31 6 143 66 783 e-mail: pieter.neerincx@gmail.com skype: pieter.online ------------------------------------------------------------- ----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
Hi Paul, On Mar 22, 2011, at 1:17 PM, Paul-Michael Agapow wrote:
Peter,
Damn - looked through the documentation and missed that authentication bit.
In your defense: I couldn't find it back myself at first; it's not always easy to find what you are looking for on the wiki...
Good to know that the large files are no problem - I'm sure the people around here are going to push it to the limit!
Well, the upload shouldn't be a problem at all, but making sure your server(s) don't run out of disk space in no time can be quite problematic with NGS data :o... Galaxy stores all data - inputs, intermediate files and final results - by default. Files will only vanish if users actively delete them (and the admin runs the clean up scripts): This is very convenient for the end-users :), but it may be a headache for the sys admin... Cheers, Pi
Thanks again
p
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
-----Original Message----- From: Pieter Neerincx [mailto:pieter.neerincx@gmail.com] Sent: 22 March 2011 12:07 To: Paul-Michael Agapow Cc: galaxy-user@lists.bx.psu.edu Subject: Re: [galaxy-user] Upload file size and user authorization
Hi Paul,
On Mar 22, 2011, at 11:59 AM, Paul-Michael Agapow wrote:
Two questions, either of which may belong more on the developers list. But lets try here first:
1. As we migrate services to Galaxy, a stakeholder expressed concern at uploading large datafiles of "several gigabytes". Given that we can require users to upload from an ftp site or something similar, does anyone have experience with how well this works with gigabyte-size files? I figure I can always write a tool to upload masses of data via a slow but robust method, but it would be good to know if this is required.
Large files are no problem and you may use an FTP site. Depending how much bandwidth you have it may take some time though to move large files around. Please see:
https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer
for details on how to optimize uploads. Downloads are are different thing though as right now you can only get them as an HTTP download.
2. Similarly, someone else wanted to know if Galaxy authorization could be plugged into another system, such as Active Directory. Any pointers on this, or will I have to cut some code?
Yes, using "external authentication" via an Apache proxy as described here:
https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ApacheProxy
I'm using this to link my Galaxy to our University's LDAP and that works like a charm :).
Cheers,
Pi
thanks
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
************************************************************************ ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving.HTTP://www.HPA.org.uk ************************************************************************ **
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
------------------------------------------------------------- mobile: +31 6 143 66 783 e-mail: pieter.neerincx@gmail.com skype: pieter.online -------------------------------------------------------------
----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
------------------------------------------------------------- mobile: +31 6 143 66 783 e-mail: pieter.neerincx@gmail.com skype: pieter.online -------------------------------------------------------------
participants (2)
-
Paul-Michael Agapow -
Pieter Neerincx