Hi John
I will review the galaxy code more closely and implement it as you suggested. It's been a while since I've implemented this 'fix', so I will have to dig up the code that tries to delete the file after upload.
Once it's simple and clean, I will do a pull request.
Thanks for your input.
Iyad Kandalaft Bioinformatics Programmer Microbial Biodiversity Bioinformatics Science & Technology Branch Agriculture & Agri-Food Canada Iyad.Kandalaft@agr.gc.ca | (613) 759-1228
________________________________________ From: John Chilton [jmchilton@gmail.com] Sent: June 1, 2014 7:53 PM To: Kandalaft, Iyad Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Uploading files to galaxy from a folder
Hello Iyad,
Thanks for taking the time to describe your problem and share your solution - I am sure other institutions have desires to harness the ftp upload option the way you are doing it.
Is the only problem that occurs if the ownership is incorrect that Galaxy cannot delete the files? If so - I feel like as a deployer I would prefer to have a sudo script that just deletes the files instead of changing the ownership? It is a more... focused sudo command.
This strategy could also allow a slightly more general universe_wsgi.ini option - say a few options such as:
ftp_upload_delete_handling = __default__ # just assume ownership and delete ftp_upload_delete_handling = __none__ # don't delete files - user is responsible ftp_upload_delete_handling = sudo -E rm # command prefix to delete file
I am not saying if you clean up what you have and opened a pull request I would say no - I am just saying I would be more eager for this more general option. If you can confirm the only reason the permissions need to change is to delete the files and likewise prefer this variant but cannot figure out how to modify upload.py - let me know I can try to look into it (no promises that it is easy to implement).
-John
On Tue, May 27, 2014 at 10:28 AM, Kandalaft, Iyad Iyad.Kandalaft@agr.gc.ca wrote:
Hi Everyone
I’m throwing this out there for some feedback and recommendations.
Objective: Facilitate transferring large files (> 2GB) from an HPC cluster (and its associated fast tier storage) to galaxy for my clients. I enabled the FTP upload option in galaxy but it involves users learning to copy files over FTP.
So, I created a galaxy folder in each users’ home directory on the HPC Cluster that symbolically links to the FTP upload folder for galaxy. Hence, users can use either FTP to upload files (drag and drop in windows) or simply copy files into this folder from an ssh session on the cluster. The problem with that strategy was that galaxy had to be the owner of the file (similar to the ProFTPd configuration that sets the UID and GID of uploads files to galaxy’s UID/GID). Otherwise, galaxy threw errors when it tried deleting the original file from the FTP upload folder. I could have added the galaxy user to the same group as all user but this meant that users would have to ensure the correct permissions are set on files so that galaxy can read and delete the file thereafter. The alternative involved modifying the upload.py tool to chown/chmod files that were being uploaded. Upload.py now sudo executes an external script that sets ownership to the galaxy user and corrects the permissions if required (see attachment for code modification). The galaxy user has sudo rights on this script and the script restricts chown/chmod to the ftp folder path for security reasons.
I was planning to clean up the code and make it production ready by adding an option in universe_wsgi.ini for this “feature”, but I thought I would check with the galaxy devs first. Am I taking the wrong approach? Is there a better alternative?
As an alternative, I thought about locating the handler code for dataset.type == file and possibly making it support the SETGID sticky bits on folders. In that case, the FTP upload folder would have the sticky bit set for UID and can assume the role of the user to upload that file.
Your input is much appreciated.
Iyad Kandalaft
Bioinformatics Application Developer
Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
KW Neatby Bldg | éd. KW Neatby
960 Carling Ave| 960, avenue Carling
Ottawa, ON | Ottawa (ON) K1A 0C6
E-mail Address / Adresse courriel: Iyad.Kandalaft@agr.gc.ca
Telephone | Téléphone 613- 759-1228
Facsimile | Télécopieur 613-759-1701
Government of Canada | Gouvernement du Canada
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/