Question using sftp to upload file to galaxy
Hey Folks, I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference? What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo). Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong! Cheers, Rui modules: $ sbin/proftpd -l Compiled-in modules: mod_core.c mod_xfer.c mod_rlimit.c mod_auth_unix.c mod_auth.c mod_ls.c mod_log.c mod_site.c mod_delay.c mod_facts.c mod_sql.c mod_sql_postgres.c mod_sql_passwd.c mod_sftp.c mod_cap.c etc/proftpd.conf ServerType standalone # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. <VirtualHost 10.3.17.42> Port 2222 SFTPEngine on AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects SFTPHostKey /etc/ssh/ssh_host_rsa_key RequireValidShell no MaxLoginAttempts 6 ServerName "Galaxy SFTP" DefaultServer on Umask 077 User bioinfoadmin Group bioinfoadmin UseFtpUsers off DefaultRoot ~ AllowOverwrite on AllowStoreRestart on SQLEngine on SQLGroupInfo sftp_groups name id members # Do not authenticate against real (system) users <IfModule mod_auth_pam.c> AuthPAM off </IfModule> # Common SQL authentication options SQLPasswordEngine on SQLBackend postgres SQLConnectInfo galaxy@galaxy.my.org:5432 bioinfoadmin dbpwd SQLAuthenticate users # Configuration that handles PBKDF2 encryption # Set up mod_sql to authenticate against the Galaxy database SQLAuthTypes PBKDF2 SQLPasswordPBKDF2 SHA256 10000 24 SQLPasswordEncoding base64 SQLPasswordUserSalt sql:/GetUserSalt # Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" # Define custom query to fetch the password salt SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" </VirtualHost> # Don't use IPv6 support by default. UseIPv6 off MaxInstances 30 # To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit> # Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> ~
Hey Folks, I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-) Cheers, Rui On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?
What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).
Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!
Cheers, Rui
modules: $ sbin/proftpd -l Compiled-in modules: mod_core.c mod_xfer.c mod_rlimit.c mod_auth_unix.c mod_auth.c mod_ls.c mod_log.c mod_site.c mod_delay.c mod_facts.c mod_sql.c mod_sql_postgres.c mod_sql_passwd.c mod_sftp.c mod_cap.c
etc/proftpd.conf
ServerType standalone # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. <VirtualHost 10.3.17.42> Port 2222 SFTPEngine on AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects SFTPHostKey /etc/ssh/ssh_host_rsa_key RequireValidShell no MaxLoginAttempts 6 ServerName "Galaxy SFTP" DefaultServer on Umask 077 User bioinfoadmin Group bioinfoadmin UseFtpUsers off DefaultRoot ~ AllowOverwrite on AllowStoreRestart on SQLEngine on SQLGroupInfo sftp_groups name id members
# Do not authenticate against real (system) users <IfModule mod_auth_pam.c> AuthPAM off </IfModule>
# Common SQL authentication options SQLPasswordEngine on SQLBackend postgres SQLConnectInfo galaxy@galaxy.my.org:5432 bioinfoadmin dbpwd SQLAuthenticate users
# Configuration that handles PBKDF2 encryption # Set up mod_sql to authenticate against the Galaxy database SQLAuthTypes PBKDF2 SQLPasswordPBKDF2 SHA256 10000 24 SQLPasswordEncoding base64 SQLPasswordUserSalt sql:/GetUserSalt
# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"
# Define custom query to fetch the password salt SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" </VirtualHost>
# Don't use IPv6 support by default. UseIPv6 off MaxInstances 30
# To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit>
# Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> ~
Hi Rui, there's a fairly complete explanation and example in in https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html Hope that helps, Marius On Mon, 28 Jan 2019 at 07:35, Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)
Cheers, Rui
On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?
What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).
Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!
Cheers, Rui
modules: $ sbin/proftpd -l Compiled-in modules: mod_core.c mod_xfer.c mod_rlimit.c mod_auth_unix.c mod_auth.c mod_ls.c mod_log.c mod_site.c mod_delay.c mod_facts.c mod_sql.c mod_sql_postgres.c mod_sql_passwd.c mod_sftp.c mod_cap.c
etc/proftpd.conf
ServerType standalone # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. <VirtualHost 10.3.17.42> Port 2222 SFTPEngine on AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects SFTPHostKey /etc/ssh/ssh_host_rsa_key RequireValidShell no MaxLoginAttempts 6 ServerName "Galaxy SFTP" DefaultServer on Umask 077 User bioinfoadmin Group bioinfoadmin UseFtpUsers off DefaultRoot ~ AllowOverwrite on AllowStoreRestart on SQLEngine on SQLGroupInfo sftp_groups name id members
# Do not authenticate against real (system) users <IfModule mod_auth_pam.c> AuthPAM off </IfModule>
# Common SQL authentication options SQLPasswordEngine on SQLBackend postgres SQLConnectInfo galaxy@galaxy.my.org:5432 bioinfoadmin dbpwd SQLAuthenticate users
# Configuration that handles PBKDF2 encryption # Set up mod_sql to authenticate against the Galaxy database SQLAuthTypes PBKDF2 SQLPasswordPBKDF2 SHA256 10000 24 SQLPasswordEncoding base64 SQLPasswordUserSalt sql:/GetUserSalt
# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"
# Define custom query to fetch the password salt SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" </VirtualHost>
# Don't use IPv6 support by default. UseIPv6 off MaxInstances 30
# To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit>
# Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> ~
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Hi Marius, Thanks for the note. The link you pasted is how I came up with the config in the original question. However it doesn't work...it kept saying my password is incorrect. :-( $ sftp -oKexAlgorithms=diffie-hellman-group14-sha1 -oPort=2222 bioinfoadmin@localhost bioinfoadmin@localhost's password: Permission denied, please try again. bioinfoadmin@localhost's password: Permission denied, please try again. bioinfoadmin@localhost's password: Not sure why this would happen. :-( Have you seen this before? Cheers, Rui On Sun, Jan 27, 2019 at 10:49 PM Marius van den Beek <m.vandenbeek@gmail.com> wrote:
Hi Rui,
there's a fairly complete explanation and example in in https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html
Hope that helps, Marius
On Mon, 28 Jan 2019 at 07:35, Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)
Cheers, Rui
On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?
What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).
Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!
Cheers, Rui
modules: $ sbin/proftpd -l Compiled-in modules: mod_core.c mod_xfer.c mod_rlimit.c mod_auth_unix.c mod_auth.c mod_ls.c mod_log.c mod_site.c mod_delay.c mod_facts.c mod_sql.c mod_sql_postgres.c mod_sql_passwd.c mod_sftp.c mod_cap.c
etc/proftpd.conf
ServerType standalone # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. <VirtualHost 10.3.17.42> Port 2222 SFTPEngine on AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects SFTPHostKey /etc/ssh/ssh_host_rsa_key RequireValidShell no MaxLoginAttempts 6 ServerName "Galaxy SFTP" DefaultServer on Umask 077 User bioinfoadmin Group bioinfoadmin UseFtpUsers off DefaultRoot ~ AllowOverwrite on AllowStoreRestart on SQLEngine on SQLGroupInfo sftp_groups name id members
# Do not authenticate against real (system) users <IfModule mod_auth_pam.c> AuthPAM off </IfModule>
# Common SQL authentication options SQLPasswordEngine on SQLBackend postgres SQLConnectInfo galaxy@galaxy.my.org:5432 bioinfoadmin dbpwd SQLAuthenticate users
# Configuration that handles PBKDF2 encryption # Set up mod_sql to authenticate against the Galaxy database SQLAuthTypes PBKDF2 SQLPasswordPBKDF2 SHA256 10000 24 SQLPasswordEncoding base64 SQLPasswordUserSalt sql:/GetUserSalt
# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"
# Define custom query to fetch the password salt SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" </VirtualHost>
# Don't use IPv6 support by default. UseIPv6 off MaxInstances 30
# To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit>
# Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> ~
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Are you logging in with the email adress and password registered for your Galaxy user acccount ? The authentication happens against the Galaxy database. Also keep an eye on your proftpd logs in case that wasn't the issue. On Fri, 1 Feb 2019 at 19:25, Rui Wang <ruiwang.sz@gmail.com> wrote:
Hi Marius,
Thanks for the note. The link you pasted is how I came up with the config in the original question. However it doesn't work...it kept saying my password is incorrect. :-(
$ sftp -oKexAlgorithms=diffie-hellman-group14-sha1 -oPort=2222 bioinfoadmin@localhost bioinfoadmin@localhost's password: Permission denied, please try again. bioinfoadmin@localhost's password: Permission denied, please try again. bioinfoadmin@localhost's password:
Not sure why this would happen. :-( Have you seen this before?
Cheers, Rui
On Sun, Jan 27, 2019 at 10:49 PM Marius van den Beek < m.vandenbeek@gmail.com> wrote:
Hi Rui,
there's a fairly complete explanation and example in in https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html
Hope that helps, Marius
On Mon, 28 Jan 2019 at 07:35, Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)
Cheers, Rui
On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <ruiwang.sz@gmail.com> wrote:
Hey Folks,
I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?
What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).
Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!
Cheers, Rui
modules: $ sbin/proftpd -l Compiled-in modules: mod_core.c mod_xfer.c mod_rlimit.c mod_auth_unix.c mod_auth.c mod_ls.c mod_log.c mod_site.c mod_delay.c mod_facts.c mod_sql.c mod_sql_postgres.c mod_sql_passwd.c mod_sftp.c mod_cap.c
etc/proftpd.conf
ServerType standalone # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. <VirtualHost 10.3.17.42> Port 2222 SFTPEngine on AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects SFTPHostKey /etc/ssh/ssh_host_rsa_key RequireValidShell no MaxLoginAttempts 6 ServerName "Galaxy SFTP" DefaultServer on Umask 077 User bioinfoadmin Group bioinfoadmin UseFtpUsers off DefaultRoot ~ AllowOverwrite on AllowStoreRestart on SQLEngine on SQLGroupInfo sftp_groups name id members
# Do not authenticate against real (system) users <IfModule mod_auth_pam.c> AuthPAM off </IfModule>
# Common SQL authentication options SQLPasswordEngine on SQLBackend postgres SQLConnectInfo galaxy@galaxy.my.org:5432 bioinfoadmin dbpwd SQLAuthenticate users
# Configuration that handles PBKDF2 encryption # Set up mod_sql to authenticate against the Galaxy database SQLAuthTypes PBKDF2 SQLPasswordPBKDF2 SHA256 10000 24 SQLPasswordEncoding base64 SQLPasswordUserSalt sql:/GetUserSalt
# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"
# Define custom query to fetch the password salt SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" </VirtualHost>
# Don't use IPv6 support by default. UseIPv6 off MaxInstances 30
# To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit>
# Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> ~
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
participants (2)
-
Marius van den Beek -
Rui Wang