Hello, What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ? By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance) Youssef Ghorbal IIS / DSI Institut Pasteur ----------------------- On 26 Nov 2018, at 15:23, Previti <christopher.previti@dkfz-heidelberg.de<mailto:christopher.previti@dkfz-heidelberg.de>> wrote: Dear all, We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses. This worked fine for Galaxy version 17.09...and stopped working for Version 18.05. An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files lrwxrwxrwx 1 galaxy W190-htseq-bioinf 28 Sep 11 12:37 christopher.previti@dkfz-heidelberg.de<mailto:christopher.previti@dkfz-heidelberg.de> -> /galaxy-clients/w190_galaxy/ Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share is called "w190_galaxy") directly into a data-library (with the approriate rights). Would anybody have an idea why this doesn't work anymore in v18.05? Is it a galaxy.ini configuration issue? Thanks for any advise you could provide! Christopher -- Dr. Christopher Previti Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti@dkfz.de<http://www.dkfz.de/> www.dkfz.de<http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/

Hello, As per this https://github.com/galaxyproject/galaxy/blob/9a6fddcdca7042e5e453bdd235ed8c5... The message you are seeing "You do not have any files in your user directory. Use FTP to upload there" has nothing to do with user_library_import_symlink_whitelist The symlink control (involving user_library_import_symlink_whitelist) is done a little bit later in the code. The test code is rather straightforward it does an os.path.exists on the path contact of the user_library_import_dir setting and the user's email address. Option 1: Galaxy is not resolving the correct email address for your account, I don't see why this may change after upgrade. But it's worth double checking (especially the case) Option 2: There is an issue with permissions on the symlink target. I've checked python will return false on os.path.exists if the symlink targets a directory not accessible by the user. For me you need to double check the permissions on your server. Maybe with a shell access as the galaxy@AD.DKFZ-HEIDELBERG.DE<mailto:galaxy@AD.DKFZ-HEIDELBERG.DE> user, you can try access one of the symlinks and see how it goes. You may also check that the Galaxy service is actually running under the intended identity (galaxy@AD.DKFZ-HEIDELBERG.DE<mailto:galaxy@AD.DKFZ-HEIDELBERG.DE>) Youssef Ghorbal IIS / DSI Institut Pasteur --------------------------------- On 27 Nov 2018, at 11:43, Previti <christopher.previti@dkfz-heidelberg.de<mailto:christopher.previti@dkfz-heidelberg.de>> wrote: Dear Youssef, Thanks for answering, the target permissions have not changed and are defined via NFS4 ACLs: A::OWNER@:rwaDxtTnNcCy A::GROUP@:rwaDxtTnNcy A:fdi:galaxy@AD.DKFZ-HEIDELBERG.DE:rwaDxtTnNcy<mailto:A:fdi:galaxy@AD.DKFZ-HEIDELBERG.DE:rwaDxtTnNcy> A:fdig:W190-galaxy-admins@AD.DKFZ-HEIDELBERG.DE:rwaDxtTnNcy<mailto:A:fdig:W190-galaxy-admins@AD.DKFZ-HEIDELBERG.DE:rwaDxtTnNcy> In this case the galaxy user (i.e. the user starting the process) has full access to the share. In our instance the "user directories" are just symbolic links to the target folders mention above. When I try to import data from a user directory/NFS4 share into a data library we get the following error: "You do not habe any files in your user directory. Use FTP to upload there." (I attached a screenshot) We still use galaxy.ini to configure our instance, in the galaxy.yml I find the following parameter: # For security reasons, users may not import any files that actually # lie outside of their `user_library_import_dir` (e.g. using symbolic # links). A list of directories can be allowed by setting the # following option (the list is comma-separated). Be aware that *any* # user with library import permissions can import from anywhere in # these directories (assuming they are able to create symlinks to # them). #user_library_import_symlink_whitelist: null Would implementing this solve my issue? Cheers, Christopher On 11/26/18 4:36 PM, Youssef GHORBAL wrote: Hello, What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ? By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance) Youssef Ghorbal IIS / DSI Institut Pasteur ----------------------- On 26 Nov 2018, at 15:23, Previti <christopher.previti@dkfz-heidelberg.de<mailto:christopher.previti@dkfz-heidelberg.de>> wrote: Dear all, We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses. This worked fine for Galaxy version 17.09...and stopped working for Version 18.05. An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files lrwxrwxrwx 1 galaxy W190-htseq-bioinf 28 Sep 11 12:37 christopher.previti@dkfz-heidelberg.de<mailto:christopher.previti@dkfz-heidelberg.de> -> /galaxy-clients/w190_galaxy/ Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share is called "w190_galaxy") directly into a data-library (with the approriate rights). Would anybody have an idea why this doesn't work anymore in v18.05? Is it a galaxy.ini configuration issue? Thanks for any advise you could provide! Christopher -- Dr. Christopher Previti Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti@dkfz.de<http://www.dkfz.de/> www.dkfz.de<http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ -- Dr. Christopher Previti Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti@dkfz.de<http://www.dkfz.de/> www.dkfz.de<http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt. <Data_library_error.png>

Dear all, I got it to work again. I just added: user_library_import_symlink_whitelist = "/symlink_target_folder"/ to the galaxy.ini (we'll switch to the .yml next update). It got recognized and the symlinks can be used again. Best regards, Christopher On 11/28/18 9:25 AM, Previti wrote:

Thanks for the infos, I guess we will just have to check everything again. I wonder, did anything change from 17.09 to 18.05 that would affect the way the system resolves the email-addresses?

We'll let everybody know if we have a solution/cause for the problem.

Cheers and thanks again, Christopher

On 11/27/18 11:00 PM, Youssef GHORBAL wrote:

...

Hello,

As per this https://github.com/galaxyproject/galaxy/blob/9a6fddcdca7042e5e453bdd235ed8c5... The message you are seeing "You do not have any files in your user directory. Use FTP to upload there" has nothing to do with user_library_import_symlink_whitelist The symlink control (involving user_library_import_symlink_whitelist) is done a little bit later in the code.

The test code is rather straightforward it does an os.path.exists on the path contact of the user_library_import_dir setting and the user's email address.

Option 1: Galaxy is not resolving the correct email address for your account, I don't see why this may change after upgrade. But it's worth double checking (especially the case) Option 2: There is an issue with permissions on the symlink target. I've checked python will return false on os.path.exists if the symlink targets a directory not accessible by the user. For me you need to double check the permissions on your server. Maybe with a shell access as the galaxy@AD.DKFZ-HEIDELBERG.DE <mailto:galaxy@AD.DKFZ-HEIDELBERG.DE> user, you can try access one of the symlinks and see how it goes. You may also check that the Galaxy service is actually running under the intended identity (galaxy@AD.DKFZ-HEIDELBERG.DE <mailto:galaxy@AD.DKFZ-HEIDELBERG.DE>)

Youssef Ghorbal IIS / DSI Institut Pasteur ---------------------------------

...

On 27 Nov 2018, at 11:43, Previti <christopher.previti@dkfz-heidelberg.de <mailto:christopher.previti@dkfz-heidelberg.de>> wrote:

Dear Youssef, Thanks for answering, the target permissions have not changed and are defined via NFS4 ACLs:

A::OWNER@:rwaDxtTnNcCy A::GROUP@:rwaDxtTnNcy *A:fdi:galaxy@AD.DKFZ-HEIDELBERG.DE:rwaDxtTnNcy* A:fdig:W190-galaxy-admins@AD.DKFZ-HEIDELBERG.DE:rwaDxtTnNcy

In this case the galaxy user (i.e. the user starting the process) has full access to the share. In our instance the "user directories" are just symbolic links to the target folders mention above. When I try to import data from a user directory/NFS4 share into a data library we get the following error: 

"You do not habe any files in your user directory. Use FTP to upload there." (I attached a screenshot)

We still use galaxy.ini to configure our instance, in the galaxy.yml I find the following parameter:

# For security reasons, users may not import any files that actually   # lie outside of their `user_library_import_dir` (e.g. using symbolic   # links). A list of directories can be allowed by setting the   # following option (the list is comma-separated). Be aware that *any*   # user with library import permissions can import from anywhere in   # these directories (assuming they are able to create symlinks to   # them).   #user_library_import_symlink_whitelist: null

Would implementing this solve my issue?

Cheers, Christopher

On 11/26/18 4:36 PM, Youssef GHORBAL wrote:

...

Hello,

What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ? By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance)

Youssef Ghorbal IIS / DSI Institut Pasteur -----------------------

...

On 26 Nov 2018, at 15:23, Previti <christopher.previti@dkfz-heidelberg.de <mailto:christopher.previti@dkfz-heidelberg.de>> wrote:

Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses. This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 christopher.previti@dkfz-heidelberg.de -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy") directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05? Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher

-- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician

German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661

christopher.previti@dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client.  To manage your subscriptions to this and other Galaxy lists, please use the interface at:  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:  http://galaxyproject.org/search/

-- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician

German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661

christopher.previti@dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

<Data_library_error.png>

-- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician

German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661

christopher.previti@dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/

-- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti@dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.