BioBlend AND Galaxy: Problem using upload_file_from_server function with an associated role
Hello to all, I'm trying to use the function "/upload_file_from_server/" from the galaxy libraries api of Bioblend. It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library. Thus, it is how ""/upload_file_from_server/" is working at the moment. As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side. could somebody help me with that ? Happy new Year !!! and Have a nice day, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
I am not sure I understand the problem. Can you clarify a few points: Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing? If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind? elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API? On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing? Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance. The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script) For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use. Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable". However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone. #giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226") To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error. giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr') and I get the following error:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server
return self._do_upload(**vars)
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload
files_attached=files_attached)
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post
r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached)
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request
r.status_code, body=r.text) # @see self.body for HTTP response body
ConnectionError: Unexpected response from galaxy: 500: <html>
<head><title>Internal Server Error</title></head>
<body>
<h1>Internal Server Error</h1>
<p>The server has either erred or is incapable of performing
the requested operation.
<br/>
<!-- --></p>
<hr noshade>
<div align="right">WSGI Server</div>
</body>
</html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key) I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work. Tell me if you need more information. I'll look further on monday but i'm kind of stuck. Thank you for your help, Cheers, -- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia Le 04/01/14 22:10, Olivia Doppelt a écrit :
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing? Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance.
The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script)
For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use.
Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable".
However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone.
#giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226")
To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error.
giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr')
and I get the following error:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server
return self._do_upload(**vars)
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload
files_attached=files_attached)
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post
r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached)
File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request
r.status_code, body=r.text) # @see self.body for HTTP response body
ConnectionError: Unexpected response from galaxy: 500: <html>
<head><title>Internal Server Error</title></head>
<body>
<h1>Internal Server Error</h1>
<p>The server has either erred or is incapable of performing
the requested operation.
<br/>
<!-- --></p>
<hr noshade>
<div align="right">WSGI Server</div>
</body>
</html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key)
I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work.
Tell me if you need more information. I'll look further on monday but i'm kind of stuck.
Thank you for your help,
Cheers,
-- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
Hello Olivia, Great, thanks for the clarification! It took me a little bit, but I have caught up. First - you don't need to justify using bioblend over the provided scripts in galaxy distribution. Anything non-trivial should be using bioblend (if it weren't going to make me very unpopular I would just delete scripts/api out of Galaxy). There are a couple problems here - but the main one is simply that the roles param is taking in something odd as an argument - namely the unencoded integer id of the role as a string - it probably makes perfect sense for the UI to deal with things this way but it doesn't fit well with the API. Here (https://github.com/jmchilton/galaxy-central/commit/11377929a679fb3391481e352...) is a patch you can apply to Galaxy to make it work the way you are trying to get it to work (allowing roles to take in a single role name). Just apply that to Galaxy and I imagine your example will work as is. However, this is not "the right" way for it to work however so I am not going to apply it Galaxy directly but it is a very isolated change you can apply to your Galaxy instance if your comfortable with that. In an ideal implementation - there are many different permission types that should be exposed and the API should be consuming encoded role ids (as obtained from /api/roles) - but the backend isn't implemented for this first part and bioblend doesn't have a roles client for the second. I have created a Trello card for this feature: https://trello.com/c/qsQHnFlG. Let me know if you have any additional questions or if this is unclear in any way. -John On Sun, Jan 5, 2014 at 3:35 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia
Le 04/01/14 22:10, Olivia Doppelt a écrit :
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing?
Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance.
The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script)
For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use.
Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable".
However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone.
#giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226")
To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error.
giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr')
and I get the following error:
Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server return self._do_upload(**vars) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: <html> <head><title>Internal Server Error</title></head> <body> <h1>Internal Server Error</h1> <p>The server has either erred or is incapable of performing the requested operation. <br/> <!-- --></p> <hr noshade> <div align="right">WSGI Server</div> </body> </html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key)
I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work.
Tell me if you need more information. I'll look further on monday but i'm kind of stuck.
Thank you for your help,
Cheers,
-- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
Hello John, Thank you very much for the patch. It works now with the argument 'roles'. I tested with the wrong role in order to be sure that it filters when the role is not the right one and it does get filtered :) However, I need to add a few lines to the patch in order to avoid the use of this method without a defined role. basically, the complete patch is: + roles = payload.get("roles", None) + if roles: + roles = util.listify(roles) + + def to_id_as_str(role): + role = role.replace("__at__", "@") + if "@" in role: + role = str( trans.sa_session.query( trans.app.model.Role ).filter( trans.app.model.Role.table.c.name == role ).first().id ) + else: + role = str( trans.security.decode_id( role ) ) + return role + + payload["roles"] = map( to_id_as_str, roles ) + + else: + trans.response.status = 400 + return "Missing required roles definition" I don't know if you (could/want to) add this to your commit. Thank you very much for your help. -- Olivia Le 06/01/14 16:53, John Chilton a écrit :
Hello Olivia,
Great, thanks for the clarification! It took me a little bit, but I have caught up. First - you don't need to justify using bioblend over the provided scripts in galaxy distribution. Anything non-trivial should be using bioblend (if it weren't going to make me very unpopular I would just delete scripts/api out of Galaxy).
There are a couple problems here - but the main one is simply that the roles param is taking in something odd as an argument - namely the unencoded integer id of the role as a string - it probably makes perfect sense for the UI to deal with things this way but it doesn't fit well with the API. Here (https://github.com/jmchilton/galaxy-central/commit/11377929a679fb3391481e352...) is a patch you can apply to Galaxy to make it work the way you are trying to get it to work (allowing roles to take in a single role name). Just apply that to Galaxy and I imagine your example will work as is.
However, this is not "the right" way for it to work however so I am not going to apply it Galaxy directly but it is a very isolated change you can apply to your Galaxy instance if your comfortable with that. In an ideal implementation - there are many different permission types that should be exposed and the API should be consuming encoded role ids (as obtained from /api/roles) - but the backend isn't implemented for this first part and bioblend doesn't have a roles client for the second.
I have created a Trello card for this feature: https://trello.com/c/qsQHnFlG.
Let me know if you have any additional questions or if this is unclear in any way.
-John
On Sun, Jan 5, 2014 at 3:35 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia
Le 04/01/14 22:10, Olivia Doppelt a écrit :
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing?
Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance.
The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script)
For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use.
Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable".
However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone.
#giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226")
To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error.
giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr')
and I get the following error:
Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server return self._do_upload(**vars) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: <html> <head><title>Internal Server Error</title></head> <body> <h1>Internal Server Error</h1> <p>The server has either erred or is incapable of performing the requested operation. <br/> <!-- --></p> <hr noshade> <div align="right">WSGI Server</div> </body> </html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key)
I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work.
Tell me if you need more information. I'll look further on monday but i'm kind of stuck.
Thank you for your help,
Cheers,
-- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
Dear John, Unfortunatly, I have another problem. The galaxy version in use for our institute was last updated on December 10th. Weirdly, when i put the patch in the library_contents.py file, I get another error (which i don't have on my local version) Actually, it never goes in the code i patched... and it doesn't underdand the string which is the role. the error in the following: _____
157.99.60.187 - - [07/Jan/2014:16:36:31 +0200] "GET /api/libraries/79449a84684b4136/contents?key=API_key HTTP/1.1" 200 - "-" "python-requests/2.0.1 CPython/2.7.5+ Linux/3.11.0-12-generic" sqlalchemy.pool.QueuePool WARNING 2014-01-07 16:36:32,212 Error closing cursor: current transaction is aborted, commands ignored until end of transaction block
galaxy.web.framework ERROR 2014-01-07 16:36:32,212 Uncaught exception in exposed API method: Traceback (most recent call last): File "/xxxx/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py", line 197, in decorator rval = func( self, trans, *args, **kwargs) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/library_contents.py", line 183, in create role = str( trans.security.decode_id( role ) ) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py", line 843, in upload_library_dataset trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/security/__init__.py", line 939, in derive_roles_from_access in_roles = [ self.sa_session.query( self.model.Role ).get( x ) for x in listify( kwd.get( k + '_in', [] ) ) ] File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 775, in get return self._load_on_ident(key) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2512, in _load_on_ident return q.one() File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2184, in one ret = list(self) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2227, in __iter__ return self._execute_and_instances(context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) DataError: (DataError) invalid input syntax for integer: "odoppelt__at__pasteur.fr" LINE 3: WHERE role.id = E'odoppelt__at__pasteur.fr' ^ 'SELECT role.id AS role_id, role.create_time AS role_create_time, role.update_time AS role_update_time, role.name AS role_name, role.description AS role_description, role.type AS role_type, role.deleted AS role_deleted \nFROM role \nWHERE role.id = %(param_1)s' {'param_1': 'odoppelt__at__pasteur.fr'}
_____ Le 06/01/14 18:34, Olivia Doppelt a écrit :
Hello John,
Thank you very much for the patch. It works now with the argument 'roles'.
I tested with the wrong role in order to be sure that it filters when the role is not the right one and it does get filtered :)
However, I need to add a few lines to the patch in order to avoid the use of this method without a defined role.
basically, the complete patch is:
+ roles = payload.get("roles", None) + if roles: + roles = util.listify(roles) + + def to_id_as_str(role): + role = role.replace("__at__", "@") + if "@" in role: + role = str( trans.sa_session.query( trans.app.model.Role ).filter( trans.app.model.Role.table.c.name == role ).first().id ) + else: + role = str( trans.security.decode_id( role ) ) + return role + + payload["roles"] = map( to_id_as_str, roles ) + + else: + trans.response.status = 400 + return "Missing required roles definition"
I don't know if you (could/want to) add this to your commit.
Thank you very much for your help. -- Olivia
Le 06/01/14 16:53, John Chilton a écrit :
Hello Olivia,
Great, thanks for the clarification! It took me a little bit, but I have caught up. First - you don't need to justify using bioblend over the provided scripts in galaxy distribution. Anything non-trivial should be using bioblend (if it weren't going to make me very unpopular I would just delete scripts/api out of Galaxy).
There are a couple problems here - but the main one is simply that the roles param is taking in something odd as an argument - namely the unencoded integer id of the role as a string - it probably makes perfect sense for the UI to deal with things this way but it doesn't fit well with the API. Here (https://github.com/jmchilton/galaxy-central/commit/11377929a679fb3391481e352...) is a patch you can apply to Galaxy to make it work the way you are trying to get it to work (allowing roles to take in a single role name). Just apply that to Galaxy and I imagine your example will work as is.
However, this is not "the right" way for it to work however so I am not going to apply it Galaxy directly but it is a very isolated change you can apply to your Galaxy instance if your comfortable with that. In an ideal implementation - there are many different permission types that should be exposed and the API should be consuming encoded role ids (as obtained from /api/roles) - but the backend isn't implemented for this first part and bioblend doesn't have a roles client for the second.
I have created a Trello card for this feature:https://trello.com/c/qsQHnFlG.
Let me know if you have any additional questions or if this is unclear in any way.
-John
On Sun, Jan 5, 2014 at 3:35 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia
Le 04/01/14 22:10, Olivia Doppelt a écrit :
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing?
Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance.
The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script)
For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use.
Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable".
However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone.
#giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226")
To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error.
giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr')
and I get the following error:
Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server return self._do_upload(**vars) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: <html> <head><title>Internal Server Error</title></head> <body> <h1>Internal Server Error</h1> <p>The server has either erred or is incapable of performing the requested operation. <br/> <!-- --></p> <hr noshade> <div align="right">WSGI Server</div> </body> </html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key)
I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work.
Tell me if you need more information. I'll look further on monday but i'm kind of stuck.
Thank you for your help,
Cheers,
-- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
I am stumped, I am not sure how this would happen. How did you apply the patch against the -dist stable code? Can you send me the full contents of the patched ..api/library_contents.py file? -John On Tue, Jan 7, 2014 at 9:48 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Dear John,
Unfortunatly, I have another problem.
The galaxy version in use for our institute was last updated on December 10th.
Weirdly, when i put the patch in the library_contents.py file, I get another error (which i don't have on my local version)
Actually, it never goes in the code i patched... and it doesn't underdand the string which is the role.
the error in the following: _____
157.99.60.187 - - [07/Jan/2014:16:36:31 +0200] "GET /api/libraries/79449a84684b4136/contents?key=API_key HTTP/1.1" 200 - "-" "python-requests/2.0.1 CPython/2.7.5+ Linux/3.11.0-12-generic" sqlalchemy.pool.QueuePool WARNING 2014-01-07 16:36:32,212 Error closing cursor: current transaction is aborted, commands ignored until end of transaction block
galaxy.web.framework ERROR 2014-01-07 16:36:32,212 Uncaught exception in exposed API method:
Traceback (most recent call last): File "/xxxx/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py", line 197, in decorator rval = func( self, trans, *args, **kwargs) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/library_contents.py", line 183, in create
role = str( trans.security.decode_id( role ) ) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py", line 843, in upload_library_dataset
trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/security/__init__.py", line 939, in derive_roles_from_access in_roles = [ self.sa_session.query( self.model.Role ).get( x ) for x in listify( kwd.get( k + '_in', [] ) ) ] File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 775, in get return self._load_on_ident(key) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2512, in _load_on_ident return q.one() File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2184, in one ret = list(self) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2227, in __iter__ return self._execute_and_instances(context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) DataError: (DataError) invalid input syntax for integer: "odoppelt__at__pasteur.fr" LINE 3: WHERE role.id = E'odoppelt__at__pasteur.fr' ^ 'SELECT role.id AS role_id, role.create_time AS role_create_time, role.update_time AS role_update_time, role.name AS role_name, role.description AS role_description, role.type AS role_type, role.deleted AS role_deleted \nFROM role \nWHERE role.id = %(param_1)s' {'param_1': 'odoppelt__at__pasteur.fr'}
_____
Le 06/01/14 18:34, Olivia Doppelt a écrit :
Hello John,
Thank you very much for the patch. It works now with the argument 'roles'.
I tested with the wrong role in order to be sure that it filters when the role is not the right one and it does get filtered :)
However, I need to add a few lines to the patch in order to avoid the use of this method without a defined role.
basically, the complete patch is:
+ roles = payload.get("roles", None) + if roles: + roles = util.listify(roles) + + def to_id_as_str(role): + role = role.replace("__at__", "@") + if "@" in role: + role = str( trans.sa_session.query( trans.app.model.Role ).filter( trans.app.model.Role.table.c.name == role ).first().id ) + else: + role = str( trans.security.decode_id( role ) ) + return role + + payload["roles"] = map( to_id_as_str, roles ) + + else: + trans.response.status = 400 + return "Missing required roles definition"
I don't know if you (could/want to) add this to your commit.
Thank you very much for your help. -- Olivia
Le 06/01/14 16:53, John Chilton a écrit :
Hello Olivia,
Great, thanks for the clarification! It took me a little bit, but I have caught up. First - you don't need to justify using bioblend over the provided scripts in galaxy distribution. Anything non-trivial should be using bioblend (if it weren't going to make me very unpopular I would just delete scripts/api out of Galaxy).
There are a couple problems here - but the main one is simply that the roles param is taking in something odd as an argument - namely the unencoded integer id of the role as a string - it probably makes perfect sense for the UI to deal with things this way but it doesn't fit well with the API. Here (https://github.com/jmchilton/galaxy-central/commit/11377929a679fb3391481e352...) is a patch you can apply to Galaxy to make it work the way you are trying to get it to work (allowing roles to take in a single role name). Just apply that to Galaxy and I imagine your example will work as is.
However, this is not "the right" way for it to work however so I am not going to apply it Galaxy directly but it is a very isolated change you can apply to your Galaxy instance if your comfortable with that. In an ideal implementation - there are many different permission types that should be exposed and the API should be consuming encoded role ids (as obtained from /api/roles) - but the backend isn't implemented for this first part and bioblend doesn't have a roles client for the second.
I have created a Trello card for this feature: https://trello.com/c/qsQHnFlG.
Let me know if you have any additional questions or if this is unclear in any way.
-John
On Sun, Jan 5, 2014 at 3:35 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia
Le 04/01/14 22:10, Olivia Doppelt a écrit :
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing?
Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance.
The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script)
For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use.
Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable".
However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone.
#giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226")
To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error.
giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr')
and I get the following error:
Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server return self._do_upload(**vars) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: <html> <head><title>Internal Server Error</title></head> <body> <h1>Internal Server Error</h1> <p>The server has either erred or is incapable of performing the requested operation. <br/> <!-- --></p> <hr noshade> <div align="right">WSGI Server</div> </body> </html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key)
I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work.
Tell me if you need more information. I'll look further on monday but i'm kind of stuck.
Thank you for your help,
Cheers,
-- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
Dear John, Sorry to have disturb you with that problem actually it was linked to a local fact. We had in the past a problem with the api authentification so as a work around, we have 2 galaxy servers. one for galaxy and another for the api. Actually it is quite new (a few weeks), so i was not very familiar with it. The problem was because i only reboot galaxy and not galaxy api... so my changes were not taken into account. So your patch now works !! Thank you again, we can go on with our project using Bioblend. Have a nice day ! - Olivia Le 10/01/14 15:35, John Chilton a écrit :
I am stumped, I am not sure how this would happen. How did you apply the patch against the -dist stable code? Can you send me the full contents of the patched ..api/library_contents.py file?
-John
On Tue, Jan 7, 2014 at 9:48 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Dear John,
Unfortunatly, I have another problem.
The galaxy version in use for our institute was last updated on December 10th.
Weirdly, when i put the patch in the library_contents.py file, I get another error (which i don't have on my local version)
Actually, it never goes in the code i patched... and it doesn't underdand the string which is the role.
the error in the following: _____
157.99.60.187 - - [07/Jan/2014:16:36:31 +0200] "GET /api/libraries/79449a84684b4136/contents?key=API_key HTTP/1.1" 200 - "-" "python-requests/2.0.1 CPython/2.7.5+ Linux/3.11.0-12-generic" sqlalchemy.pool.QueuePool WARNING 2014-01-07 16:36:32,212 Error closing cursor: current transaction is aborted, commands ignored until end of transaction block
galaxy.web.framework ERROR 2014-01-07 16:36:32,212 Uncaught exception in exposed API method:
Traceback (most recent call last): File "/xxxx/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py", line 197, in decorator rval = func( self, trans, *args, **kwargs) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/library_contents.py", line 183, in create
role = str( trans.security.decode_id( role ) ) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py", line 843, in upload_library_dataset
trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) File "/xxxx/galaxy/galaxy-dist/lib/galaxy/security/__init__.py", line 939, in derive_roles_from_access in_roles = [ self.sa_session.query( self.model.Role ).get( x ) for x in listify( kwd.get( k + '_in', [] ) ) ] File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 775, in get return self._load_on_ident(key) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2512, in _load_on_ident return q.one() File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2184, in one ret = list(self) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2227, in __iter__ return self._execute_and_instances(context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/xxxx/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) DataError: (DataError) invalid input syntax for integer: "odoppelt__at__pasteur.fr" LINE 3: WHERE role.id = E'odoppelt__at__pasteur.fr' ^ 'SELECT role.id AS role_id, role.create_time AS role_create_time, role.update_time AS role_update_time, role.name AS role_name, role.description AS role_description, role.type AS role_type, role.deleted AS role_deleted \nFROM role \nWHERE role.id = %(param_1)s' {'param_1': 'odoppelt__at__pasteur.fr'}
_____
Le 06/01/14 18:34, Olivia Doppelt a écrit :
Hello John,
Thank you very much for the patch. It works now with the argument 'roles'.
I tested with the wrong role in order to be sure that it filters when the role is not the right one and it does get filtered :)
However, I need to add a few lines to the patch in order to avoid the use of this method without a defined role.
basically, the complete patch is:
+ roles = payload.get("roles", None) + if roles: + roles = util.listify(roles) + + def to_id_as_str(role): + role = role.replace("__at__", "@") + if "@" in role: + role = str( trans.sa_session.query( trans.app.model.Role ).filter( trans.app.model.Role.table.c.name == role ).first().id ) + else: + role = str( trans.security.decode_id( role ) ) + return role + + payload["roles"] = map( to_id_as_str, roles ) + + else: + trans.response.status = 400 + return "Missing required roles definition"
I don't know if you (could/want to) add this to your commit.
Thank you very much for your help. -- Olivia
Le 06/01/14 16:53, John Chilton a écrit :
Hello Olivia,
Great, thanks for the clarification! It took me a little bit, but I have caught up. First - you don't need to justify using bioblend over the provided scripts in galaxy distribution. Anything non-trivial should be using bioblend (if it weren't going to make me very unpopular I would just delete scripts/api out of Galaxy).
There are a couple problems here - but the main one is simply that the roles param is taking in something odd as an argument - namely the unencoded integer id of the role as a string - it probably makes perfect sense for the UI to deal with things this way but it doesn't fit well with the API. Here (https://github.com/jmchilton/galaxy-central/commit/11377929a679fb3391481e352...) is a patch you can apply to Galaxy to make it work the way you are trying to get it to work (allowing roles to take in a single role name). Just apply that to Galaxy and I imagine your example will work as is.
However, this is not "the right" way for it to work however so I am not going to apply it Galaxy directly but it is a very isolated change you can apply to your Galaxy instance if your comfortable with that. In an ideal implementation - there are many different permission types that should be exposed and the API should be consuming encoded role ids (as obtained from /api/roles) - but the backend isn't implemented for this first part and bioblend doesn't have a roles client for the second.
I have created a Trello card for this feature: https://trello.com/c/qsQHnFlG.
Let me know if you have any additional questions or if this is unclear in any way.
-John
On Sun, Jan 5, 2014 at 3:35 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia
Le 04/01/14 22:10, Olivia Doppelt a écrit :
Le 03/01/14 15:44, John Chilton a écrit :
I am not sure I understand the problem. Can you clarify a few points:
Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing?
Dear John,
In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our "user_library_import_dir" and one in an "output" directory to enable the users to export their results from the instance.
The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script)
For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use.
Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are "uploadable".
However, using bioblend function from libraries class "upload_file_from_server", a user with its key can upload within its galaxy library, the data of anyone.
#giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxx/links/otherUser@pasteur.fr/","F0045a9ffd77d1226")
To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error.
giOlivia.libraries.upload_file_from_server("OLIVIA_API_key","/xxxxxxxx/links/olivia@pasteur.fr/", "F0045a9ffd77d1226", roles='olivia@pasteur.fr')
and I get the following error:
Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 242, in upload_file_from_server return self._do_upload(**vars) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py", line 190, in _do_upload files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py", line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File "/usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py", line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: <html> <head><title>Internal Server Error</title></head> <body> <h1>Internal Server Error</h1> <p>The server has either erred or is incapable of performing the requested operation. <br/> <!-- --></p> <hr noshade> <div align="right">WSGI Server</div> </body> </html>
If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind?
elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message
the api does not produce any error when the api key corresponds to the user which tries to import its own data.
Has your institute made modifications Galaxy to enforce this constraint - ("it can't be possible that any user may upload any other user's data in their Galaxy Data library")? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API?
I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key)
I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work.
Tell me if you need more information. I'll look further on monday but i'm kind of stuck.
Thank you for your help,
Cheers,
-- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur
On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt <olivia.doppelt@pasteur.fr> wrote:
Hello to all,
I'm trying to use the function "upload_file_from_server" from the galaxy libraries api of Bioblend.
It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library.
Thus, it is how ""upload_file_from_server" is working at the moment.
As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side.
could somebody help me with that ?
Happy new Year !!! and Have a nice day,
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
-- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur
participants (2)
-
John Chilton -
Olivia Doppelt