Hello,
Is there any way to provide a protected resource link to the external application? My tool is supposed to call a Web service via SSL which should access to the Galaxy resources. In ideal it would be something like ftp://user:password@mygalaxy.net/fasta.gz
The Web service then downloads the resource and do the computation.
Since user is already provided the login to galaxy, I do not want to ask the password again on a tool page.
Sincerely,
Dmitry
Hello Dmitry,
This use case isn't really addressed by Galaxy currently.
There is not a generic way of doing grabbing the users credentials like this or producing URLs from inside the tool. This depends on a lot of different things - how Galaxy's proxy is configured, etc... so in many (all?) cases Galaxy would not be able to fetch the users raw credentials. Your best bet is going to be to use the Galaxy's API - tools can fetch a user's API key and you can use Galaxy's API to build a URL that includes this key.
Your users will all need API keys set to use the tool. There have been some other galaxy-dev discussions about using the API from tools - http://osdir.com/ml/galaxy-development-source-control/2013-12/msg00105.html. In general I would discourage this because it is difficult to do in a generic way - for instance there is no way for the tool to accurately determine what the URL of Galaxy should be - but it should be possible to tailor something to your setup if absolutely needed.
If you are really keen to use the Galaxy API to do this and you understand the limitations - we can try to work through the exact details.
If it is at all possible to send the data to the web service instead of sending a reference - the genomespace exporter (https://bitbucket.org/galaxy/galaxy-central/src/74b6e23ed7882f3c091d2b66ce85...) might be a good example of how to do this in a more generic fashion.
-John
On Tue, Apr 29, 2014 at 11:16 AM, Dmitry redmitry@list.ru wrote:
Hello,
Is there any way to provide a protected resource link to the external application? My tool is supposed to call a Web service via SSL which should access to the Galaxy resources. In ideal it would be something like ftp://user:password@mygalaxy.net/fasta.gz
The Web service then downloads the resource and do the computation.
Since user is already provided the login to galaxy, I do not want to ask the password again on a tool page.
Sincerely,
Dmitry ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello John,
If it is at all possible to send the data to the web service instead ...
Unfortunately, I cannot push the data to the web service, because this is the BES one and the interface is defined in the spec.
As far as I understand reading the mailing list there is no way to pull data through FTP (only upload).
The API key looks interesting even I do not completely understand it.
Is it possible to manipulate the key(s) programmatically? It would be great to generate a temporal key, execute the service and remove the key after that.
In general I would discourage this because it is difficult to do in a generic way ...
I understand that the problem is when Galaxy is behind a proxy. In my case I control the URL for the galaxy.
${ __app__.model.User.get( $__user_id__ ).api_keys[0].key }
I absolutely lost here... should I know $__user_id__ or this is a variable already known to the tool?
Thank you very much for the help,
Dmitry
On 5/6/2014 4:39 AM, John Chilton wrote:
Hello Dmitry,
This use case isn't really addressed by Galaxy currently.
There is not a generic way of doing grabbing the users credentials like this or producing URLs from inside the tool. This depends on a lot of different things - how Galaxy's proxy is configured, etc... so in many (all?) cases Galaxy would not be able to fetch the users raw credentials. Your best bet is going to be to use the Galaxy's API - tools can fetch a user's API key and you can use Galaxy's API to build a URL that includes this key.
Your users will all need API keys set to use the tool. There have been some other galaxy-dev discussions about using the API from tools - http://osdir.com/ml/galaxy-development-source-control/2013-12/msg00105.html. In general I would discourage this because it is difficult to do in a generic way - for instance there is no way for the tool to accurately determine what the URL of Galaxy should be - but it should be possible to tailor something to your setup if absolutely needed.
If you are really keen to use the Galaxy API to do this and you understand the limitations - we can try to work through the exact details.
If it is at all possible to send the data to the web service instead of sending a reference - the genomespace exporter (https://bitbucket.org/galaxy/galaxy-central/src/74b6e23ed7882f3c091d2b66ce85...) might be a good example of how to do this in a more generic fashion.
-John
On Tue, Apr 29, 2014 at 11:16 AM, Dmitry redmitry@list.ru wrote:
Hello,
Is there any way to provide a protected resource link to the external application? My tool is supposed to call a Web service via SSL which should access to the Galaxy resources. In ideal it would be something like ftp://user:password@mygalaxy.net/fasta.gz
The Web service then downloads the resource and do the computation.
Since user is already provided the login to galaxy, I do not want to ask the password again on a tool page.
Sincerely,
Dmitry ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello Dmitry,
Sorry the delayed response. It occurs to me that data source tools may be an alternative I should have pointed you to right away - https://wiki.galaxyproject.org/Admin/Internals/DataSources. I am not sure they are a perfect fit but regular tools are not a perfect fit either.
On Tue, May 6, 2014 at 4:26 AM, Dmitry redmitry@list.ru wrote:
Hello John,
If it is at all possible to send the data to the web service instead ...
Unfortunately, I cannot push the data to the web service, because this is the BES one and the interface is defined in the spec.
As far as I understand reading the mailing list there is no way to pull data through FTP (only upload).
This is correct.
The API key looks interesting even I do not completely understand it.
Is it possible to manipulate the key(s) programmatically? It would be great to generate a temporal key, execute the service and remove the key after that.
When you phrase the problem like this it occurs to me I recently added this functionality to Galaxy. The LWR (a tool to remotely execute Galaxy tools) can now be configured to pull files from and send files to Galaxy with single purpose keys tied to particular jobs. The details are outlined in the following pull request (https://bitbucket.org/galaxy/galaxy-central/pull-request/327/implement-job-f...). Piggy backing on this is probably the right way to go because you don't have to worry about whether user's have API keys defined - jobs have implicit keys not users.
If you wanted to go down this road - you will need to throw Galaxy's job object or at least the id into the context Galaxy uses to evaluate tool xml files. This patch https://gist.github.com/jmchilton/ddec237d2ad63821addc for instances SHOULD give tools access to a $__job_id__ variable (haven't tested it).
Then you should be able to construct URLs inside of your Galaxy XML for your inputs and outputs as follows:
#set $encode_job_id = ${__app__.security.encode_id( $__job_id__ )} #set $job_key = ${__app__.security.encode_id( $__job_id__, kind="jobs_files" ) } #set $input_path = str($input) --url "http://mygalaxy.com/api/files/$encode_job_id/files?job_key=$job_key&path..."
#set $encode_job_id = ${__app__.security.encode_id( $__job_id__ )} #set $job_key = ${__app__.security.encode_id( $__job_id__, kind="jobs_files" ) } #set $output_path = str($output) --output_url "http://mygalaxy.com/api/files/$encode_job_id/files?job_key=$job_key&path..."
This API endpoint I believe will respond to GETs for the inputs and POSTs for the output.
There are probably some syntax errors in above tool/cheetah snipplets (Galaxy doesn't formally support this, I am just giving you some informal advice on how one might proceed).
In general I would discourage this because it is difficult to do in a generic way ...
I understand that the problem is when Galaxy is behind a proxy. In my case I control the URL for the galaxy.
${ __app__.model.User.get( $__user_id__ ).api_keys[0].key }
Again, wouldn't need this if you are using the newer job files API endpoint.
Obviously this e-mail is very informal and hand-wavy because I am likely too busy to walk through and verify all of these steps. But I have created a Trello card (https://trello.com/c/JjIF9l4h) to keep track of this feature request and outline this approach, I will try to find some time to think about this use case some more in the future.
Sorry I don't have more time to give you a more complete response. Galaxy is generally very focused on wrapping command-line utilities for analysis and it doesn't necessarily target web services like this very well.
-John
I absolutely lost here... should I know $__user_id__ or this is a variable already known to the tool?
Thank you very much for the help,
Dmitry
On 5/6/2014 4:39 AM, John Chilton wrote:
Hello Dmitry,
This use case isn't really addressed by Galaxy currently.
There is not a generic way of doing grabbing the users credentials like this or producing URLs from inside the tool. This depends on a lot of different things - how Galaxy's proxy is configured, etc... so in many (all?) cases Galaxy would not be able to fetch the users raw credentials. Your best bet is going to be to use the Galaxy's API - tools can fetch a user's API key and you can use Galaxy's API to build a URL that includes this key.
Your users will all need API keys set to use the tool. There have been some other galaxy-dev discussions about using the API from tools -
http://osdir.com/ml/galaxy-development-source-control/2013-12/msg00105.html. In general I would discourage this because it is difficult to do in a generic way - for instance there is no way for the tool to accurately determine what the URL of Galaxy should be - but it should be possible to tailor something to your setup if absolutely needed.
If you are really keen to use the Galaxy API to do this and you understand the limitations - we can try to work through the exact details.
If it is at all possible to send the data to the web service instead of sending a reference - the genomespace exporter
(https://bitbucket.org/galaxy/galaxy-central/src/74b6e23ed7882f3c091d2b66ce85...) might be a good example of how to do this in a more generic fashion.
-John
On Tue, Apr 29, 2014 at 11:16 AM, Dmitry redmitry@list.ru wrote:
Hello,
Is there any way to provide a protected resource link to the external application? My tool is supposed to call a Web service via SSL which should access to the Galaxy resources. In ideal it would be something like ftp://user:password@mygalaxy.net/fasta.gz
The Web service then downloads the resource and do the computation.
Since user is already provided the login to galaxy, I do not want to ask the password again on a tool page.
Sincerely,
Dmitry ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello John,
Thank you very much for the very extended response. It seems I have enough reading to be calm for a month :-).
Thank you again,
Dmitry
On 5/19/2014 4:42 PM, John Chilton wrote:
Hello Dmitry,
Sorry the delayed response. It occurs to me that data source tools may be an alternative I should have pointed you to right away - https://wiki.galaxyproject.org/Admin/Internals/DataSources. I am not sure they are a perfect fit but regular tools are not a perfect fit either.
On Tue, May 6, 2014 at 4:26 AM, Dmitry redmitry@list.ru wrote:
Hello John,
If it is at all possible to send the data to the web service instead ...
Unfortunately, I cannot push the data to the web service, because this is the BES one and the interface is defined in the spec.
As far as I understand reading the mailing list there is no way to pull data through FTP (only upload).
This is correct.
The API key looks interesting even I do not completely understand it.
Is it possible to manipulate the key(s) programmatically? It would be great to generate a temporal key, execute the service and remove the key after that.
When you phrase the problem like this it occurs to me I recently added this functionality to Galaxy. The LWR (a tool to remotely execute Galaxy tools) can now be configured to pull files from and send files to Galaxy with single purpose keys tied to particular jobs. The details are outlined in the following pull request (https://bitbucket.org/galaxy/galaxy-central/pull-request/327/implement-job-f...). Piggy backing on this is probably the right way to go because you don't have to worry about whether user's have API keys defined - jobs have implicit keys not users.
If you wanted to go down this road - you will need to throw Galaxy's job object or at least the id into the context Galaxy uses to evaluate tool xml files. This patch https://gist.github.com/jmchilton/ddec237d2ad63821addc for instances SHOULD give tools access to a $__job_id__ variable (haven't tested it).
Then you should be able to construct URLs inside of your Galaxy XML for your inputs and outputs as follows:
#set $encode_job_id = ${__app__.security.encode_id( $__job_id__ )} #set $job_key = ${__app__.security.encode_id( $__job_id__, kind="jobs_files" ) } #set $input_path = str($input) --url "http://mygalaxy.com/api/files/$encode_job_id/files?job_key=$job_key&path..."
#set $encode_job_id = ${__app__.security.encode_id( $__job_id__ )} #set $job_key = ${__app__.security.encode_id( $__job_id__, kind="jobs_files" ) } #set $output_path = str($output) --output_url "http://mygalaxy.com/api/files/$encode_job_id/files?job_key=$job_key&path..."
This API endpoint I believe will respond to GETs for the inputs and POSTs for the output.
There are probably some syntax errors in above tool/cheetah snipplets (Galaxy doesn't formally support this, I am just giving you some informal advice on how one might proceed).
In general I would discourage this because it is difficult to do in a generic way ...
I understand that the problem is when Galaxy is behind a proxy. In my case I control the URL for the galaxy.
${ __app__.model.User.get( $__user_id__ ).api_keys[0].key }
Again, wouldn't need this if you are using the newer job files API endpoint.
Obviously this e-mail is very informal and hand-wavy because I am likely too busy to walk through and verify all of these steps. But I have created a Trello card (https://trello.com/c/JjIF9l4h) to keep track of this feature request and outline this approach, I will try to find some time to think about this use case some more in the future.
Sorry I don't have more time to give you a more complete response. Galaxy is generally very focused on wrapping command-line utilities for analysis and it doesn't necessarily target web services like this very well.
-John
I absolutely lost here... should I know $__user_id__ or this is a variable already known to the tool?
Thank you very much for the help,
Dmitry
On 5/6/2014 4:39 AM, John Chilton wrote:
Hello Dmitry,
This use case isn't really addressed by Galaxy currently.
There is not a generic way of doing grabbing the users credentials like this or producing URLs from inside the tool. This depends on a lot of different things - how Galaxy's proxy is configured, etc... so in many (all?) cases Galaxy would not be able to fetch the users raw credentials. Your best bet is going to be to use the Galaxy's API - tools can fetch a user's API key and you can use Galaxy's API to build a URL that includes this key.
Your users will all need API keys set to use the tool. There have been some other galaxy-dev discussions about using the API from tools -
http://osdir.com/ml/galaxy-development-source-control/2013-12/msg00105.html. In general I would discourage this because it is difficult to do in a generic way - for instance there is no way for the tool to accurately determine what the URL of Galaxy should be - but it should be possible to tailor something to your setup if absolutely needed.
If you are really keen to use the Galaxy API to do this and you understand the limitations - we can try to work through the exact details.
If it is at all possible to send the data to the web service instead of sending a reference - the genomespace exporter
(https://bitbucket.org/galaxy/galaxy-central/src/74b6e23ed7882f3c091d2b66ce85...) might be a good example of how to do this in a more generic fashion.
-John
On Tue, Apr 29, 2014 at 11:16 AM, Dmitry redmitry@list.ru wrote:
Hello,
Is there any way to provide a protected resource link to the external application? My tool is supposed to call a Web service via SSL which should access to the Galaxy resources. In ideal it would be something like ftp://user:password@mygalaxy.net/fasta.gz
The Web service then downloads the resource and do the computation.
Since user is already provided the login to galaxy, I do not want to ask the password again on a tool page.
Sincerely,
Dmitry ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Dmitry -
John Chilton