Specifying _JAVA_OPTIONS in job_conf.xml
Hi, to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file? It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :) Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool> Thanks! Bjoern
You can create multiple variants of the 10G_memory tool with different native specifications right? Does this work around result in too much XML? Dynamic destinations could also be used - if you are interested I could post a demo rule that you might be able to use. The memory card referenced above might help, but a much easier cards might help this would be... https://trello.com/c/t1FH1Q5P Also tool style macros for job_conf.xml might help right - keep size of XML down? https://trello.com/c/wt1LflNh Feel free to create another card for this request (allowing parameterized tools to influence destination parameters) - seems reasonable. -John On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi,
to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file?
It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :)
Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool>
Thanks! Bjoern
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi John, Am 22.04.2014 20:45, schrieb John Chilton:
You can create multiple variants of the 10G_memory tool with different native specifications right? Does this work around result in too much XML? Dynamic destinations could also be used - if you are interested I could post a demo rule that you might be able to use.
The problem with multiple variants (multiple handlers) is that I only can specify one ENV variable that is known to the current shell. I don't think I can set the variable directly inside the nativeSpecification tag, or? Moreover, the variable need to have the name _JAVA_OPTIONS. To solve that I need to set the variable somehow inside the nativeSpecification tag. Dynamic destinations may work, but what happens if I change a ENV inside of python ... it is not known to the parent process or? Even if it is known, is it only known to that specific tool?
The memory card referenced above might help, but a much easier cards might help this would be...
Yes, but that is really hackish :). If there is no other way, I will try something like that ...
Also tool style macros for job_conf.xml might help right - keep size of XML down?
Unfortunately, macros do not work here. In GATK different tools do have different requirements. I can probationally work around that with HEAP_SIZE/2 or something like that, but a job_conf.xml based solution would be better.
Feel free to create another card for this request (allowing parameterized tools to influence destination parameters) - seems reasonable.
Ok I will wait a few more hours and than create such a card. Thanks for your ideas, Bjoern
-John
On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi,
to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file?
It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :)
Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool>
Thanks! Bjoern
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Tue, Apr 22, 2014 at 1:59 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi John,
Am 22.04.2014 20:45, schrieb John Chilton:
You can create multiple variants of the 10G_memory tool with different native specifications right? Does this work around result in too much XML? Dynamic destinations could also be used - if you are interested I could post a demo rule that you might be able to use.
You are right of course - very sorry. Dynamic destinations and macros would make sense if there was some way to alter the environment based on the destination - which I am not sure is possible with nativeSpecification at least generally. (It sounds like you can with PBS/Torque). Here is patch that would add the ability to specify <env id="NAME">VALUE</env> tags to job destinations in job_conf.xml and have them auto-populated in jobs. https://github.com/jmchilton/galaxy-central/compare/job_env?expand=1 https://github.com/jmchilton/galaxy-central/commit/d05e15d712c9a3175552e7c5f... Any interest in testing it for me :)? If not, I can open a PR for it as it is.
The problem with multiple variants (multiple handlers) is that I only can specify one ENV variable that is known to the current shell. I don't think I can set the variable directly inside the nativeSpecification tag, or? Moreover, the variable need to have the name _JAVA_OPTIONS. To solve that I need to set the variable somehow inside the nativeSpecification tag.
Dynamic destinations may work, but what happens if I change a ENV inside of python ... it is not known to the parent process or? Even if it is known, is it only known to that specific tool?
The memory card referenced above might help, but a much easier cards might help this would be...
Yes, but that is really hackish :). If there is no other way, I will try something like that ...
Also tool style macros for job_conf.xml might help right - keep size of XML down?
Unfortunately, macros do not work here. In GATK different tools do have different requirements. I can probationally work around that with HEAP_SIZE/2 or something like that, but a job_conf.xml based solution would be better.
Feel free to create another card for this request (allowing parameterized tools to influence destination parameters) - seems reasonable.
Ok I will wait a few more hours and than create such a card. Thanks for your ideas, Bjoern
-John
On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi,
to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file?
It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :)
Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool>
Thanks! Bjoern
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
:) Speechless ... I will stress test it tomorrow! Am 22.04.2014 21:24, schrieb John Chilton:
On Tue, Apr 22, 2014 at 1:59 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi John,
Am 22.04.2014 20:45, schrieb John Chilton:
You can create multiple variants of the 10G_memory tool with different native specifications right? Does this work around result in too much XML? Dynamic destinations could also be used - if you are interested I could post a demo rule that you might be able to use.
You are right of course - very sorry. Dynamic destinations and macros would make sense if there was some way to alter the environment based on the destination - which I am not sure is possible with nativeSpecification at least generally. (It sounds like you can with PBS/Torque).
Here is patch that would add the ability to specify <env id="NAME">VALUE</env> tags to job destinations in job_conf.xml and have them auto-populated in jobs.
https://github.com/jmchilton/galaxy-central/compare/job_env?expand=1 https://github.com/jmchilton/galaxy-central/commit/d05e15d712c9a3175552e7c5f...
Any interest in testing it for me :)?
If not, I can open a PR for it as it is.
The problem with multiple variants (multiple handlers) is that I only can specify one ENV variable that is known to the current shell. I don't think I can set the variable directly inside the nativeSpecification tag, or? Moreover, the variable need to have the name _JAVA_OPTIONS. To solve that I need to set the variable somehow inside the nativeSpecification tag.
Dynamic destinations may work, but what happens if I change a ENV inside of python ... it is not known to the parent process or? Even if it is known, is it only known to that specific tool?
The memory card referenced above might help, but a much easier cards might help this would be...
Yes, but that is really hackish :). If there is no other way, I will try something like that ...
Also tool style macros for job_conf.xml might help right - keep size of XML down?
Unfortunately, macros do not work here. In GATK different tools do have different requirements. I can probationally work around that with HEAP_SIZE/2 or something like that, but a job_conf.xml based solution would be better.
Feel free to create another card for this request (allowing parameterized tools to influence destination parameters) - seems reasonable.
Ok I will wait a few more hours and than create such a card. Thanks for your ideas, Bjoern
-John
On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi,
to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file?
It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :)
Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool>
Thanks! Bjoern
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi John, I tested it a day long on our server and it seems to work. It actually a nice solution, as you can do many things with it I think. Maybe it's worth to document, that <env ..>'foo bar'</env> is necessary. The >'< is important. Thanks! Bjoern Am 22.04.2014 22:01, schrieb Björn Grüning:
:) Speechless ... I will stress test it tomorrow!
Am 22.04.2014 21:24, schrieb John Chilton:
On Tue, Apr 22, 2014 at 1:59 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi John,
Am 22.04.2014 20:45, schrieb John Chilton:
You can create multiple variants of the 10G_memory tool with different native specifications right? Does this work around result in too much XML? Dynamic destinations could also be used - if you are interested I could post a demo rule that you might be able to use.
You are right of course - very sorry. Dynamic destinations and macros would make sense if there was some way to alter the environment based on the destination - which I am not sure is possible with nativeSpecification at least generally. (It sounds like you can with PBS/Torque).
Here is patch that would add the ability to specify <env id="NAME">VALUE</env> tags to job destinations in job_conf.xml and have them auto-populated in jobs.
https://github.com/jmchilton/galaxy-central/compare/job_env?expand=1 https://github.com/jmchilton/galaxy-central/commit/d05e15d712c9a3175552e7c5f...
Any interest in testing it for me :)?
If not, I can open a PR for it as it is.
The problem with multiple variants (multiple handlers) is that I only can specify one ENV variable that is known to the current shell. I don't think I can set the variable directly inside the nativeSpecification tag, or? Moreover, the variable need to have the name _JAVA_OPTIONS. To solve that I need to set the variable somehow inside the nativeSpecification tag.
Dynamic destinations may work, but what happens if I change a ENV inside of python ... it is not known to the parent process or? Even if it is known, is it only known to that specific tool?
The memory card referenced above might help, but a much easier cards might help this would be...
Yes, but that is really hackish :). If there is no other way, I will try something like that ...
Also tool style macros for job_conf.xml might help right - keep size of XML down?
Unfortunately, macros do not work here. In GATK different tools do have different requirements. I can probationally work around that with HEAP_SIZE/2 or something like that, but a job_conf.xml based solution would be better.
Feel free to create another card for this request (allowing parameterized tools to influence destination parameters) - seems reasonable.
Ok I will wait a few more hours and than create such a card. Thanks for your ideas, Bjoern
-John
On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi,
to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file?
It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :)
Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool>
Thanks! Bjoern
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hey Bjoern, On Wed, Apr 23, 2014 at 5:06 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi John,
I tested it a day long on our server and it seems to work. It actually a nice solution, as you can do many things with it I think.
Thanks for testing this! I am glad it worked and I agree that this is pretty flexible.
Maybe it's worth to document, that <env ..>'foo bar'</env> is necessary. The
Opened up a pull request for an improvement on this that doesn't require the quotation marks - you know so the documented example actually works now :). https://bitbucket.org/galaxy/galaxy-central/pull-request/378/allow-specifica... Unfortunately if you are tracking galaxy-central#stable the patches aren't going to apply cleanly like the last ones I sent you because I merged my job metrics PR last night and it touches some of the same files. Sorry about that. Thanks again! -John
'< is important.
Thanks! Bjoern
Am 22.04.2014 22:01, schrieb Björn Grüning:
:) Speechless ... I will stress test it tomorrow!
Am 22.04.2014 21:24, schrieb John Chilton:
On Tue, Apr 22, 2014 at 1:59 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi John,
Am 22.04.2014 20:45, schrieb John Chilton:
You can create multiple variants of the 10G_memory tool with different native specifications right? Does this work around result in too much XML? Dynamic destinations could also be used - if you are interested I could post a demo rule that you might be able to use.
You are right of course - very sorry. Dynamic destinations and macros would make sense if there was some way to alter the environment based on the destination - which I am not sure is possible with nativeSpecification at least generally. (It sounds like you can with PBS/Torque).
Here is patch that would add the ability to specify <env id="NAME">VALUE</env> tags to job destinations in job_conf.xml and have them auto-populated in jobs.
https://github.com/jmchilton/galaxy-central/compare/job_env?expand=1
https://github.com/jmchilton/galaxy-central/commit/d05e15d712c9a3175552e7c5f...
Any interest in testing it for me :)?
If not, I can open a PR for it as it is.
The problem with multiple variants (multiple handlers) is that I only can specify one ENV variable that is known to the current shell. I don't think I can set the variable directly inside the nativeSpecification tag, or? Moreover, the variable need to have the name _JAVA_OPTIONS. To solve that I need to set the variable somehow inside the nativeSpecification tag.
Dynamic destinations may work, but what happens if I change a ENV inside of python ... it is not known to the parent process or? Even if it is known, is it only known to that specific tool?
The memory card referenced above might help, but a much easier cards might help this would be...
Yes, but that is really hackish :). If there is no other way, I will try something like that ...
Also tool style macros for job_conf.xml might help right - keep size of XML down?
Unfortunately, macros do not work here. In GATK different tools do have different requirements. I can probationally work around that with HEAP_SIZE/2 or something like that, but a job_conf.xml based solution would be better.
Feel free to create another card for this request (allowing parameterized tools to influence destination parameters) - seems reasonable.
Ok I will wait a few more hours and than create such a card. Thanks for your ideas, Bjoern
-John
On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi,
to specify -Xmx and -Xms for java based programs I set my preferred values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but the downside is that I can not have different options for different tools. Is there a trick I can specify arbitrary $ENVs to a specific tool? Any ideas how to solve that, without hacking the -Xmx and -Xms parameters into the wrapper file?
It is related to https://trello.com/c/HGEYPQf6. I would be interested in workarounds until that card is closed :)
Something like that would be great: <tool id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_indel_realigner" destination="10G_memory"> <param id="nativeParam">export _JAVA_OPTIONS='-Xmx=6GB'</param> </tool>
Thanks! Bjoern
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (2)
-
Björn Grüning -
John Chilton