Dear list, I'm struggling to set java options for galaxy tools. Currently I use `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script I explored two ways to set java options: `_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"` But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is be interpreted as error by some tools. If I'm correct this is currently the default .. I have seen this happening, but can't remember the name of the tool. So I switched to this 'trick': `alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'` but the problem is that the tool script is called and not sourced and therefore aliases are not used (could this be changed?). Furthermore, for tools which explicitly set the java options my settings would be ignored anyway (an example is the MSGFPlusAdapter which calles `java -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to the corresponding parameter might be a solution for this tool). Any thought or suggestions on how to set the parameters in a production environment are very welcome. Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------
Hi Matthias, if a tool prints something to the stderr that is indeed considered to indicate a failure, and it's the default mostly because we don't want to break legacy tools that make this assumption. If you include a profile version in your tool or change your command section to <command detect_errors="exit_code"></commands> this will more reasonably determine if a tool failed based on the exit code. To me this sounds like something that really should be fixed in the tool as presumably this affects every deployer. Best, Marius On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <m.bernt@ufz.de> wrote:
Dear list,
I'm struggling to set java options for galaxy tools. Currently I use `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script I explored two ways to set java options:
`_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is be interpreted as error by some tools. If I'm correct this is currently the default .. I have seen this happening, but can't remember the name of the tool.
So I switched to this 'trick':
`alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
but the problem is that the tool script is called and not sourced and therefore aliases are not used (could this be changed?). Furthermore, for tools which explicitly set the java options my settings would be ignored anyway (an example is the MSGFPlusAdapter which calles `java -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to the corresponding parameter might be a solution for this tool).
Any thought or suggestions on how to set the parameters in a production environment are very welcome.
Cheers, Matthias
--
------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de, www.ufz.de
Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann ------------------------------------------- ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Hi Marius, Thanks for your comments. That's what I thought.
To me this sounds like something that really should be fixed in the tool as presumably this affects every deployer.
Hmm. The 'tool' would be java. As far as I have seen java is never/seldom in the tool dependencies. Hence, the system's java installation is used for which we do not know if it is Sun/OpenJDK or which version it is (in addition the installation will likely not change on production systems for a long time). So I guess fixing this in java would not help here in reasonable time anyway. So, it seems that the best option is to: - switching back to _JAVA_OPTIONS (which seems to be implemented also on usegalaxy.org) - handling the output in each java tool Are there any guidelines for wrapping java based applications (or other programming languages)? - add java to dependencies? I think this should be done, or? - if yes: is there a favored conda package? Cheers, Matthias P.S. I just added java to my list of despicable programming languages. On 27.08.2018 12:00, Marius van den Beek wrote:
Hi Matthias,
if a tool prints something to the stderr that is indeed considered to indicate a failure, and it's the default mostly because we don't want to break legacy tools that make this assumption. If you include a profile version in your tool or change your command section to <command detect_errors="exit_code"></commands> this will more reasonably determine if a tool failed based on the exit code.
Best, Marius
On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <m.bernt@ufz.de <mailto:m.bernt@ufz.de>> wrote:
Dear list,
I'm struggling to set java options for galaxy tools. Currently I use `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script I explored two ways to set java options:
`_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is be interpreted as error by some tools. If I'm correct this is currently the default .. I have seen this happening, but can't remember the name of the tool.
So I switched to this 'trick':
`alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
but the problem is that the tool script is called and not sourced and therefore aliases are not used (could this be changed?). Furthermore, for tools which explicitly set the java options my settings would be ignored anyway (an example is the MSGFPlusAdapter which calles `java -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to the corresponding parameter might be a solution for this tool).
Any thought or suggestions on how to set the parameters in a production environment are very welcome.
Cheers, Matthias
--
------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de <mailto:m.bernt@ufz.de>, www.ufz.de <http://www.ufz.de>
Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann ------------------------------------------- ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
-- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------
Hi Matthias,
Hi Marius,
Thanks for your comments. That's what I thought.
To me this sounds like something that really should be fixed in the tool as presumably this affects every deployer.
Hmm. The 'tool' would be java. As far as I have seen java is never/seldom in the tool dependencies. Hence, the system's java installation is used for which we do not know if it is Sun/OpenJDK or which version it is (in addition the installation will likely not change on production systems for a long time). So I guess fixing this in java would not help here in reasonable time anyway.
So, it seems that the best option is to: - switching back to _JAVA_OPTIONS (which seems to be implemented also on usegalaxy.org) - handling the output in each java tool
This is also what usegalaxy.eu is using and we have special destinations for java tools.
Are there any guidelines for wrapping java based applications (or other programming languages)?
- add java to dependencies? I think this should be done, or?
This should be part of the dependencies of your tool already, isn't it? So MSGFplus should depend on a java version of their choice. See here: https://github.com/bioconda/bioconda-recipes/blob/master/recipes/msgf_plus/m...
- if yes: is there a favored conda package?
openjdk if needed. Ciao, Bjoern
Cheers, Matthias
P.S. I just added java to my list of despicable programming languages.
On 27.08.2018 12:00, Marius van den Beek wrote:
Hi Matthias,
if a tool prints something to the stderr that is indeed considered to indicate a failure, and it's the default mostly because we don't want to break legacy tools that make this assumption. If you include a profile version in your tool or change your command section to <command detect_errors="exit_code"></commands> this will more reasonably determine if a tool failed based on the exit code.
Best, Marius
On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <m.bernt@ufz.de <mailto:m.bernt@ufz.de>> wrote:
Dear list,
I'm struggling to set java options for galaxy tools. Currently I use `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script I explored two ways to set java options:
`_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is be interpreted as error by some tools. If I'm correct this is currently the default .. I have seen this happening, but can't remember the name of the tool.
So I switched to this 'trick':
`alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
but the problem is that the tool script is called and not sourced and therefore aliases are not used (could this be changed?). Furthermore, for tools which explicitly set the java options my settings would be ignored anyway (an example is the MSGFPlusAdapter which calles `java -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to the corresponding parameter might be a solution for this tool).
Any thought or suggestions on how to set the parameters in a production environment are very welcome.
Cheers, Matthias
-- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de <mailto:m.bernt@ufz.de>, www.ufz.de <http://www.ufz.de>
Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann ------------------------------------------- ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Hi Bjoern,
So, it seems that the best option is to: - switching back to _JAVA_OPTIONS (which seems to be implemented also on usegalaxy.org) - handling the output in each java tool
This is also what usegalaxy.eu is using and we have special destinations for java tools.
Indeed, this seems to be the best option.
Are there any guidelines for wrapping java based applications (or other programming languages)?
- add java to dependencies? I think this should be done, or?
This should be part of the dependencies of your tool already, isn't it? So MSGFplus should depend on a java version of their choice.
See here: https://github.com/bioconda/bioconda-recipes/blob/master/recipes/msgf_plus/m...
I still tend to forget that conda also has dependencies .. of course this makes more sense than having a galaxy dependency.
- if yes: is there a favored conda package?
openjdk if needed.
OK. Thanks also to you. Best, Matthias
Ciao, Bjoern
Cheers, Matthias
P.S. I just added java to my list of despicable programming languages.
On 27.08.2018 12:00, Marius van den Beek wrote:
Hi Matthias,
if a tool prints something to the stderr that is indeed considered to indicate a failure, and it's the default mostly because we don't want to break legacy tools that make this assumption. If you include a profile version in your tool or change your command section to <command detect_errors="exit_code"></commands> this will more reasonably determine if a tool failed based on the exit code.
Best, Marius
On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <m.bernt@ufz.de <mailto:m.bernt@ufz.de>> wrote:
Dear list,
I'm struggling to set java options for galaxy tools. Currently I use `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script I explored two ways to set java options:
`_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is be interpreted as error by some tools. If I'm correct this is currently the default .. I have seen this happening, but can't remember the name of the tool.
So I switched to this 'trick':
`alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
but the problem is that the tool script is called and not sourced and therefore aliases are not used (could this be changed?). Furthermore, for tools which explicitly set the java options my settings would be ignored anyway (an example is the MSGFPlusAdapter which calles `java -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to the corresponding parameter might be a solution for this tool).
Any thought or suggestions on how to set the parameters in a production environment are very welcome.
Cheers, Matthias
-- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de <mailto:m.bernt@ufz.de>, www.ufz.de <http://www.ufz.de>
Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann ------------------------------------------- ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
-- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt@ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------
participants (3)
-
Björn Grüning -
Marius van den Beek -
Matthias Bernt