Hi Geert, thanks for the information. Are you running GATK-2.4 or GATK-Lite-2.4. As far I know the lite version is deprecated. JJ are your wrappers in bitbucket/github? If you agree I would try to migrate to a recent GATK version and implement GALAXY_SLOTS. Not sure about the installation recipe ... Ciao, Bjoern

Hi,

I have GATK 2.4 running in our local galaxy, next to the default galaxy 1.x version. Both are in the tool menu, but only users that have provided a licence or proof that they are from an academic site have access to the 2.4 version. This is based on a usergroup check in a dynamic job runner configuration.

I used & modified the GATK 2 tool configuration from the test toolshed (by jjohnson). Modifications to the xmls included: --max_jvm_heap "xG" instead of --max_jvm_heap_fraction "1" : we use shared nodes, so it's not allowed to use all ram. --num_threads : some hard coded thread settings were changed made sure the '--et 'NO_ET' is commented out, as we don't have a key.

relevant dynamic_runner part:

from galaxy.jobs.mapper import JobMappingException if 'gatk2' in tool_id >= 0: user_group_assocs = user.groups or [] user_has_license = 'gatk2_license' in [user_group_assoc.group.name for user_group_assoc in user_group_assocs] if not user_has_license: raise JobMappingException("GATKv2 is only available to non-profit, academic users. Please send a mail to geert.vandeweyer2@uantwerpen.be to confirm this.")

Best,

Geert

On 11/19/2013 12:26 PM, Bjoern Gruening wrote:

...

Hi,

I found that the GATK tools do not have the same requirement tag. For example indel-realigner has version 1.4, realigner_target_creator has version 1.3. Is that correct? What is the recent version that is required for GATK?

Also realigner_target_creator uses the "gatk_picard_indexes" location file, but under tool-data is only a gatk_sorted_picard_indexes.loc file.

Are there any concrete plans for the GATK wrappers. Especially to update to a newer version. Or is that not feasible because of this complicated license policy and the phone-home option ...

I found the following trello card, but its a little bit outdated: https://trello.com/c/IPkT2spd

Is anyone running a recent GATK version in Galaxy?

Thanks! Bjoern

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

I pushed a copy of GATK2 from the testtoolshed to bitbucket: https://bitbucket.org/jjohnson/gatk2 The existing tool dependency no longer makes sense with the current licensing and access policies by Broad. Also, much of the duplication in the individual tool wrappers could be moved to macros, as Chilton did in the galaxy-central GATK: tools/gatk/gatk_macros.xml Thanks, JJ On 11/19/13, 1:28 PM, Björn Grüning wrote:

Hi Geert,

thanks for the information. Are you running GATK-2.4 or GATK-Lite-2.4. As far I know the lite version is deprecated.

JJ are your wrappers in bitbucket/github? If you agree I would try to migrate to a recent GATK version and implement GALAXY_SLOTS. Not sure about the installation recipe ...

Ciao, Bjoern

...

Hi,

I have GATK 2.4 running in our local galaxy, next to the default galaxy 1.x version. Both are in the tool menu, but only users that have provided a licence or proof that they are from an academic site have access to the 2.4 version. This is based on a usergroup check in a dynamic job runner configuration.

I used & modified the GATK 2 tool configuration from the test toolshed (by jjohnson). Modifications to the xmls included: --max_jvm_heap "xG" instead of --max_jvm_heap_fraction "1" : we use shared nodes, so it's not allowed to use all ram. --num_threads : some hard coded thread settings were changed made sure the '--et 'NO_ET' is commented out, as we don't have a key.

relevant dynamic_runner part:

from galaxy.jobs.mapper import JobMappingException if 'gatk2' in tool_id >= 0: user_group_assocs = user.groups or [] user_has_license = 'gatk2_license' in [user_group_assoc.group.name for user_group_assoc in user_group_assocs] if not user_has_license: raise JobMappingException("GATKv2 is only available to non-profit, academic users. Please send a mail to geert.vandeweyer2@uantwerpen.be to confirm this.")

Best,

Geert

On 11/19/2013 12:26 PM, Bjoern Gruening wrote:

...

Hi,

I found that the GATK tools do not have the same requirement tag. For example indel-realigner has version 1.4, realigner_target_creator has version 1.3. Is that correct? What is the recent version that is required for GATK?

Also realigner_target_creator uses the "gatk_picard_indexes" location file, but under tool-data is only a gatk_sorted_picard_indexes.loc file.

Are there any concrete plans for the GATK wrappers. Especially to update to a newer version. Or is that not feasible because of this complicated license policy and the phone-home option ...

I found the following trello card, but its a little bit outdated: https://trello.com/c/IPkT2spd

Is anyone running a recent GATK version in Galaxy?

Thanks! Bjoern

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

-- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota

Bjoern, Looks good. This page also had some info on per tool use of -nt and -nct http://www.broadinstitute.org/gatk/guide/tagged?tag=nct If remember correctly, GATK doesn't ignore options that don't apply, it just does an ungraceful exit. I not sure if I had that sorted out when I last worked on this. I'm not sure how much the -nct / --num_cpu_threads_per_data_thread value should be tuned, but it may make sense for that to be a variable or in a macro. I gave you write permission on: http://testtoolshed.g2.bx.psu.edu/view/jjohnson/gatk2 if you care to use that, otherwise we could migrate to a new tool shed repository. Thanks, JJ On 11/20/13, 10:19 AM, Bjoern Gruening wrote:

Thanks Jim,

I cloned it under: https://bitbucket.org/BjoernGruening/gatk2

and added macros.

If you agree I will try to do the following during the next week:

- add GALAXY_SLOTS -------------------- --num_threads | -nt will be \${GALAXY_SLOTS:-4} and -nct / --num_cpu_threads_per_data_thread will be set to a fixed value according to http://www.broadinstitute.org/gatk/guide/tagged?tag=multithreading

- adding new features if necessary - Adopting the tool_dependencies.xml - Adopting the readme file

Help and suggestions would be appreciated. Thanks! Bjoern

...

I pushed a copy of GATK2 from the testtoolshed to bitbucket:

https://bitbucket.org/jjohnson/gatk2

The existing tool dependency no longer makes sense with the current licensing and access policies by Broad.

Also, much of the duplication in the individual tool wrappers could be moved to macros, as Chilton did in the galaxy-central GATK: tools/gatk/gatk_macros.xml

Thanks,

JJ

On 11/19/13, 1:28 PM, Björn Grüning wrote:

...

Hi Geert,

thanks for the information. Are you running GATK-2.4 or GATK-Lite-2.4. As far I know the lite version is deprecated.

JJ are your wrappers in bitbucket/github? If you agree I would try to migrate to a recent GATK version and implement GALAXY_SLOTS. Not sure about the installation recipe ...

Ciao, Bjoern

...

Hi,

I have GATK 2.4 running in our local galaxy, next to the default galaxy 1.x version. Both are in the tool menu, but only users that have provided a licence or proof that they are from an academic site have access to the 2.4 version. This is based on a usergroup check in a dynamic job runner configuration.

I used & modified the GATK 2 tool configuration from the test toolshed (by jjohnson). Modifications to the xmls included: --max_jvm_heap "xG" instead of --max_jvm_heap_fraction "1" : we use shared nodes, so it's not allowed to use all ram. --num_threads : some hard coded thread settings were changed made sure the '--et 'NO_ET' is commented out, as we don't have a key.

relevant dynamic_runner part:

from galaxy.jobs.mapper import JobMappingException if 'gatk2' in tool_id >= 0: user_group_assocs = user.groups or [] user_has_license = 'gatk2_license' in [user_group_assoc.group.name for user_group_assoc in user_group_assocs] if not user_has_license: raise JobMappingException("GATKv2 is only available to non-profit, academic users. Please send a mail to geert.vandeweyer2@uantwerpen.be to confirm this.")

Best,

Geert

On 11/19/2013 12:26 PM, Bjoern Gruening wrote:

...

Hi,

I found that the GATK tools do not have the same requirement tag. For example indel-realigner has version 1.4, realigner_target_creator has version 1.3. Is that correct? What is the recent version that is required for GATK?

Also realigner_target_creator uses the "gatk_picard_indexes" location file, but under tool-data is only a gatk_sorted_picard_indexes.loc file.

Are there any concrete plans for the GATK wrappers. Especially to update to a newer version. Or is that not feasible because of this complicated license policy and the phone-home option ...

I found the following trello card, but its a little bit outdated: https://trello.com/c/IPkT2spd

Is anyone running a recent GATK version in Galaxy?

Thanks! Bjoern

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

-- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota

-- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota

Bjoern ,

Thanks! I don't mind, as you like. If you have time to integrate patches, we can go with your repository. I personally prefer git, but I can use that without problems. As you like :) Let's use your repository. Do you think we should create an IUC repository for GATK2? I think IUC tool shed repositories would be a good idea. Do you know how the plans are for GATK2 inside of Galaxy main? No clue on their plans. I've heard displeasure with the Broad's licensing moves. I would like to keep duplication at a minimum.

Agreed. I'd rather have one well-maintained source. Thanks, JJ On 11/20/13, 4:02 PM, Björn Grüning wrote:

Hi Jim,

latest changes here: https://bitbucket.org/BjoernGruening/gatk2

- more use of macros - new tool_dependencies that just sets the GATK2_PATH and GATK2_SITE_OPTIONS - added GALAXY_SLOTS to all tools - changes some inconsistency in loc file naming

...

This page also had some info on per tool use of -nt and -nct http://www.broadinstitute.org/gatk/guide/tagged?tag=nct If remember correctly, GATK doesn't ignore options that don't apply, it just does an ungraceful exit. I not sure if I had that sorted out when I last worked on this. Ok, I added a few of these parameters and will hopefully find some testers in the next weeks.

...

I'm not sure how much the -nct / --num_cpu_threads_per_data_thread value should be tuned, but it may make sense for that to be a variable or in a macro. -nc is now a macro -nct is hardcoded in 5 tools and they all have a different value

...

I gave you write permission on: http://testtoolshed.g2.bx.psu.edu/view/jjohnson/gatk2 if you care to use that, otherwise we could migrate to a new tool shed repository. Thanks! I don't mind, as you like. If you have time to integrate patches, we can go with your repository. I personally prefer git, but I can use that without problems. As you like :) Do you think we should create an IUC repository for GATK2? Do you know how the plans are for GATK2 inside of Galaxy main? I would like to keep duplication at a minimum.

@Geert: -et is now configurable with a ENV variable in the installed env.sh file (default is 'not set') and max_jvm_heap_fraction is deactivated by default.

Ciao, Bjoern

...

Thanks,

JJ

On 11/20/13, 10:19 AM, Bjoern Gruening wrote:

...

Thanks Jim,

I cloned it under: https://bitbucket.org/BjoernGruening/gatk2

and added macros.

If you agree I will try to do the following during the next week:

- add GALAXY_SLOTS -------------------- --num_threads | -nt will be \${GALAXY_SLOTS:-4} and -nct / --num_cpu_threads_per_data_thread will be set to a fixed value according to http://www.broadinstitute.org/gatk/guide/tagged?tag=multithreading

- adding new features if necessary - Adopting the tool_dependencies.xml - Adopting the readme file

Help and suggestions would be appreciated. Thanks! Bjoern

...

I pushed a copy of GATK2 from the testtoolshed to bitbucket:

https://bitbucket.org/jjohnson/gatk2

The existing tool dependency no longer makes sense with the current licensing and access policies by Broad.

Also, much of the duplication in the individual tool wrappers could be moved to macros, as Chilton did in the galaxy-central GATK: tools/gatk/gatk_macros.xml

Thanks,

JJ

On 11/19/13, 1:28 PM, Björn Grüning wrote:

...

Hi Geert,

thanks for the information. Are you running GATK-2.4 or GATK-Lite-2.4. As far I know the lite version is deprecated.

JJ are your wrappers in bitbucket/github? If you agree I would try to migrate to a recent GATK version and implement GALAXY_SLOTS. Not sure about the installation recipe ...

Ciao, Bjoern

...

Hi,

I have GATK 2.4 running in our local galaxy, next to the default galaxy 1.x version. Both are in the tool menu, but only users that have provided a licence or proof that they are from an academic site have access to the 2.4 version. This is based on a usergroup check in a dynamic job runner configuration.

I used & modified the GATK 2 tool configuration from the test toolshed (by jjohnson). Modifications to the xmls included: --max_jvm_heap "xG" instead of --max_jvm_heap_fraction "1" : we use shared nodes, so it's not allowed to use all ram. --num_threads : some hard coded thread settings were changed made sure the '--et 'NO_ET' is commented out, as we don't have a key.

relevant dynamic_runner part:

from galaxy.jobs.mapper import JobMappingException if 'gatk2' in tool_id >= 0: user_group_assocs = user.groups or [] user_has_license = 'gatk2_license' in [user_group_assoc.group.name for user_group_assoc in user_group_assocs] if not user_has_license: raise JobMappingException("GATKv2 is only available to non-profit, academic users. Please send a mail to geert.vandeweyer2@uantwerpen.be to confirm this.")

Best,

Geert

On 11/19/2013 12:26 PM, Bjoern Gruening wrote: > Hi, > > I found that the GATK tools do not have the same requirement tag. For > example indel-realigner has version 1.4, realigner_target_creator has > version 1.3. Is that correct? What is the recent version that is > required for GATK? > > Also realigner_target_creator uses the "gatk_picard_indexes" location > file, but under tool-data is only a gatk_sorted_picard_indexes.loc > file. > > Are there any concrete plans for the GATK wrappers. Especially to update > to a newer version. Or is that not feasible because of this complicated > license policy and the phone-home option ... > > I found the following trello card, but its a little bit outdated: > https://trello.com/c/IPkT2spd > > Is anyone running a recent GATK version in Galaxy? > > Thanks! > Bjoern > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/

-- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota