I've implemented support for proprietary datatypes that use class modules included in tool shed repositories. To see how this works, you'll need at least change set revision 6479:4d131422777f, which is currently available only from our central repo at https://bitbucket.org/galaxy/galaxy-central.

I've documented the way this works in the following 2 sections of the tool shed wiki. In the second section, I've taken the liberty of using your gmap tool repository as an example. i hope you don't mind. I've written the document section assuming that your gmap repository includes the 2 changes I've described below.

There are 2 categories of datatypes that are currently supported:

1. data types that subclass from the datatype classes included in the Galaxy distribution - these require no code files that define proprietary datatype classes to be included in the tool shed repository, and are documented in the first wiki section listed above.

2. datatypes that use proprietary classes defined in code files included in the tool shed repository - documented in the second wiki section listed above. Your gmap tool suite falls into this category.

If you make the following changes to your gmap tool suite, your proprietary data types will automatically load into a local Galaxy instance when the Galaxy admin installs your tool suite to that instance. The data types will be loaded at the time of installation as well as whenever the Galaxy server is stopped / restarted. I'll send you a separate message detailing the changes you'll need to make to your mothur tool suite.

Add a file named datatypes_conf.xml to your repository. This is the approach I'm using to support proprietary datatypes included in tool shed repositories instead f your proposed addition of datatypes in the tool config's <requirements> tag set. The datatypes_conf.xml file can be located anywhere in the repository, but the the obvious location for your gmap repository is your ~/tool-data directory.

This file should contain the following datatype definitions.

<?xml version="1.0"?>
<datatypes>
    <datatype_files>
        <datatype_file name="gmap.py"/>
    </datatype_files>
    <registration>
        <datatype extension="gmapdb" type="galaxy.datatypes.gmap:GmapDB" display_in_upload="False"/>
        <datatype extension="gmapsnpindex" type="galaxy.datatypes.gmap:GmapSnpIndex" display_in_upload="False"/>
        <datatype extension="iit" type="galaxy.datatypes.gmap:IntervalIndexTree" display_in_upload="True"/>
        <datatype extension="splicesites.iit" type="galaxy.datatypes.gmap:SpliceSitesIntervalIndexTree" display_in_upload="True"/>
        <datatype extension="introns.iit" type="galaxy.datatypes.gmap:IntronsIntervalIndexTree" display_in_upload="True"/>
        <datatype extension="snps.iit" type="galaxy.datatypes.gmap:SNPsIntervalIndexTree" display_in_upload="True"/>
        <datatype extension="gmap_annotation" type="galaxy.datatypes.gmap:IntervalAnnotation" display_in_upload="False"/>
        <datatype extension="gmap_splicesites" type="galaxy.datatypes.gmap:SpliceSiteAnnotation" display_in_upload="True"/>
        <datatype extension="gmap_introns" type="galaxy.datatypes.gmap:IntronAnnotation" display_in_upload="True"/>
        <datatype extension="gmap_snps" type="galaxy.datatypes.gmap:SNPAnnotation" display_in_upload="True"/>
    </registration>
    <sniffers>
        <sniffer type="galaxy.datatypes.gmap:IntervalAnnotation"/>
        <sniffer type="galaxy.datatypes.gmap:SpliceSiteAnnotation"/>
        <sniffer type="galaxy.datatypes.gmap:IntronAnnotation"/>
        <sniffer type="galaxy.datatypes.gmap:SNPAnnotation"/>
    </sniffers>
</datatypes>

I noticed that your README in your current gmap repository on the main Galaxy tool shed includes the following datatype definitions, but they refer to classes that are not included in your repository so I've eliminated them from the above datatypes_conf.xml file. You may need to add the classes to your current gmap.py datatypes class file and add them to the above datatypes_conf.xml file if your tools actually require them.

Modules that include proprietary datatype class definitions cannot use relative import references for imported modules. Imports must be defined as absolute from the galaxy subdirectory inside the Galaxy root's lib subdirectory. So for your ~/lib/galaxy/datatypes/gmap.py datatypes module in your gmap repository, the following changes are necessary.

Your current imports look like this:

import logging
import os,os.path,re
import data
from data import Text
from galaxy import util
from metadata import MetadataElement

But they need to be changed to this - note the elimination of relative imports:

import logging

import os,os.path,re

import galaxy.datatypes.data

from galaxy.datatypes.data import Text

from galaxy import util

from galaxy.datatypes.metadata import MetadataElement

Thanks very much for helping out with this, and please let me know if you bump into any problems.

On Oct 21, 2011, at 1:13 PM, Jim Johnson wrote:

Greg,

I put the gmap tool suite in the galaxy Tool Shed, let me know if there is more I should do.

It has 5 galaxy tools:

    GMAP   -  Genomic Mapping and Alignment Program for mRNA and EST sequences
    GSNAP    - Genomic Short-read Nucleotide Alignment Program
    GMAP Build    -  a database genome index for GMAP and GSNAP     ( calls:  gmap_build, iit_store, snpindex, cmetindex, atoiindex )
    GMAP SNP Index    - build index files for known SNPs                     (calls:  iit_store, snpindex)
    GMAP IIT    - Create a map store for known genes or SNPs              (calls:  iit_store)

It uses these added datatypes:

% grep -E '(^class | file_ext)' lib/galaxy/datatypes/gmap.py
class GmapDB( Text ):
    file_ext = 'gmapdb'
class GmapSnpIndex( Text ):
    file_ext = 'gmapsnpindex'
class IntervalIndexTree( Text ):
    file_ext = 'iit'
class SpliceSitesIntervalIndexTree( IntervalIndexTree ):
    file_ext = 'splicesites.iit'
class IntronsIntervalIndexTree( IntervalIndexTree ):
    file_ext = 'introns.iit'
class SNPsIntervalIndexTree( IntervalIndexTree ):
    file_ext = 'snps.iit'
class IntervalAnnotation( Text ):
    file_ext = 'gmap_annotation'
class SpliceSiteAnnotation(IntervalAnnotation):
    file_ext = 'gmap_splicesites'
class IntronAnnotation(IntervalAnnotation):
    file_ext = 'gmap_introns'
class SNPAnnotation(IntervalAnnotation):
    file_ext = 'gmap_snps'

I added a requirement tag for the datatypes to the tool-configs:

% grep 'requirement.*datatype' *.xml
gmap_build.xml:      <requirement type="datatype">gmapdb</requirement>
gmap_build.xml:      <requirement type="datatype">gmap_snps</requirement>
gmap.xml:    <requirement type="datatype">gmapdb</requirement>
gmap.xml:    <requirement type="datatype">gmap_annotation</requirement>
gmap.xml:    <requirement type="datatype">gmap_splicesites</requirement>
gmap.xml:    <requirement type="datatype">gmap_introns</requirement>
gmap.xml:    <requirement type="datatype">gmap_snps</requirement>
gsnap.xml:      <requirement type="datatype">gmapdb</requirement>
gsnap.xml:      <requirement type="datatype">gmapsnpindex</requirement>
gsnap.xml:      <requirement type="datatype">splicesites.iit</requirement>
gsnap.xml:      <requirement type="datatype">introns.iit</requirement>
iit_store.xml:      <requirement type="datatype">gmap_annotation</requirement>
iit_store.xml:      <requirement type="datatype">gmap_snps</requirement>
iit_store.xml:      <requirement type="datatype">iit</requirement>
iit_store.xml:      <requirement type="datatype">splicesites.iit</requirement>
iit_store.xml:      <requirement type="datatype">introns.iit</requirement>
iit_store.xml:      <requirement type="datatype">snps.iit</requirement>
snpindex.xml:      <requirement type="datatype">gmapsnpindex</requirement>
snpindex.xml:      <requirement type="datatype">gmapdb</requirement>
snpindex.xml:      <requirement type="datatype">gmap_snps</requirement>
snpindex.xml:      <requirement type="datatype">snps.iit</requirement>

Thanks,

JJ

Greg Von Kuster

Galaxy Development Team

greg@bx.psu.edu