I have been putting datatype def files in relative path:
lib/galaxy/datatypes/
This was just to make it more clear for someone manually modifying
their own galaxy installation.
Is there any preferred best practice for where a datatypes
implementation file should be?
Hello Jim,
I've implemented support for proprietary datatypes that use
class modules included in tool shed repositories. To see how
this works, you'll need at least change set
revision 6479:4d131422777f, which is currently available only
from our central repo at
https://bitbucket.org/galaxy/galaxy-central.
I've documented the way this works in the following 2
sections of the tool shed wiki. In the second section, I've
taken the liberty of using your gmap tool repository as an
example. i hope you don't mind. I've written the document
section assuming that your gmap repository includes the 2
changes I've described below.
There are 2 categories of datatypes that are currently
supported:
1. data types that subclass from the datatype classes
included in the Galaxy distribution - these require no code
files that define proprietary datatype classes to be included in
the tool shed repository, and are documented in the first wiki
section listed above.
2. datatypes that use proprietary classes defined in code
files included in the tool shed repository - documented in the
second wiki section listed above. Your gmap tool suite falls
into this category.
If you make the following changes to your gmap tool suite,
your proprietary data types will automatically load into a local
Galaxy instance when the Galaxy admin installs your tool suite
to that instance. The data types will be loaded at the time of
installation as well as whenever the Galaxy server is stopped /
restarted. I'll send you a separate message detailing the
changes you'll need to make to your mothur tool suite.
CHANGE 1
----------------
Add a file named datatypes_conf.xml to your repository. This
is the approach I'm using to support proprietary datatypes
included in tool shed repositories instead f your proposed
addition of datatypes in the tool config's <requirements>
tag set. The datatypes_conf.xml file can be located anywhere in
the repository, but the the obvious location for your gmap
repository is your ~/tool-data directory.
This file should contain the following datatype definitions.
<?xml version="1.0"?>
<datatypes>
<datatype_files>
<datatype_file name="gmap.py"/>
</datatype_files>
<registration>
<datatype extension="gmapdb" type="galaxy.datatypes.gmap:GmapDB" display_in_upload="False"/>
<datatype extension="gmapsnpindex" type="galaxy.datatypes.gmap:GmapSnpIndex" display_in_upload="False"/>
<datatype extension="iit" type="galaxy.datatypes.gmap:IntervalIndexTree" display_in_upload="True"/>
<datatype extension="splicesites.iit" type="galaxy.datatypes.gmap:SpliceSitesIntervalIndexTree" display_in_upload="True"/>
<datatype extension="introns.iit" type="galaxy.datatypes.gmap:IntronsIntervalIndexTree" display_in_upload="True"/>
<datatype extension="snps.iit" type="galaxy.datatypes.gmap:SNPsIntervalIndexTree" display_in_upload="True"/>
<datatype extension="gmap_annotation" type="galaxy.datatypes.gmap:IntervalAnnotation" display_in_upload="False"/>
<datatype extension="gmap_splicesites" type="galaxy.datatypes.gmap:SpliceSiteAnnotation" display_in_upload="True"/>
<datatype extension="gmap_introns" type="galaxy.datatypes.gmap:IntronAnnotation" display_in_upload="True"/>
<datatype extension="gmap_snps" type="galaxy.datatypes.gmap:SNPAnnotation" display_in_upload="True"/>
</registration>
<sniffers>
<sniffer type="galaxy.datatypes.gmap:IntervalAnnotation"/>
<sniffer type="galaxy.datatypes.gmap:SpliceSiteAnnotation"/>
<sniffer type="galaxy.datatypes.gmap:IntronAnnotation"/>
<sniffer type="galaxy.datatypes.gmap:SNPAnnotation"/>
</sniffers>
</datatypes>
I noticed that your README in your current gmap repository on
the main Galaxy tool shed includes the following datatype
definitions, but they refer to classes that are not included in
your repository so I've eliminated them from the above
datatypes_conf.xml file. You may need to add the classes to
your current gmap.py datatypes class file and add them to the
above datatypes_conf.xml file if your tools actually require
them.
<datatype extension="tally.iit" type="galaxy.datatypes.gmap:TallyIntervalIndexTree" display_in_upload="True"/>
<datatype extension="gsnap_tally" type="galaxy.datatypes.gmap:TallyAnnotation" display_in_upload="True"/>
<datatype extension="gsnap" type="galaxy.datatypes.gmap:GsnapResult" display_in_upload="True"/>
CHANGE 2
----------------
Modules that include proprietary datatype class definitions
cannot use relative import references for imported modules.
Imports must be defined as absolute from the galaxy
subdirectory inside the Galaxy root's lib subdirectory. So for
your ~/lib/galaxy/datatypes/gmap.py datatypes module in your
gmap repository, the following changes are necessary.
Your current imports look like this:
import logging
import os,os.path,re
import data
from data import Text
from galaxy import util
from metadata import MetadataElement
But they need to be changed to this - note the elimination of
relative imports:
import logging
import os,os.path,re
import galaxy.datatypes.data
from galaxy.datatypes.data import Text
from galaxy import util
from galaxy.datatypes.metadata import MetadataElement
Thanks very much for helping out with this, and please let me
know if you bump into any problems.
Greg Von Kuster
On Oct 21, 2011, at 1:13 PM, Jim Johnson wrote:
Greg,
I put the gmap tool suite in the galaxy Tool Shed, let me
know if there is more I should do.
It has 5 galaxy tools:
GMAP - Genomic Mapping and Alignment Program for mRNA and EST sequences
GSNAP - Genomic Short-read Nucleotide Alignment Program
GMAP Build - a database genome index for GMAP and GSNAP ( calls: gmap_build, iit_store, snpindex, cmetindex, atoiindex )
GMAP SNP Index - build index files for known SNPs (calls: iit_store, snpindex)
GMAP IIT - Create a map store for known genes or SNPs (calls: iit_store)
It uses these added datatypes:
% grep -E '(^class | file_ext)'
lib/galaxy/datatypes/gmap.py
class GmapDB( Text ):
file_ext = 'gmapdb'
class GmapSnpIndex( Text ):
file_ext = 'gmapsnpindex'
class IntervalIndexTree( Text ):
file_ext = 'iit'
class SpliceSitesIntervalIndexTree( IntervalIndexTree ):
file_ext = 'splicesites.iit'
class IntronsIntervalIndexTree( IntervalIndexTree ):
file_ext = 'introns.iit'
class SNPsIntervalIndexTree( IntervalIndexTree ):
file_ext = 'snps.iit'
class IntervalAnnotation( Text ):
file_ext = 'gmap_annotation'
class SpliceSiteAnnotation(IntervalAnnotation):
file_ext = 'gmap_splicesites'
class IntronAnnotation(IntervalAnnotation):
file_ext = 'gmap_introns'
class SNPAnnotation(IntervalAnnotation):
file_ext = 'gmap_snps'
I added a requirement tag for the datatypes to the
tool-configs:
% grep 'requirement.*datatype' *.xml
gmap_build.xml: <requirement
type="datatype">gmapdb</requirement>
gmap_build.xml: <requirement
type="datatype">gmap_snps</requirement>
gmap.xml: <requirement
type="datatype">gmapdb</requirement>
gmap.xml: <requirement
type="datatype">gmap_annotation</requirement>
gmap.xml: <requirement
type="datatype">gmap_splicesites</requirement>
gmap.xml: <requirement
type="datatype">gmap_introns</requirement>
gmap.xml: <requirement
type="datatype">gmap_snps</requirement>
gsnap.xml: <requirement
type="datatype">gmapdb</requirement>
gsnap.xml: <requirement
type="datatype">gmapsnpindex</requirement>
gsnap.xml: <requirement
type="datatype">splicesites.iit</requirement>
gsnap.xml: <requirement
type="datatype">introns.iit</requirement>
iit_store.xml: <requirement
type="datatype">gmap_annotation</requirement>
iit_store.xml: <requirement
type="datatype">gmap_snps</requirement>
iit_store.xml: <requirement
type="datatype">iit</requirement>
iit_store.xml: <requirement
type="datatype">splicesites.iit</requirement>
iit_store.xml: <requirement
type="datatype">introns.iit</requirement>
iit_store.xml: <requirement
type="datatype">snps.iit</requirement>
snpindex.xml: <requirement
type="datatype">gmapsnpindex</requirement>
snpindex.xml: <requirement
type="datatype">gmapdb</requirement>
snpindex.xml: <requirement
type="datatype">gmap_snps</requirement>
snpindex.xml: <requirement
type="datatype">snps.iit</requirement>
Thanks,
JJ
Greg Von Kuster
Galaxy Development Team