Hello Jim,

Thanks for sending your converter.  I've committed change set 6484:4fdceec512f5 to our central repository.  I now have things working for properly handling proprietary datatype converters and indexers.  I've also added the following paragraph to the tool shed wiki.  It doesn't apply to your mothur data types since you use only 1 converter, but you should be aware of this requirement for future tool development.  If you make the changes we discussed yesterday to your mothur tool suite (and add your missing converter ), all data types and the converter should properly load when your repository is installed to a local Galaxy instance. 

Thanks very much for your help on this, and please let me know if you bump into any issues.

If you include datatype converters or indexers in your repository, all converter files (the disk file referred to by the value of the "file" attribute) must be located in the same directory in your repository hierarchy.  The same requirement applies to indexers.  If you include both converters and indexers in your repository, the relevant files may all be located within the same directory or you could decide to keep all converters in one directory and all indexers in a different directory within your repository hierarchy.  This is critical because the Galaxy components that load these proprietary items assume they are all located in the same directory.




On Jan 5, 2012, at 4:15 PM, Jim Johnson wrote:

I'll also upload those to the toolshed soon.

Big Question?   
When I started creating all those datatype classes for mothur, I just labeled the the file_ext as mothur generated them for its output files. 
Will we have namespace issues? 
Should the file_ext fields include the toolshed name, e.g.   should "otu"   be named "mothur.otu" to avoid conflicts with other downloaded tools from the toolshed? 
Seems like this would be the time to establish rules/practices for such concerns.

JJ



On 1/5/12 3:06 PM, Greg Von Kuster wrote:
I will make sure that the converters are functional when installed, but I'm fairly sure it is currently not working.  If you could pass your 2 files along to me, I'll make sure to fix whatever bugs may exist.

On Jan 5, 2012, at 3:51 PM, Jim Johnson wrote:


This was a converter that I used on my local installation, but forgot to include for the ToolShed:

        <datatype extension="ref.taxonomy" type="galaxy.datatypes.metagenomics:RefTaxonomy" display_in_upload="true">
            <converter file="ref_to_seq_taxonomy_converter.xml" target_datatype="seq.taxonomy"/>
        </datatype>

$ find lib  -name ref_to_seq_taxonomy_converter.xml
lib/galaxy/datatypes/converters/ref_to_seq_taxonomy_converter.xml
$ find lib  -name ref_to_seq_taxonomy_converter.py
lib/galaxy/datatypes/converters/ref_to_seq_taxonomy_converter.py

I'll add those 2 files to my repository along with the other changes you specified.  
Can converters as such be auto-installed as well?  

Thanks,

JJ



On 1/5/12 2:14 PM, Greg Von Kuster wrote:
Hi Jim,

Here are the changes you'll need to make to your mothur tool suite.

CHANGE 1
----------------
Add the following datatypes.conf.xml file to your repository.

<?xml version="1.0"?>
<datatypes>
    <datatype_files>
        <datatype_file name="metagenomics.py"/>
    </datatype_files>
    <registration>
        <datatype extension="otu" type="galaxy.datatypes.metagenomics:Otu" display_in_upload="true"/>
        <datatype extension="list" type="galaxy.datatypes.metagenomics:OtuList" display_in_upload="true"/>
        <datatype extension="sabund" type="galaxy.datatypes.metagenomics:Sabund" display_in_upload="true"/>
        <datatype extension="rabund" type="galaxy.datatypes.metagenomics:Rabund" display_in_upload="true"/>
        <datatype extension="shared" type="galaxy.datatypes.metagenomics:SharedRabund" display_in_upload="true"/>
        <datatype extension="relabund" type="galaxy.datatypes.metagenomics:RelAbund" display_in_upload="true"/>
        <datatype extension="names" type="galaxy.datatypes.metagenomics:Names" display_in_upload="true"/>
        <datatype extension="design" type="galaxy.datatypes.metagenomics:Design" display_in_upload="true"/>
        <datatype extension="summary" type="galaxy.datatypes.metagenomics:Summary" display_in_upload="true"/>
        <datatype extension="groups" type="galaxy.datatypes.metagenomics:Group" display_in_upload="true"/>
        <datatype extension="oligos" type="galaxy.datatypes.metagenomics:Oligos" display_in_upload="true"/>
        <datatype extension="align" type="galaxy.datatypes.metagenomics:SequenceAlignment" display_in_upload="true"/>
        <datatype extension="accnos" type="galaxy.datatypes.metagenomics:AccNos" display_in_upload="true"/>
        <datatype extension="map" type="galaxy.datatypes.metagenomics:SecondaryStructureMap" display_in_upload="true"/>
        <datatype extension="align.check" type="galaxy.datatypes.metagenomics:AlignCheck" display_in_upload="true"/>
        <datatype extension="align.report" type="galaxy.datatypes.metagenomics:AlignReport" display_in_upload="true"/>
        <datatype extension="filter" type="galaxy.datatypes.metagenomics:LaneMask" display_in_upload="true"/>
        <datatype extension="dist" type="galaxy.datatypes.metagenomics:DistanceMatrix" display_in_upload="true"/>
        <datatype extension="pair.dist" type="galaxy.datatypes.metagenomics:PairwiseDistanceMatrix" display_in_upload="true"/>
        <datatype extension="square.dist" type="galaxy.datatypes.metagenomics:SquareDistanceMatrix" display_in_upload="true"/>
        <datatype extension="lower.dist" type="galaxy.datatypes.metagenomics:LowerTriangleDistanceMatrix" display_in_upload="true"/>
        <datatype extension="ref.taxonomy" type="galaxy.datatypes.metagenomics:RefTaxonomy" display_in_upload="true">
            <converter file="ref_to_seq_taxonomy_converter.xml" target_datatype="seq.taxonomy"/>
        </datatype>
        <datatype extension="seq.taxonomy" type="galaxy.datatypes.metagenomics:SequenceTaxonomy" display_in_upload="true"/>
        <datatype extension="rdp.taxonomy" type="galaxy.datatypes.metagenomics:RDPSequenceTaxonomy" display_in_upload="true"/>
        <datatype extension="cons.taxonomy" type="galaxy.datatypes.metagenomics:ConsensusTaxonomy" display_in_upload="true"/>
        <datatype extension="tax.summary" type="galaxy.datatypes.metagenomics:TaxonomySummary" display_in_upload="true"/>
        <datatype extension="freq" type="galaxy.datatypes.metagenomics:Frequency" display_in_upload="true"/>
        <datatype extension="quan" type="galaxy.datatypes.metagenomics:Quantile" display_in_upload="true"/>
        <datatype extension="filtered.quan" type="galaxy.datatypes.metagenomics:FilteredQuantile" display_in_upload="true"/>
        <datatype extension="masked.quan" type="galaxy.datatypes.metagenomics:MaskedQuantile" display_in_upload="true"/>
        <datatype extension="filtered.masked.quan" type="galaxy.datatypes.metagenomics:FilteredMaskedQuantile" display_in_upload="true"/>
        <datatype extension="axes" type="galaxy.datatypes.metagenomics:Axes" display_in_upload="true"/>
        <datatype extension="sff.flow" type="galaxy.datatypes.metagenomics:SffFlow" display_in_upload="true"/>
    </registration>
</datatypes>

I'm probably not correctly handling the converter for your ref.taxonomy data type - I've not been able to find the ref_to_seq_taxonomy_converter.xml file.  Can you pass it along to me so I can see if I have some debugging to do?

Also, I've eliminated the following entry from your README in the above file because the Newick class is not included in your metagenomics.py class module.  It seems you may have include the Newick class in your local copy of ~/lib/galaxy/datatypes/data.py.  If your tools use this class, it should be added to either your metagenomics.py class file or another class file in your repository and the value of the "type" attribute in the following should be changed accordingly.

<datatype extension="tre" type="galaxy.datatypes.data:Newick" display_in_upload="true"/>

CHANGE 2
---------------

The following relative imports in your metagenomics.py class module:

import data
from sniff import *

need to look like this:

from galaxy.datatypes import data
from galaxy.datatypes.sniff import *

CHANGE 3
---------------
You can optionally choose to remove your suite_config.xml file from your repository as it is no longer used in any way.

Thanks!

Greg Von Kuster


On Oct 18, 2011, at 11:03 AM, Jim Johnson wrote:

Greg,

The mothur_toolsuite in the ToolShed  contains a file with added datatypes for metagenomics (used by mothur and some by qiime):
mothur_toolsuite/mothur/lib/galaxy/datatypes/metagenomics.py
The README has info on how I incorporated mothur into our local galaxy server.

I'm also working on GMAP/GSNAP  (  http://research-pub.gene.com/gmap/ )
So far I've created a GmapDB class,  analogous to the ngsindex.BowtieIndex class, but with more metadata.
I'm also adding a IntervalIndexTree class for indexing maps of splice junctions, introns, and SNPs.
I'll send you this as soon as I've got it working.

Thanks,

JJ


Greg Von Kuster
Galaxy Development Team





Greg Von Kuster
Galaxy Development Team




<ref_to_seq_taxonomy_converter.py><ref_to_seq_taxonomy_converter.xml>

Greg Von Kuster
Galaxy Development Team
greg@bx.psu.edu