January 2012 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Tool shed and datatypes
by Greg Von Kuster 06 Jan '12

06 Jan '12

Hello Jim, Thanks for sending your converter. I've committed change set 6484:4fdceec512f5 to our central repository. I now have things working for properly handling proprietary datatype converters and indexers. I've also added the following paragraph to the tool shed wiki. It doesn't apply to your mothur data types since you use only 1 converter, but you should be aware of this requirement for future tool development. If you make the changes we discussed yesterday to your mothur tool suite (and add your missing converter ), all data types and the converter should properly load when your repository is installed to a local Galaxy instance. Thanks very much for your help on this, and please let me know if you bump into any issues. If you include datatype converters or indexers in your repository, all converter files (the disk file referred to by the value of the "file" attribute) must be located in the same directory in your repository hierarchy. The same requirement applies to indexers. If you include both converters and indexers in your repository, the relevant files may all be located within the same directory or you could decide to keep all converters in one directory and all indexers in a different directory within your repository hierarchy. This is critical because the Galaxy components that load these proprietary items assume they are all located in the same directory. On Jan 5, 2012, at 4:15 PM, Jim Johnson wrote: > I'll also upload those to the toolshed soon. > > Big Question? > When I started creating all those datatype classes for mothur, I just labeled the the file_ext as mothur generated them for its output files. > Will we have namespace issues? > Should the file_ext fields include the toolshed name, e.g. should "otu" be named "mothur.otu" to avoid conflicts with other downloaded tools from the toolshed? > Seems like this would be the time to establish rules/practices for such concerns. > > JJ > > > > On 1/5/12 3:06 PM, Greg Von Kuster wrote: >> >> I will make sure that the converters are functional when installed, but I'm fairly sure it is currently not working. If you could pass your 2 files along to me, I'll make sure to fix whatever bugs may exist. >> >> On Jan 5, 2012, at 3:51 PM, Jim Johnson wrote: >> >>> >>> This was a converter that I used on my local installation, but forgot to include for the ToolShed: >>> >>> <datatype extension="ref.taxonomy" type="galaxy.datatypes.metagenomics:RefTaxonomy" display_in_upload="true"> >>> <converter file="ref_to_seq_taxonomy_converter.xml" target_datatype="seq.taxonomy"/> >>> </datatype> >>> >>> $ find lib -name ref_to_seq_taxonomy_converter.xml >>> lib/galaxy/datatypes/converters/ref_to_seq_taxonomy_converter.xml >>> $ find lib -name ref_to_seq_taxonomy_converter.py >>> lib/galaxy/datatypes/converters/ref_to_seq_taxonomy_converter.py >>> >>> I'll add those 2 files to my repository along with the other changes you specified. >>> Can converters as such be auto-installed as well? >>> >>> Thanks, >>> >>> JJ >>> >>> >>> >>> On 1/5/12 2:14 PM, Greg Von Kuster wrote: >>>> >>>> Hi Jim, >>>> >>>> Here are the changes you'll need to make to your mothur tool suite. >>>> >>>> CHANGE 1 >>>> ---------------- >>>> Add the following datatypes.conf.xml file to your repository. >>>> >>>> <?xml version="1.0"?> >>>> <datatypes> >>>> <datatype_files> >>>> <datatype_file name="metagenomics.py"/> >>>> </datatype_files> >>>> <registration> >>>> <datatype extension="otu" type="galaxy.datatypes.metagenomics:Otu" display_in_upload="true"/> >>>> <datatype extension="list" type="galaxy.datatypes.metagenomics:OtuList" display_in_upload="true"/> >>>> <datatype extension="sabund" type="galaxy.datatypes.metagenomics:Sabund" display_in_upload="true"/> >>>> <datatype extension="rabund" type="galaxy.datatypes.metagenomics:Rabund" display_in_upload="true"/> >>>> <datatype extension="shared" type="galaxy.datatypes.metagenomics:SharedRabund" display_in_upload="true"/> >>>> <datatype extension="relabund" type="galaxy.datatypes.metagenomics:RelAbund" display_in_upload="true"/> >>>> <datatype extension="names" type="galaxy.datatypes.metagenomics:Names" display_in_upload="true"/> >>>> <datatype extension="design" type="galaxy.datatypes.metagenomics:Design" display_in_upload="true"/> >>>> <datatype extension="summary" type="galaxy.datatypes.metagenomics:Summary" display_in_upload="true"/> >>>> <datatype extension="groups" type="galaxy.datatypes.metagenomics:Group" display_in_upload="true"/> >>>> <datatype extension="oligos" type="galaxy.datatypes.metagenomics:Oligos" display_in_upload="true"/> >>>> <datatype extension="align" type="galaxy.datatypes.metagenomics:SequenceAlignment" display_in_upload="true"/> >>>> <datatype extension="accnos" type="galaxy.datatypes.metagenomics:AccNos" display_in_upload="true"/> >>>> <datatype extension="map" type="galaxy.datatypes.metagenomics:SecondaryStructureMap" display_in_upload="true"/> >>>> <datatype extension="align.check" type="galaxy.datatypes.metagenomics:AlignCheck" display_in_upload="true"/> >>>> <datatype extension="align.report" type="galaxy.datatypes.metagenomics:AlignReport" display_in_upload="true"/> >>>> <datatype extension="filter" type="galaxy.datatypes.metagenomics:LaneMask" display_in_upload="true"/> >>>> <datatype extension="dist" type="galaxy.datatypes.metagenomics:DistanceMatrix" display_in_upload="true"/> >>>> <datatype extension="pair.dist" type="galaxy.datatypes.metagenomics:PairwiseDistanceMatrix" display_in_upload="true"/> >>>> <datatype extension="square.dist" type="galaxy.datatypes.metagenomics:SquareDistanceMatrix" display_in_upload="true"/> >>>> <datatype extension="lower.dist" type="galaxy.datatypes.metagenomics:LowerTriangleDistanceMatrix" display_in_upload="true"/> >>>> <datatype extension="ref.taxonomy" type="galaxy.datatypes.metagenomics:RefTaxonomy" display_in_upload="true"> >>>> <converter file="ref_to_seq_taxonomy_converter.xml" target_datatype="seq.taxonomy"/> >>>> </datatype> >>>> <datatype extension="seq.taxonomy" type="galaxy.datatypes.metagenomics:SequenceTaxonomy" display_in_upload="true"/> >>>> <datatype extension="rdp.taxonomy" type="galaxy.datatypes.metagenomics:RDPSequenceTaxonomy" display_in_upload="true"/> >>>> <datatype extension="cons.taxonomy" type="galaxy.datatypes.metagenomics:ConsensusTaxonomy" display_in_upload="true"/> >>>> <datatype extension="tax.summary" type="galaxy.datatypes.metagenomics:TaxonomySummary" display_in_upload="true"/> >>>> <datatype extension="freq" type="galaxy.datatypes.metagenomics:Frequency" display_in_upload="true"/> >>>> <datatype extension="quan" type="galaxy.datatypes.metagenomics:Quantile" display_in_upload="true"/> >>>> <datatype extension="filtered.quan" type="galaxy.datatypes.metagenomics:FilteredQuantile" display_in_upload="true"/> >>>> <datatype extension="masked.quan" type="galaxy.datatypes.metagenomics:MaskedQuantile" display_in_upload="true"/> >>>> <datatype extension="filtered.masked.quan" type="galaxy.datatypes.metagenomics:FilteredMaskedQuantile" display_in_upload="true"/> >>>> <datatype extension="axes" type="galaxy.datatypes.metagenomics:Axes" display_in_upload="true"/> >>>> <datatype extension="sff.flow" type="galaxy.datatypes.metagenomics:SffFlow" display_in_upload="true"/> >>>> </registration> >>>> </datatypes> >>>> >>>> I'm probably not correctly handling the converter for your ref.taxonomy data type - I've not been able to find the ref_to_seq_taxonomy_converter.xml file. Can you pass it along to me so I can see if I have some debugging to do? >>>> >>>> Also, I've eliminated the following entry from your README in the above file because the Newick class is not included in your metagenomics.py class module. It seems you may have include the Newick class in your local copy of ~/lib/galaxy/datatypes/data.py. If your tools use this class, it should be added to either your metagenomics.py class file or another class file in your repository and the value of the "type" attribute in the following should be changed accordingly. >>>> >>>> <datatype extension="tre" type="galaxy.datatypes.data:Newick" display_in_upload="true"/> >>>> >>>> CHANGE 2 >>>> --------------- >>>> >>>> The following relative imports in your metagenomics.py class module: >>>> >>>> import data >>>> from sniff import * >>>> >>>> need to look like this: >>>> >>>> from galaxy.datatypes import data >>>> from galaxy.datatypes.sniff import * >>>> >>>> CHANGE 3 >>>> --------------- >>>> You can optionally choose to remove your suite_config.xml file from your repository as it is no longer used in any way. >>>> >>>> Thanks! >>>> >>>> Greg Von Kuster >>>> >>>> >>>> On Oct 18, 2011, at 11:03 AM, Jim Johnson wrote: >>>> >>>>> Greg, >>>>> >>>>> The mothur_toolsuite in the ToolShed contains a file with added datatypes for metagenomics (used by mothur and some by qiime): >>>>> mothur_toolsuite/mothur/lib/galaxy/datatypes/metagenomics.py >>>>> The README has info on how I incorporated mothur into our local galaxy server. >>>>> >>>>> I'm also working on GMAP/GSNAP ( http://research-pub.gene.com/gmap/ ) >>>>> So far I've created a GmapDB class, analogous to the ngsindex.BowtieIndex class, but with more metadata. >>>>> I'm also adding a IntervalIndexTree class for indexing maps of splice junctions, introns, and SNPs. >>>>> I'll send you this as soon as I've got it working. >>>>> >>>>> Thanks, >>>>> >>>>> JJ >>>>> >>>> >>>> Greg Von Kuster >>>> Galaxy Development Team >>>> greg(a)bx.psu.edu >>>> >>>> >>>> >>> >> >> Greg Von Kuster >> Galaxy Development Team >> greg(a)bx.psu.edu >> >> >> > > <ref_to_seq_taxonomy_converter.py><ref_to_seq_taxonomy_converter.xml> Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

Information about server availability
by sarah Maman 06 Jan '12

06 Jan '12

Dear all, I would like to get the latest copy from the repository (https://bitbucket.org/galaxy/galaxy-dist) in order to install Galaxy. But, unfortunatly, no server is available to handle this request (503 Service Unavailable). Could you, please, help me ? Do you know when this information will be available ? Please, excuse me if this information is available on your web and if I have not seen it. Happy new year, Sarah Maman -- Sarah Maman INRA - LCG - SIGENAE Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Tax: +33(0)5.61.28.57.53

3 3

Re: [galaxy-dev] Information about server availability
by Nate Coraor 06 Jan '12

06 Jan '12

On Jan 6, 2012, at 3:49 AM, sarah Maman wrote: > Dear, > > In your tarball (http://dist.g2.bx.psu.edu/galaxy-dist.b258de1e6cea.tar.gz) I have found the repository named "galaxy-dist" but I did not found "galaxy-central" and "limp". Could you please explain me where I could find these repositories ? I'm not sure what limp is, but galaxy-central is just the in-development version of galaxy-dist (i.e. it has the most recent commits). Since bitbucket is back up, you can access galaxy-central here: https://bitbucket.org/galaxy/galaxy-central --nate > > Thank you in advance, > Sarah Maman > > > > Nate Coraor <nate(a)bx.psu.edu> a écrit : > >> On Jan 5, 2012, at 10:52 AM, sarah Maman wrote: >> >>> Dear all, >>> >>> I would like to get the latest copy from the repository (https://bitbucket.org/galaxy/galaxy-dist) in order to install Galaxy. >>> But, unfortunatly, no server is available to handle this request (503 Service Unavailable). >>> Could you, please, help me ? Do you know when this information will be available ? >>> Please, excuse me if this information is available on your web and if I have not seen it. >> >> Hi Sarah, >> >> Bitbucket is currently down. This is a temporary situation, but I am not sure when they'll have service restored. In the meantime, I've put a tarball up at: >> >> http://dist.g2.bx.psu.edu/galaxy-dist.b258de1e6cea.tar.gz >> >> Sorry for the inconvenience and thanks for using Galaxy. >> >> --nate >> >>> >>> Happy new year, >>> Sarah Maman >>> >>> >>> >>> -- >>> Sarah Maman >>> INRA - LCG - SIGENAE >>> Chemin de Borde-Rouge - Auzeville - BP 52627 >>> 31326 Castanet-Tolosan cedex - FRANCE >>> Tel: +33(0)5.61.28.57.08 >>> Tax: +33(0)5.61.28.57.53 >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> >> > > > > -- > Sarah Maman > INRA - LCG - SIGENAE > Chemin de Borde-Rouge - Auzeville - BP 52627 > 31326 Castanet-Tolosan cedex - FRANCE > Tel: +33(0)5.61.28.57.08 > Tax: +33(0)5.61.28.57.53 >

1 0

Re: [galaxy-dev] Galaxy Proxy by an External Apache Server
by Derrick LIN 05 Jan '12

05 Jan '12

Hi Carlos, You are right! I put the same config to site-available and now it works! Will do more testing/configuring base on that. Do you know why the location of the settings cause this issue? Thanks Derrick On Fri, Jan 6, 2012 at 12:42 PM, Carlos Borroto <carlos.borroto(a)gmail.com>wrote: > > I'm not sure this will solve the problem, but I would put this > configuration in /etc/apache2/sites-available/default if your server > is not serving any other virtualhost. If you are serving publicnet.org > from a virtualhost configured in /etc/apache2/sites-available/, put > this configuration in the corresponding file. > > Remember to do reload apache configuration after doing the changes: > $ sudo invoke-rc.d apache2 reload > > > I also want to clarify if I need to create the symlink to galaxy inside > > /var/www/? > > > > No, you don't need this link. > > Hope it helps, > Carlos >

2 2

Deleting Data Libraries
by Ryan Golhar 05 Jan '12

05 Jan '12

I haven't found a way to fix the status on BAM files being imported. They still read "Job still running". So I thought I'd just delete the whole data library and start a new. From the Admin page, I click Manage Data Libraries, then select all the data libraries by clicking the checkbox next to "Data library name" in the table header. I then click delete. I get an error "Invalid Library id ( a799d38679e985db,5969b1f7201f12ae ) specified.". If I only check the checkboxes for the data libraries, I still get the error. I have to delete the libraries one at a time.

1 0

Script can't find other script
by Christopher Fletez-Brant 05 Jan '12

05 Jan '12

Good afternoon Galaxy list, I've got a Python script which calls an R script. When I run a job using Galaxy, I get the following output from R: Error in file(file, "r", encoding = encoding) : cannot open the connection In addition: Warning message: In file(file, "r", encoding = encoding) : cannot open file 'rocprc.R': No such file or directory The difficult thing here is that both scripts are in the same directory. I tried using os.path.realpath(PYTHON SCRIPT) to get the name of the current directory so that I could force R to operate in same; I still get this error. Telling Python to print the results from os.path.realpath(PYTHON SCRIPT) gave me "~/galaxy-dist/database/job_working_directory". Also, the Python script works perfectly when called from the command line, so I know this is a Galaxy issue. Please help, Kipper Fletez-Brant

1 0

Re: [galaxy-dev] Tool shed and datatypes
by Greg Von Kuster 05 Jan '12

05 Jan '12

Yes, this is certainly important, but I think the hope is that proprietary data types will not become so prevalent that name-spacing the extensions is necessary. On Jan 5, 2012, at 4:15 PM, Jim Johnson wrote: > > Big Question? > When I started creating all those datatype classes for mothur, I just labeled the the file_ext as mothur generated them for its output files. > Will we have namespace issues? > Should the file_ext fields include the toolshed name, e.g. should "otu" be named "mothur.otu" to avoid conflicts with other downloaded tools from the toolshed? > Seems like this would be the time to establish rules/practices for such concerns. > > JJ > > >

1 1

Fwd: Galaxy Proxy by an External Apache Server
by Derrick LIN 05 Jan '12

05 Jan '12

Hi guys, Thanks for the helps. I used http://publicnet.org/galaxy for visiting the site, so it's my typo. On Fri, Jan 6, 2012 at 3:49 AM, Carlos Borroto <carlos.borroto(a)gmail.com>wrote: > > The fact your apache server is looking in /var/www/galaxy means the > 'RewriteRule' is not working. Did you check if modwrite is activated > in your apache install? > I think the module is loaded and active. I checked it with apache2ctl -t -D DUMP_MODULES Loaded Modules: ..... rewrite_module (shared) ..... Syntax OK I am not very confident with my config though. The Apache server is on ubuntu 10.10 and was installed from apt-get. It has its own conf files structure compares to the generic Apache: ls -la /etc/apache2/mods-enabled/ .... lrwxrwxrwx 1 root root 28 2011-05-13 13:54 proxy.conf -> ../mods-available/proxy.conf lrwxrwxrwx 1 root root 33 2011-05-13 14:56 proxy_http.load -> ../mods-available/proxy_http.load lrwxrwxrwx 1 root root 28 2011-05-13 13:54 proxy.load -> ../mods-available/proxy.load lrwxrwxrwx 1 root root 33 2011-04-11 14:23 reqtimeout.conf -> ../mods-available/reqtimeout.conf lrwxrwxrwx 1 root root 33 2011-04-11 14:23 reqtimeout.load -> ../mods-available/reqtimeout.load lrwxrwxrwx 1 root root 30 2012-01-04 15:07 rewrite.load -> ../mods-available/rewrite.load .... We have already had some other proxy settings inside proxy.conf, therefore I also put the rewrite config into the proxy.conf. The contents inside proxy.conf is something like: <IfModule mod_proxy.c> # If you want to use apache2 as a forward proxy, uncomment the # 'ProxyRequests On' line and the <Proxy *> block below. # WARNING: Be careful to restrict access inside the <Proxy *> block. # Open proxy servers are dangerous both to your network and to the # Internet at large. # # If you only want to use apache2 as a reverse proxy/gateway in # front of some web application server, you DON'T need # 'ProxyRequests On'. #ProxyRequests On #<Proxy *> # AddDefaultCharset off # Order deny,allow # Deny from all # #Allow from .example.com #</Proxy> # Enable/disable the handling of HTTP/1.1 "Via:" headers. # ("Full" adds the server version; "Block" removes all outgoing Via: headers) # Set to one of: Off | On | Full | Block #ProxyVia Off ProxyPass /caarray http://caarray.privatenet.org:38080/caarray ProxyPassReverse /caarray http://caarray.privatenet.org:38080/caarray </IfModule> <IfModule mod_rewrite.c> RewriteEngine on RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /NFS/PATH/TO/galaxy/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /NFS/PATH/TO/galaxy/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /NFS/PATH/TO/galaxy/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /NFS/PATH/TO/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /NFS/PATH/TO/galaxy/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://galaxy.privatenet.org:8080$1 [P] </IfModule> Have I put the rewrite settings in the wrong place? I also want to clarify if I need to create the symlink to galaxy inside /var/www/? Please let me know if other details are needed. Cheers, Derrick

1 0

Re: [galaxy-dev] Tool shed and datatypes
by Greg Von Kuster 05 Jan '12

05 Jan '12

Your approach is great since it models the Galaxy distribution, and as you say, make sit clear to those downloading your repository. However, your datatype class module files will be found no matter where they are located within your repository hierarchy. On Jan 5, 2012, at 3:25 PM, Jim Johnson wrote: > Greg, > > I have been putting datatype def files in relative path: lib/galaxy/datatypes/ > This was just to make it more clear for someone manually modifying their own galaxy installation. > Is there any preferred best practice for where a datatypes implementation file should be? > > Thanks, > > JJ > > On 1/5/12 1:38 PM, Greg Von Kuster wrote: >> >> Hello Jim, >> >> I've implemented support for proprietary datatypes that use class modules included in tool shed repositories. To see how this works, you'll need at least change set revision 6479:4d131422777f, which is currently available only from our central repo at https://bitbucket.org/galaxy/galaxy-central. >> >> I've documented the way this works in the following 2 sections of the tool shed wiki. In the second section, I've taken the liberty of using your gmap tool repository as an example. i hope you don't mind. I've written the document section assuming that your gmap repository includes the 2 changes I've described below. >> >> http://wiki.g2.bx.psu.edu/Tool%20Shed#Including_proprietary_data_types_that… >> http://wiki.g2.bx.psu.edu/Tool%20Shed#Including_proprietary_data_types_that… >> >> There are 2 categories of datatypes that are currently supported: >> >> 1. data types that subclass from the datatype classes included in the Galaxy distribution - these require no code files that define proprietary datatype classes to be included in the tool shed repository, and are documented in the first wiki section listed above. >> >> 2. datatypes that use proprietary classes defined in code files included in the tool shed repository - documented in the second wiki section listed above. Your gmap tool suite falls into this category. >> >> If you make the following changes to your gmap tool suite, your proprietary data types will automatically load into a local Galaxy instance when the Galaxy admin installs your tool suite to that instance. The data types will be loaded at the time of installation as well as whenever the Galaxy server is stopped / restarted. I'll send you a separate message detailing the changes you'll need to make to your mothur tool suite. >> >> >> CHANGE 1 >> ---------------- >> Add a file named datatypes_conf.xml to your repository. This is the approach I'm using to support proprietary datatypes included in tool shed repositories instead f your proposed addition of datatypes in the tool config's <requirements> tag set. The datatypes_conf.xml file can be located anywhere in the repository, but the the obvious location for your gmap repository is your ~/tool-data directory. >> >> This file should contain the following datatype definitions. >> >> <?xml version="1.0"?> >> <datatypes> >> <datatype_files> >> <datatype_file name="gmap.py"/> >> </datatype_files> >> <registration> >> <datatype extension="gmapdb" type="galaxy.datatypes.gmap:GmapDB" display_in_upload="False"/> >> <datatype extension="gmapsnpindex" type="galaxy.datatypes.gmap:GmapSnpIndex" display_in_upload="False"/> >> <datatype extension="iit" type="galaxy.datatypes.gmap:IntervalIndexTree" display_in_upload="True"/> >> <datatype extension="splicesites.iit" type="galaxy.datatypes.gmap:SpliceSitesIntervalIndexTree" display_in_upload="True"/> >> <datatype extension="introns.iit" type="galaxy.datatypes.gmap:IntronsIntervalIndexTree" display_in_upload="True"/> >> <datatype extension="snps.iit" type="galaxy.datatypes.gmap:SNPsIntervalIndexTree" display_in_upload="True"/> >> <datatype extension="gmap_annotation" type="galaxy.datatypes.gmap:IntervalAnnotation" display_in_upload="False"/> >> <datatype extension="gmap_splicesites" type="galaxy.datatypes.gmap:SpliceSiteAnnotation" display_in_upload="True"/> >> <datatype extension="gmap_introns" type="galaxy.datatypes.gmap:IntronAnnotation" display_in_upload="True"/> >> <datatype extension="gmap_snps" type="galaxy.datatypes.gmap:SNPAnnotation" display_in_upload="True"/> >> </registration> >> <sniffers> >> <sniffer type="galaxy.datatypes.gmap:IntervalAnnotation"/> >> <sniffer type="galaxy.datatypes.gmap:SpliceSiteAnnotation"/> >> <sniffer type="galaxy.datatypes.gmap:IntronAnnotation"/> >> <sniffer type="galaxy.datatypes.gmap:SNPAnnotation"/> >> </sniffers> >> </datatypes> >> >> I noticed that your README in your current gmap repository on the main Galaxy tool shed includes the following datatype definitions, but they refer to classes that are not included in your repository so I've eliminated them from the above datatypes_conf.xml file. You may need to add the classes to your current gmap.py datatypes class file and add them to the above datatypes_conf.xml file if your tools actually require them. >> >> <datatype extension="tally.iit" type="galaxy.datatypes.gmap:TallyIntervalIndexTree" display_in_upload="True"/> >> <datatype extension="gsnap_tally" type="galaxy.datatypes.gmap:TallyAnnotation" display_in_upload="True"/> >> <datatype extension="gsnap" type="galaxy.datatypes.gmap:GsnapResult" display_in_upload="True"/> >> >> >> CHANGE 2 >> ---------------- >> Modules that include proprietary datatype class definitions cannot use relative import references for imported modules. Imports must be defined as absolute from the galaxy subdirectory inside the Galaxy root's lib subdirectory. So for your ~/lib/galaxy/datatypes/gmap.py datatypes module in your gmap repository, the following changes are necessary. >> >> Your current imports look like this: >> >> import logging >> import os,os.path,re >> import data >> from data import Text >> from galaxy import util >> from metadata import MetadataElement >> >> But they need to be changed to this - note the elimination of relative imports: >> >> import logging >> import os,os.path,re >> import galaxy.datatypes.data >> from galaxy.datatypes.data import Text >> from galaxy import util >> from galaxy.datatypes.metadata import MetadataElement >> >> Thanks very much for helping out with this, and please let me know if you bump into any problems. >> >> Greg Von Kuster >> >> >> On Oct 21, 2011, at 1:13 PM, Jim Johnson wrote: >> >>> Greg, >>> >>> I put the gmap tool suite in the galaxy Tool Shed, let me know if there is more I should do. >>> >>> It has 5 galaxy tools: >>> GMAP - Genomic Mapping and Alignment Program for mRNA and EST sequences >>> GSNAP - Genomic Short-read Nucleotide Alignment Program >>> GMAP Build - a database genome index for GMAP and GSNAP ( calls: gmap_build, iit_store, snpindex, cmetindex, atoiindex ) >>> GMAP SNP Index - build index files for known SNPs (calls: iit_store, snpindex) >>> GMAP IIT - Create a map store for known genes or SNPs (calls: iit_store) >>> >>> It uses these added datatypes: >>> % grep -E '(^class | file_ext)' lib/galaxy/datatypes/gmap.py >>> class GmapDB( Text ): >>> file_ext = 'gmapdb' >>> class GmapSnpIndex( Text ): >>> file_ext = 'gmapsnpindex' >>> class IntervalIndexTree( Text ): >>> file_ext = 'iit' >>> class SpliceSitesIntervalIndexTree( IntervalIndexTree ): >>> file_ext = 'splicesites.iit' >>> class IntronsIntervalIndexTree( IntervalIndexTree ): >>> file_ext = 'introns.iit' >>> class SNPsIntervalIndexTree( IntervalIndexTree ): >>> file_ext = 'snps.iit' >>> class IntervalAnnotation( Text ): >>> file_ext = 'gmap_annotation' >>> class SpliceSiteAnnotation(IntervalAnnotation): >>> file_ext = 'gmap_splicesites' >>> class IntronAnnotation(IntervalAnnotation): >>> file_ext = 'gmap_introns' >>> class SNPAnnotation(IntervalAnnotation): >>> file_ext = 'gmap_snps' >>> >>> I added a requirement tag for the datatypes to the tool-configs: >>> % grep 'requirement.*datatype' *.xml >>> gmap_build.xml: <requirement type="datatype">gmapdb</requirement> >>> gmap_build.xml: <requirement type="datatype">gmap_snps</requirement> >>> gmap.xml: <requirement type="datatype">gmapdb</requirement> >>> gmap.xml: <requirement type="datatype">gmap_annotation</requirement> >>> gmap.xml: <requirement type="datatype">gmap_splicesites</requirement> >>> gmap.xml: <requirement type="datatype">gmap_introns</requirement> >>> gmap.xml: <requirement type="datatype">gmap_snps</requirement> >>> gsnap.xml: <requirement type="datatype">gmapdb</requirement> >>> gsnap.xml: <requirement type="datatype">gmapsnpindex</requirement> >>> gsnap.xml: <requirement type="datatype">splicesites.iit</requirement> >>> gsnap.xml: <requirement type="datatype">introns.iit</requirement> >>> iit_store.xml: <requirement type="datatype">gmap_annotation</requirement> >>> iit_store.xml: <requirement type="datatype">gmap_snps</requirement> >>> iit_store.xml: <requirement type="datatype">iit</requirement> >>> iit_store.xml: <requirement type="datatype">splicesites.iit</requirement> >>> iit_store.xml: <requirement type="datatype">introns.iit</requirement> >>> iit_store.xml: <requirement type="datatype">snps.iit</requirement> >>> snpindex.xml: <requirement type="datatype">gmapsnpindex</requirement> >>> snpindex.xml: <requirement type="datatype">gmapdb</requirement> >>> snpindex.xml: <requirement type="datatype">gmap_snps</requirement> >>> snpindex.xml: <requirement type="datatype">snps.iit</requirement> >>> >>> Thanks, >>> >>> JJ >>> >> >> Greg Von Kuster >> Galaxy Development Team >> greg(a)bx.psu.edu >> >> >> > Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 1

Re: [galaxy-dev] Tool shed and datatypes
by Jim Johnson 05 Jan '12

05 Jan '12

Greg, It would be great if there were a way to expand upon the core datatypes using the ToolShed. Would it be possible to have a separate datatype repository within the ToolShed? Datatype name="" description="" datatype_dependencies=[] definition=<python code> The tool config could be expanded to have requirement for datatypes. <requirement type="datatype">ssmap</requirement> Table datatype Column | Type | Modifiers -------------+-----------------------------+--------------------------------------------------- id | integer | not null default nextval('datatype_id_seq'::regclass) name | character varying(255) | version | character varying(40) | description | text | definition | text | UNIQUE (name) Table datatype_datatype_association Column | Type | Modifiers -------------+-----------------------------+--------------------------------------------------- id | integer | not null default nextval('datatype_id_seq'::regclass) datatype_id | integer | requires_id | integer | FOREIGN KEY (datatype_id) REFERENCES datatype(id) FOREIGN KEY (requires_id) REFERENCES datatype(id) Then for my mothur metagenomics tools I could define: name="ssmap" description="Secondary Structure Map" version="1.0" datatype_dependencies=[tabular] definition= from galaxy.datatypes.tabular import Tabular class SecondaryStructureMap(Tabular): file_ext = 'ssmap' def __init__(self, **kwd): """Initialize secondary structure map datatype""" Tabular.__init__( self, **kwd ) self.column_names = ['Map'] def sniff( self, filename ): """ Determines whether the file is a secondary structure map format A single column with an integer value which indicates the row that this row maps to. check you make sure is structMap[10] = 380 then structMap[380] = 10. """ ... Then the align.check.xml tool_config could require the 'ssmap' datatype: <tool id="mothur_align_check" name="Align.check" version="1.19.0"> <description>Calculate the number of potentially misaligned bases</description> <requirements> <requirement type="binary">mothur</requirement> <requirement type="datatype">ssmap</requirement> </requirements> > John, > > I've been following this message thread, and it seems it's gone in a direction that differs from your initial question about the possibility for Galaxy to handle automatic editing of the datatypes_conf.xml file when certain Galaxy tool shed tools are automatically installed. There are some complexities to consider in attempting this. One of the issues to consider is that the work for adding support for a new datatype to Galaxy lies outside of the intended function of the tool shed. If new support is added to the Galaxy code base, an entry for that new datatype should be manually added to the table at the same time. There may be benefits to enabling automatic changes to datatype entries that already exist in the file (e.g., adding a new converter for an existing datatype entry), but perhaps adding a completely new datatype to the file may not be appropriate. I'll continue to think about this - send additional thought and feedback, as doing so is always helpful > > Thanks! > > Greg > > > On Oct 5, 2011, at 11:48 PM, Duddy, John wrote: > >> One of the things we’re facing is the sheer size of a whole human genome at 30x coverage. An effective way to deal with that is by compressing the FASTQ files. That works for BWA and our ELAND, which can directly read a compressed FASTQ, but other tools crash when reading compressed FASTQ filesfiles. One way to address that would be to introduce a new type, for example “CompressedFastQ”, with a conversion to FASTQ defined. BWA could take both types as input. This would allow the best of both worlds – efficient storage and use by all existing tools. >> >> Another example would be adding the CASAVA tools to Galaxy. Some of the statistics generation tools use custom file formats. To be able to make the use of those tools optional and configurable, they should be separate from the aligner, but that would require that Galaxy be made aware of the custom file formats – we’d have to add a datatype. >> >> John Duddy >> Sr. Staff Software Engineer >> Illumina, Inc. >> 9885 Towne Centre Drive >> San Diego, CA 92121 >> Tel: 858-736-3584 >> E-mail: jduddy at illumina.com >> >> From: Greg Von Kuster [mailto:greg at bx.psu.edu] >> Sent: Wednesday, October 05, 2011 6:25 PM >> To: Duddy, John >> Cc: galaxy-dev at lists.bx.psu.edu >> Subject: Re: [galaxy-dev] Tool shed and datatypes >> >> Hello John, >> >> The Galaxy tool shed currently is not enabled to automatically edit the datatypes_conf.xml file, although I could add this feature if the need exists. Can you elaborate on what you are looking to do regarding this? >> >> Thanks! >> >> >> On Oct 5, 2011, at 1:52 PM, Duddy, John wrote: >> >> >> Can we introduce new file types via tools in the tool shed? It seems Galaxy can load them if they are in the datatypes configuration file. Does tool installation automate the editing of that file? >> >> >> John Duddy >> Sr. Staff Software Engineer >> Illumina, Inc. >> 9885 Towne Centre Drive >> San Diego, CA 92121 >> Tel: 858-736-3584 >> E-mail: jduddy at illumina.com >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ >> >> Greg Von Kuster >> Galaxy Development Team >> greg at bx.psu.edu >>

7 15