Hi everyone,

I'm trying to add a tool which generates 2 files, that I will call ".xxx" (a text file) and ".yyy" (a binary file)  . Both files are needed to use the result of my tool with an other tool I've added.
So I wanted to create a composite datatype , that I will call ".composite", whose components are ".xxx" and ".yyy".

I've declared the datatype ".xxx", ".yyy" and ".composite" in the datatypes_conf.xml file, and written the required python files . Now, ".xxx", ".yyy" and ".composite" appear in Get Data's "file format" .


These are my files :

In datatype_conf.xml :

<datatype extension="xxx" type="galaxy.datatypes.xxx:xxx" mimetype="text/html" display_in_upload = "True" subclass="True"/>
    <datatype extension="yyy" type="galaxy.datatypes.yyy:yyy" mimetype="application/octet-stream" display_in_upload = "True" subclass="True" />
    <datatype extension="composite" type="galaxy.datatypes.composite:Composite" mimetype="text/html" display_in_upload="True"/>


xxx.py (summarized) :

import logging
from metadata import MetadataElement
from data import Text

log = logging.getLogger(__name__)

class xxx(Text):      
    file_ext = "xxx"

        def __init__( self, **kwd ):
        Text.__init__( self, **kwd )
    


yyy.py (summarized) :  

import logging
from metadata import MetadataElement
from data import Text

log = logging.getLogger(__name__)

# yyy is a binary file, don't know what to put instead of "Text". "Binary" and "Bin" don't work.
class yyy(Text):     
    file_ext = "yyy"

        def __init__( self, **kwd ):
        Text.__init__( self, **kwd )
        


composite.py (summarized) :

import logging
from metadata import MetadataElement
from data import Text

log = logging.getLogger(__name__)

class Composite(Text):      
    composite_type = 'auto_primary_file'
    MetadataElement( name="base_name", desc="base name for all transformed versions of this index dataset", default="your_index", readonly=True, set_in_upload=True)
    file_ext = 'composite'

    def __init__( self, **kwd ):
        Text.__init__( self, **kwd )
        self.add_composite_file( '%s.xxx', description = "XXX file", substitute_name_with_metadata = 'base_name')
        self.add_composite_file( '%s.yyy', description = "YYY file", substitute_name_with_metadata = 'base_name', is_binary = True )



Atfer having read Composite Datatypes in the wiki, my myTool.xml looks like :

<tool id="my tool">
   <command> path/to/crac-index-wrapper.sh
   ${os.path.join( $output_name_yyy.extra_files_path, '%s.yyy')} ${os.path.join( $output_name_xxx.extra_files_path, '%s.xxx' )} $input_file
   </command>
   <inputs>
      <param name="output_name" type="text" value ="IndexOutput" label="Output name"/>
      <param name="input_file" type="data" label="Source file" format="fasta"/>
   </inputs> 
   <outputs>
      <data format="ssa" name="output_name_ssa" from_work_dir="crac-index_output.ssa" label="CRAC-index: ${output_name}.ssa">
      </data>
    <data format="conf" name="output_name_conf" from_work_dir="crac-index_output.conf" label="CRAC-index: ${output_name}.conf">
     </data>
   </outputs>
</tool>



I have 2 main problems  :

When I upload a xxx file via "Get Data", there's no problem. However, when I upload a yyy file (the binary one), history bloc rests eternally blue ("uploading dataset") , even for a small file.


The second problem is that I want my tool to only generate the .composite file on the history, and not each of .xxx and .yyy.
. But when I run my tool I still have 2 outputs displayed in the history : one for xxx and one for yyy. Furthermore, neither of them work, and I have the following message :

path/to/myTool-wrapper.sh: 6: path/to/myTool-wrapper.sh.sh: cannot create /home/myName/work/galaxy-dist/database/files/000/dataset_302_files/%s.yyy.xxx: Directory nonexistent
path/to/myTool-wrapper.sh: 6: path/to/myTool-wrapper.sh: cannot create /home/myName/work/galaxy-dist/database/files/000/dataset_302_files/%s.yyy.yyy: Directory nonexistent
path/to/myTool-wrapper.sh: 11: path/to/myTool-wrapper.sh: Syntax error: redirection unexpected


So I've checked manually in /home/myName/work/galaxy-dist/database/files/000/ and there's only "dataset_302.dat", an empty file.
(And whatsmore, I don't understand why I get in the message "%s.yyy.xxx" and "%s.yyy.yyy" instead of "%s.yyy" and "%s.xxx" ...)


Then I've looked the example of rgenetics.xml, and tried to change the command line and the output :
 
<tool id="my tool">
   <command> path/to/myTool-wrapper.sh '$output_name.extra_files_path/$output_name.metadata.base_name' $input_file
   </command>
   <inputs>
      <param name="output_name" type="text" value ="IndexOutput" label="Output name" />
      <param name="input_file" type="data" label="Source file" />
   </inputs>
   <outputs>    
      <data format="html" name="output" label="myTool: ${output_name}.html" metadata_source="input_file"/>
   </outputs>
</tool>

This gave me :

Traceback (most recent call last):
  File "/home/myName/work/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 59, in run_job
    job_wrapper.prepare()
  File "/home/myName/work/galaxy-dist/lib/galaxy/jobs/__init__.py", line 429, in prepare
    self.command_line = self.tool.build_command_line( param_dict )
  File "/home/myName/work/galaxy-dist/lib/galaxy/tools/__init__.py", line 1971, in build_command_line
    command_line = fill_template( self.command, context=param_dict )
  File "/home/myName/work/galaxy-dist/lib/galaxy/util/template.py", line 9, in fill_template
    return str( Template( source=template_text, searchList=[context] ) )
  File "/home/myName/work/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__
    return getattr(self, mainMethName)()
  File "cheetah_DynamicallyCompiledCheetahTemplate_1339157051_58_87978.py", line 83, in respond
NotFound: cannot find 'extra_files_path' while searching for 'output_name.extra_files_path'


So now I don't know which way is the one to follow : the first one inspired by the example in the wiki, or the second one inspired by rgenetics.xml. And what's wrong with it...
I will really appreciate any suggestion !

Best regards,

Marine