Marine,
Sorry to hear you're having problems - composite objects definitely do
work but they are definitely not simple or properly documented.
I don't really have time to figure out exactly what the problem is but
one very obvious error message
NotFound: cannot find 'extra_files_path' while searching for
'output_name.extra_files_path'
is telling you that a new output file does not have an
extra_files_path at job submission.
Try output_name.files_path - why it differs is something I do not
understand but I've learned to live with....
If you grep for files_path in your tool/*/*.xml files, you'll find
lots of examples of tools using files_path and extra_files_path
(mostly html files) and studying those working examples might be
useful in getting your code to work?
cheers...
On Tue, Jun 12, 2012 at 6:24 PM, Marine
Rohmer <
marine.rohmer@yahoo.fr> wrote:
> Hi,
>
> Maybe my message was not understandable enough. I really need your help, so
> I'll try to be more concise :
>
> How do I make a composite output from 2 datatypes that I have declared
> myself ?
> I've followed the "Composite Datatypes" wiki but it seems that I've missed
> something...
> My composite datatype appears well in "file format" from Get Data's upload
> file section, but when I run my tool, I have 2 outputs which are the
> components of my primary datatype, instead of only one output.
>
> Best regards,
>
> Marine
>
>
>
>
> ________________________________
> De : Marine Rohmer <
marine.rohmer@yahoo.fr>
> À : "
galaxy-dev@lists.bx.psu.edu" <
galaxy-dev@lists.bx.psu.edu>
> Envoyé le : Vendredi 8 juin 2012 15h15
> Objet : Composite output with self-declarated datatype
>
> Hi everyone,
>
> I'm trying to add a tool which generates 2 files, that I will call ".xxx" (a
> text file) and ".yyy" (a binary file) . Both files are needed to use the
> result of my tool with an other tool I've added.
> So I wanted to create a composite datatype , that I will call ".composite",
> whose components are ".xxx" and ".yyy".
>
> I've declared the datatype ".xxx", ".yyy" and ".composite" in the
> datatypes_conf.xml
file, and written the required python files . Now,
> ".xxx", ".yyy" and ".composite" appear in Get Data's "file format" .
>
>
> These are my files :
>
> In datatype_conf.xml :
>
> <datatype extension="xxx" type="galaxy.datatypes.xxx:xxx"
> mimetype="text/html" display_in_upload = "True" subclass="True"/>
> <datatype extension="yyy" type="galaxy.datatypes.yyy:yyy"
> mimetype="application/octet-stream" display_in_upload = "True"
> subclass="True" />
> <datatype extension="composite"
> type="galaxy.datatypes.composite:Composite" mimetype="text/html"
> display_in_upload="True"/>
>
>
> xxx.py (summarized) :
>
> import logging
> from metadata import MetadataElement
> from data import Text
>
> log = logging.getLogger(__name__)
>
> class
xxx(Text):
> file_ext = "xxx"
>
> def __init__( self, **kwd ):
> Text.__init__( self, **kwd )
>
>
>
> yyy.py (summarized) :
>
> import logging
> from metadata import MetadataElement
> from data import Text
>
> log = logging.getLogger(__name__)
>
> # yyy is a binary file, don't know what to put instead of "Text". "Binary"
> and "Bin" don't work.
> class yyy(Text):
> file_ext = "yyy"
>
> def __init__( self, **kwd ):
> Text.__init__( self, **kwd )
>
>
>
> composite.py (summarized) :
>
> import logging
> from metadata import MetadataElement
> from data import Text
>
> log
= logging.getLogger(__name__)
>
> class Composite(Text):
> composite_type = 'auto_primary_file'
> MetadataElement( name="base_name", desc="base name for all transformed
> versions of this index dataset", default="your_index", readonly=True,
> set_in_upload=True)
> file_ext = 'composite'
>
> def __init__( self, **kwd ):
> Text.__init__( self, **kwd )
> self.add_composite_file( '%s.xxx', description = "XXX file",
> substitute_name_with_metadata = 'base_name')
> self.add_composite_file( '%s.yyy', description = "YYY file",
> substitute_name_with_metadata = 'base_name', is_binary = True )
>
>
>
> Atfer having read Composite Datatypes in the wiki, my myTool.xml looks
like
> :
>
> <tool id="my tool">
> <command> path/to/crac-index-wrapper.sh
> ${os.path.join( $output_name_yyy.extra_files_path, '%s.yyy')}
> ${os.path.join( $output_name_xxx.extra_files_path, '%s.xxx' )} $input_file
> </command>
> <inputs>
> <param name="output_name" type="text" value ="IndexOutput"
> label="Output name"/>
> <param name="input_file" type="data" label="Source file"
> format="fasta"/>
> </inputs>
> <outputs>
> <data format="ssa" name="output_name_ssa"
> from_work_dir="crac-index_output.ssa" label="CRAC-index:
> ${output_name}.ssa">
> </data>
>
<data format="conf" name="output_name_conf"
> from_work_dir="crac-index_output.conf" label="CRAC-index:
> ${output_name}.conf">
> </data>
> </outputs>
> </tool>
>
>
>
> I have 2 main problems :
>
> When I upload a xxx file via "Get Data", there's no problem. However, when I
> upload a yyy file (the binary one), history bloc rests eternally blue
> ("uploading dataset") , even for a small file.
>
>
> The second problem is that I want my tool to only generate the .composite
> file on the history, and not each of .xxx and .yyy.
> . But when I run my tool I still have 2 outputs displayed in the history :
> one for xxx and one for yyy. Furthermore, neither of them work, and I have
> the following message :
>
> path/to/myTool-wrapper.sh: 6:
path/to/
myTool-wrapper.sh.sh: cannot create
> /home/myName/work/galaxy-dist/database/files/000/dataset_302_files/%s.yyy.xxx:
> Directory nonexistent
> path/to/myTool-wrapper.sh: 6: path/to/myTool-wrapper.sh: cannot create
> /home/myName/work/galaxy-dist/database/files/000/dataset_302_files/%s.yyy.yyy:
> Directory nonexistent
> path/to/myTool-wrapper.sh: 11: path/to/myTool-wrapper.sh: Syntax error:
> redirection unexpected
>
>
> So I've checked manually in
> /home/myName/work/galaxy-dist/database/files/000/ and there's only
> "dataset_302.dat", an empty file.
> (And whatsmore, I don't understand why I get in the message "%s.yyy.xxx" and
> "%s.yyy.yyy" instead of "%s.yyy" and "%s.xxx" ...)
>
>
> Then I've looked the example of rgenetics.xml, and tried to change the
> command line and the output :
>
> <tool id="my tool">
>
<command> path/to/myTool-wrapper.sh
> '$output_name.extra_files_path/$output_name.metadata.base_name' $input_file
> </command>
> <inputs>
> <param name="output_name" type="text" value ="IndexOutput"
> label="Output name" />
> <param name="input_file" type="data" label="Source file" />
> </inputs>
> <outputs>
> <data format="html" name="output" label="myTool: ${output_name}.html"
> metadata_source="input_file"/>
> </outputs>
> </tool>
>
> This gave me :
>
> Traceback (most recent call last):
> File "/home/myName/work/galaxy-dist/lib/galaxy/jobs/runners/local.py",
> line 59, in run_job
>
job_wrapper.prepare()
> File "/home/myName/work/galaxy-dist/lib/galaxy/jobs/__init__.py", line
> 429, in prepare
> self.command_line = self.tool.build_command_line( param_dict )
> File "/home/myName/work/galaxy-dist/lib/galaxy/tools/__init__.py", line
> 1971, in build_command_line
> command_line = fill_template( self.command, context=param_dict )
> File "/home/myName/work/galaxy-dist/lib/galaxy/util/template.py", line 9,
> in fill_template
> return str( Template( source=template_text, searchList=[context] ) )
> File
> "/home/myName/work/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py",
> line 1004, in __str__
> return getattr(self, mainMethName)()
> File "cheetah_DynamicallyCompiledCheetahTemplate_1339157051_58_87978.py",
>
line 83, in respond
> NotFound: cannot find 'extra_files_path' while searching for
> 'output_name.extra_files_path'
>
>
> So now I don't know which way is the one to follow : the first one inspired
> by the example in the wiki, or the second one inspired by rgenetics.xml. And
> what's wrong with it...
> I will really appreciate any suggestion !