simple example of composite data types
Greetings. Does anyone per chance have a simple working example of using composite data types? I've gotten as far as I can with the galaxy documentation, and I can't seem to get the additional files bundled into an extra files directory. (note, I have my example submitted w/ files attached, but currently pending moderator approval) much appreciated, ~brian -- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
On Sun, May 17, 2015 at 5:44 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
Greetings.
Does anyone per chance have a simple working example of using composite data types? I've gotten as far as I can with the galaxy documentation, and I can't seem to get the additional files bundled into an extra files directory.
(note, I have my example submitted w/ files attached, but currently pending moderator approval)
much appreciated,
~brian
Hi Brian, I can suggest looking at the BLAST database datatypes defined here: https://github.com/peterjc/galaxy_blast/tree/master/datatypes/blast_datatype... Also study the built in examples like HTML (with images as extra files): https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/data.p... Regards, Peter
Thanks, Peter! I've verified that I can get your code to work - bundling the composite files. I'll keep plugging away at my little example and see if I can figure out what's going awry. more later. Thx again! ~brian
On Mon, May 18, 2015 at 1:55 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
Thanks, Peter! I've verified that I can get your code to work - bundling the composite files. I'll keep plugging away at my little example and see if I can figure out what's going awry.
more later. Thx again!
Good luck - compound datatypes are one of the complex ways to extend Galaxy. If your code was online in GitHub or BitBucket, it would be easier for us (including the Galaxy team) to comment on. Regards, Peter
In the next couple of hours, if I'm not able to resolve this, I'll post my ultra-simple example on github and I'll be grateful for others to take a look. cheers, ~brian On Mon, May 18, 2015 at 9:23 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, May 18, 2015 at 1:55 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
Thanks, Peter! I've verified that I can get your code to work - bundling the composite files. I'll keep plugging away at my little example and see if I can figure out what's going awry.
more later. Thx again!
Good luck - compound datatypes are one of the complex ways to extend Galaxy. If your code was online in GitHub or BitBucket, it would be easier for us (including the Galaxy team) to comment on.
Regards,
Peter
-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
Hi Brian, Your original email with attachments has made it to the list now, http://dev.list.galaxyproject.org/difficulty-creating-composite-data-type-tp... Title: "difficulty creating composite data type" Peter On Mon, May 18, 2015 at 2:25 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
In the next couple of hours, if I'm not able to resolve this, I'll post my ultra-simple example on github and I'll be grateful for others to take a look.
cheers,
~brian
Fantastic! I've actually made a few changes. I'm putting my latest code up on github now and will share it in just a moment. It'll be much easier this way - as you suggested. best, ~brian On Mon, May 18, 2015 at 9:58 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi Brian,
Your original email with attachments has made it to the list now,
http://dev.list.galaxyproject.org/difficulty-creating-composite-data-type-tp...
Title: "difficulty creating composite data type"
Peter
On Mon, May 18, 2015 at 2:25 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
In the next couple of hours, if I'm not able to resolve this, I'll post my ultra-simple example on github and I'll be grateful for others to take a look.
cheers,
~brian
-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
I've uploaded my simple example here: https://github.com/brianjohnhaas/galaxy_composite_datatype_test I can't seem to get it to bundle the output files no matter what I've tried. Help is much appreciated. :) best, ~brian
Hi Peter, I'm actually more confused now that I was when I started... I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored. I then make 'composite_type = None' and it still stored the files to my dismay. Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect. Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods. I'm clearly not grasping something fundamental here. I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;) Any ideas on where to go to troubleshoot this further? many thanks, ~brian
Hi Brian, In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path. Then you create that directory and place the files you want within that directory. Thanks for using Galaxy, Dan On May 18, 2015, at 2:02 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
Hi Peter,
I'm actually more confused now that I was when I started... I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored. I then make 'composite_type = None' and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
THANK YOU!!! :)
The fundamental piece that I was missing at:
https://wiki.galaxyproject.org/Admin/Datatypes/Composite%20Datatypes
was the
<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path,
'results.txt' )} ${os.path.join( input1.extra_files_path,
'results.dat' )} $output1</command>
It would be great if the documentation could be updated a bit for clarity
on the importance of the '.extra_files_path', such as 'important: your tool
needs to create this directory and store the additional files that you want
there'.
This is a big leap forward for me. I'll now get on to testing the other
aspects of it. (still climbing the galaxy learning curve).
thanks again!
~brian
On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg <dan@bx.psu.edu> wrote:
> Hi Brian,
>
> In your example tool, you need to pass “${ test_cdt.files_path }" to your
> perl script, which is a not-yet-existing directory for the files you want
> to be part of the dataset’s extra files path.
> Then you create that directory and place the files you want within that
> directory.
>
>
>
> Thanks for using Galaxy,
>
> Dan
>
>
> On May 18, 2015, at 2:02 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
>
> > Hi Peter,
> >
> > I'm actually more confused now that I was when I started... I was able
> to get your code working, focusing on the 'makeblastdb' with a nucleotide
> fasta file. I figured that experimenting with this example would lead to
> some key insights, particularly by dropping certain files that are stored
> as extra-files area. I copied everything over as a new tool, made it a
> new data type, and trimmed the blast.py code down to the bare essentials
> for testing just the nucleotide makeblastdb functionality. However,
> commenting out 'self.add_composite_file()' for various files had no effect
> - the files were still being stored. I then make 'composite_type = None'
> and it still stored the files to my dismay.
> >
> > Since I couldn't get my copy to work as I had expected, I went back over
> to your 'blast.py' and just commented out the ''self.add_composite_file()'
> lines and set 'composite_type = None' .... and no effect.
> >
> > Of course, I restarted galaxy after each modification, and could verify
> that it was reading the new files and had byte-compiled the py mods.
> >
> > I'm clearly not grasping something fundamental here. I can't get my
> code to bundle files, and I can't get your code to *not* bundle files. ;)
> >
> > Any ideas on where to go to troubleshoot this further?
> >
> > many thanks,
> >
> > ~brian
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client. To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> > https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
>
>
--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
It is a wiki, if you miss something - please add it.
Thanks
Martin
On Mon, May 18, 2015 at 3:08 PM Brian Haas <bhaas@broadinstitute.org> wrote:
> THANK YOU!!! :)
>
> The fundamental piece that I was missing at:
> https://wiki.galaxyproject.org/Admin/Datatypes/Composite%20Datatypes
>
> was the
>
> <command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} ${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>
>
>
> It would be great if the documentation could be updated a bit for clarity
> on the importance of the '.extra_files_path', such as 'important: your tool
> needs to create this directory and store the additional files that you want
> there'.
>
> This is a big leap forward for me. I'll now get on to testing the other
> aspects of it. (still climbing the galaxy learning curve).
>
> thanks again!
>
> ~brian
>
>
>
>
> On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg <dan@bx.psu.edu>
> wrote:
>
>> Hi Brian,
>>
>> In your example tool, you need to pass “${ test_cdt.files_path }" to your
>> perl script, which is a not-yet-existing directory for the files you want
>> to be part of the dataset’s extra files path.
>> Then you create that directory and place the files you want within that
>> directory.
>>
>>
>>
>> Thanks for using Galaxy,
>>
>> Dan
>>
>>
>> On May 18, 2015, at 2:02 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
>>
>> > Hi Peter,
>> >
>> > I'm actually more confused now that I was when I started... I was able
>> to get your code working, focusing on the 'makeblastdb' with a nucleotide
>> fasta file. I figured that experimenting with this example would lead to
>> some key insights, particularly by dropping certain files that are stored
>> as extra-files area. I copied everything over as a new tool, made it a
>> new data type, and trimmed the blast.py code down to the bare essentials
>> for testing just the nucleotide makeblastdb functionality. However,
>> commenting out 'self.add_composite_file()' for various files had no effect
>> - the files were still being stored. I then make 'composite_type = None'
>> and it still stored the files to my dismay.
>> >
>> > Since I couldn't get my copy to work as I had expected, I went back
>> over to your 'blast.py' and just commented out the
>> ''self.add_composite_file()' lines and set 'composite_type = None' .... and
>> no effect.
>> >
>> > Of course, I restarted galaxy after each modification, and could verify
>> that it was reading the new files and had byte-compiled the py mods.
>> >
>> > I'm clearly not grasping something fundamental here. I can't get my
>> code to bundle files, and I can't get your code to *not* bundle files. ;)
>> >
>> > Any ideas on where to go to troubleshoot this further?
>> >
>> > many thanks,
>> >
>> > ~brian
>> >
>> > ___________________________________________________________
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client. To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> > https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> > http://galaxyproject.org/search/mailinglists/
>>
>>
>
>
> --
> --
> Brian J. Haas
> The Broad Institute
> http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
Ah! will do. :) On Mon, May 18, 2015 at 3:09 PM, Martin Čech <marten@bx.psu.edu> wrote:
It is a wiki, if you miss something - please add it.
Thanks
Martin
On Mon, May 18, 2015 at 3:08 PM Brian Haas <bhaas@broadinstitute.org> wrote:
THANK YOU!!! :)
The fundamental piece that I was missing at: https://wiki.galaxyproject.org/Admin/Datatypes/Composite%20Datatypes
was the
<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} ${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>
It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.
This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).
thanks again!
~brian
On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg <dan@bx.psu.edu> wrote:
Hi Brian,
In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path. Then you create that directory and place the files you want within that directory.
Thanks for using Galaxy,
Dan
On May 18, 2015, at 2:02 PM, Brian Haas <bhaas@broadinstitute.org> wrote:
Hi Peter,
I'm actually more confused now that I was when I started... I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored. I then make 'composite_type = None' and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- -- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas <http://broad.mit.edu/~bhaas>
I've updated my code, which now generates the .extra_files directory: https://github.com/brianjohnhaas/galaxy_composite_datatype_test I'm now trying to get the composite_type = 'auto_primary_file*' *to work, but all I'm getting is an empty file. My module code simply includes: class CompositeDataTypeTest( Text ):
file_ext = 'cdt' composite_type = 'auto_primary_file'
def generate_primary_file( self, dataset = None ): log.debug("\n\n######## generate_primary_file() for Test data #######\n\n") return('<html><body>my test html</body></html>')
and from what I can tell, the generate_primary_file() isn't being called, as I'm not detecting the log entry in the console. any ideas? thanks in advance! ~brian
From some more testing it appears that the generate_primary_file() is called when a user manually uploads a composite data type (cdt), but it's not called when the cdt is generated as a product of a run. That's fine. I can work around that.
I am curious about how cdts are uploaded manually, though. If I select multiple files to upload as a cdt, they're not linked but instead uploaded as individual cdts. Is there a way to upload multiple files as a cdt using the latest dev galaxy? (I noticed the file upload gui has changed from earlier releases, and I'm wondering if support for cdt uploads is now lacking) much appreciated, ~brian
participants (4)
-
Brian Haas -
Daniel Blankenberg -
Martin Čech -
Peter Cock