Greetings.
Does anyone per chance have a simple working example of using composite data types? I've gotten as far as I can with the galaxy documentation, and I can't seem to get the additional files bundled into an extra files directory.
(note, I have my example submitted w/ files attached, but currently pending moderator approval)
much appreciated,
~brian
On Sun, May 17, 2015 at 5:44 PM, Brian Haas bhaas@broadinstitute.org wrote:
Greetings.
Does anyone per chance have a simple working example of using composite data types? I've gotten as far as I can with the galaxy documentation, and I can't seem to get the additional files bundled into an extra files directory.
(note, I have my example submitted w/ files attached, but currently pending moderator approval)
much appreciated,
~brian
Hi Brian,
I can suggest looking at the BLAST database datatypes defined here: https://github.com/peterjc/galaxy_blast/tree/master/datatypes/blast_datatype...
Also study the built in examples like HTML (with images as extra files): https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/data.p...
Regards,
Peter
Thanks, Peter! I've verified that I can get your code to work - bundling the composite files. I'll keep plugging away at my little example and see if I can figure out what's going awry.
more later. Thx again!
~brian
On Mon, May 18, 2015 at 1:55 PM, Brian Haas bhaas@broadinstitute.org wrote:
Thanks, Peter! I've verified that I can get your code to work - bundling the composite files. I'll keep plugging away at my little example and see if I can figure out what's going awry.
more later. Thx again!
Good luck - compound datatypes are one of the complex ways to extend Galaxy. If your code was online in GitHub or BitBucket, it would be easier for us (including the Galaxy team) to comment on.
Regards,
Peter
In the next couple of hours, if I'm not able to resolve this, I'll post my ultra-simple example on github and I'll be grateful for others to take a look.
cheers,
~brian
On Mon, May 18, 2015 at 9:23 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
On Mon, May 18, 2015 at 1:55 PM, Brian Haas bhaas@broadinstitute.org wrote:
Thanks, Peter! I've verified that I can get your code to work -
bundling
the composite files. I'll keep plugging away at my little example and
see
if I can figure out what's going awry.
more later. Thx again!
Good luck - compound datatypes are one of the complex ways to extend Galaxy. If your code was online in GitHub or BitBucket, it would be easier for us (including the Galaxy team) to comment on.
Regards,
Peter
Hi Brian,
Your original email with attachments has made it to the list now, http://dev.list.galaxyproject.org/difficulty-creating-composite-data-type-tp...
Title: "difficulty creating composite data type"
Peter
On Mon, May 18, 2015 at 2:25 PM, Brian Haas bhaas@broadinstitute.org wrote:
In the next couple of hours, if I'm not able to resolve this, I'll post my ultra-simple example on github and I'll be grateful for others to take a look.
cheers,
~brian
Fantastic!
I've actually made a few changes. I'm putting my latest code up on github now and will share it in just a moment. It'll be much easier this way - as you suggested.
best,
~brian
On Mon, May 18, 2015 at 9:58 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
Hi Brian,
Your original email with attachments has made it to the list now,
http://dev.list.galaxyproject.org/difficulty-creating-composite-data-type-tp...
Title: "difficulty creating composite data type"
Peter
On Mon, May 18, 2015 at 2:25 PM, Brian Haas bhaas@broadinstitute.org wrote:
In the next couple of hours, if I'm not able to resolve this, I'll post
my
ultra-simple example on github and I'll be grateful for others to take a look.
cheers,
~brian
I've uploaded my simple example here:
https://github.com/brianjohnhaas/galaxy_composite_datatype_test
I can't seem to get it to bundle the output files no matter what I've tried. Help is much appreciated. :)
best,
~brian
Hi Peter,
I'm actually more confused now that I was when I started... I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored. I then make 'composite_type = None' and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
Hi Brian,
In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path. Then you create that directory and place the files you want within that directory.
Thanks for using Galaxy,
Dan
On May 18, 2015, at 2:02 PM, Brian Haas bhaas@broadinstitute.org wrote:
Hi Peter,
I'm actually more confused now that I was when I started... I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored. I then make 'composite_type = None' and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
THANK YOU!!! :)
The fundamental piece that I was missing at: https://wiki.galaxyproject.org/Admin/Datatypes/Composite%20Datatypes
was the
<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} ${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>
It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.
This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).
thanks again!
~brian
On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg dan@bx.psu.edu wrote:
Hi Brian,
In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path. Then you create that directory and place the files you want within that directory.
Thanks for using Galaxy,
Dan
On May 18, 2015, at 2:02 PM, Brian Haas bhaas@broadinstitute.org wrote:
Hi Peter,
I'm actually more confused now that I was when I started... I was able
to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect
- the files were still being stored. I then make 'composite_type = None'
and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back over
to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could verify
that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my
code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
It is a wiki, if you miss something - please add it.
Thanks
Martin
On Mon, May 18, 2015 at 3:08 PM Brian Haas bhaas@broadinstitute.org wrote:
THANK YOU!!! :)
The fundamental piece that I was missing at: https://wiki.galaxyproject.org/Admin/Datatypes/Composite%20Datatypes
was the
<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} ${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>
It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.
This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).
thanks again!
~brian
On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg dan@bx.psu.edu wrote:
Hi Brian,
In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path. Then you create that directory and place the files you want within that directory.
Thanks for using Galaxy,
Dan
On May 18, 2015, at 2:02 PM, Brian Haas bhaas@broadinstitute.org wrote:
Hi Peter,
I'm actually more confused now that I was when I started... I was able
to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect
- the files were still being stored. I then make 'composite_type = None'
and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back
over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could verify
that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my
code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
--
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Ah! will do. :)
On Mon, May 18, 2015 at 3:09 PM, Martin Čech marten@bx.psu.edu wrote:
It is a wiki, if you miss something - please add it.
Thanks
Martin
On Mon, May 18, 2015 at 3:08 PM Brian Haas bhaas@broadinstitute.org wrote:
THANK YOU!!! :)
The fundamental piece that I was missing at: https://wiki.galaxyproject.org/Admin/Datatypes/Composite%20Datatypes
was the
<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} ${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>
It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.
This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).
thanks again!
~brian
On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg dan@bx.psu.edu wrote:
Hi Brian,
In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path. Then you create that directory and place the files you want within that directory.
Thanks for using Galaxy,
Dan
On May 18, 2015, at 2:02 PM, Brian Haas bhaas@broadinstitute.org wrote:
Hi Peter,
I'm actually more confused now that I was when I started... I was
able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file. I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area. I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored. I then make 'composite_type = None' and it still stored the files to my dismay.
Since I couldn't get my copy to work as I had expected, I went back
over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
Of course, I restarted galaxy after each modification, and could
verify that it was reading the new files and had byte-compiled the py mods.
I'm clearly not grasping something fundamental here. I can't get my
code to bundle files, and I can't get your code to *not* bundle files. ;)
Any ideas on where to go to troubleshoot this further?
many thanks,
~brian
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
--
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I've updated my code, which now generates the .extra_files directory: https://github.com/brianjohnhaas/galaxy_composite_datatype_test
I'm now trying to get the
composite_type = 'auto_primary_file*' *to work, but all I'm getting is an empty file. My module code simply includes:
class CompositeDataTypeTest( Text ):
file_ext = 'cdt' composite_type = 'auto_primary_file'
def generate_primary_file( self, dataset = None ): log.debug("\n\n######## generate_primary_file() for Test data#######\n\n") return('<html><body>my test html</body></html>')
and from what I can tell, the generate_primary_file() isn't being called, as I'm not detecting the log entry in the console.
any ideas?
thanks in advance!
~brian
From some more testing it appears that the generate_primary_file() is
called when a user manually uploads a composite data type (cdt), but it's not called when the cdt is generated as a product of a run. That's fine. I can work around that.
I am curious about how cdts are uploaded manually, though. If I select multiple files to upload as a cdt, they're not linked but instead uploaded as individual cdts. Is there a way to upload multiple files as a cdt using the latest dev galaxy? (I noticed the file upload gui has changed from earlier releases, and I'm wondering if support for cdt uploads is now lacking)
much appreciated,
~brian
galaxy-dev@lists.galaxyproject.org
-
Brian Haas -
Daniel Blankenberg -
Martin Čech -
Peter Cock