Hi Bob,

Thank you Eric,

But I want to be clear after spending 3 days with help from others trying to get this to work.

1. Config/plugin/visualization is where code goes to have a Galaxy internal module display a user defined data type -- mine is binary

Yes.

2. Datatype/data_applciations is where code goes for external displays that use get and post reside.

Yes.

3. which if any of 1or2 above use the Tools/Visualization directory. Is that for the return of a Get or Post?

This is a tool. A completely different concept. Tools are entities that can run one a cluster and can run separated from the user interface.

4. I create my own datatype in data_config_type.xml with a <datatype> tag

Yes.

5.

5 is missing :) Cheers, Bjoern

----- Original Message -----From: Eric Rasche <esr@tamu.edu>To: rbrown1422@comcast.netCc: galaxy-dev <galaxy-dev@lists.bx.psu.edu>Sent: Tue, 08 Dec 2015 02:34:47 -0000 (UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool?

Hi Bob,

Yes, galaxy supports output formats of HTML+JS, though the tool producing them may need to be whitelisted (grep for 'whitelist' in your galaxy.ini) for display to work properly.

If you wish to write your own visualization plugin for a specific datatype, there are assorted pieces of documentation on the wiki that might help you (though I usually just look at one of the other existing visualization plugins and copy+paste).

Cheers, Eric

2015-12-07 20:08 GMT-06:00 <rbrown1422@comcast.net>:

Good evening Galaxy team,

Is it possible to run JavaScript or HTML as a tool or a display module?

I see support for URLs, e.g. UCSC Genome Browser, but I would like to process a specific datatype to allow it to be viewed in Galaxy -- but the application is a combination of HTML and JavaScript.

Does Galaxy support this?

Thank you, Bob ___________________________________________________________

Please keep all replies on the list by using "reply all"

in your mail client. To manage your subscriptions to this

and other Galaxy lists, please use the interface at:

https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:

http://galaxyproject.org/search/mailinglists/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Thank you Eric, But I want to be clear after spending 5 days, reading many Galaxy wiki links, looking at samples, and with help from others trying to get this to work. I have written many tools but never tried to display a unique datatype before. Below is a series of questions and steps I followed trying the suggestions from early posts. My goals are: -- To get the visualization icon to show in the bottom left of my history output file? -- click the visualization icon to display my data (that I will process to display) inside Galaxy. The steps below are in order of urgency:) Thank you for you patience. 1. /Config/plugin/visualization is where code goes to have a Galaxy internal module display a user defined data type? -- mine is binary Without getting to in the weeds under this directory I have a mydatatype dir with subdirectories /config w xml, /static w .js, /templates w mako made of html 2. Under /lib/galaxy/datatypes/display_applications I tried the same approach with no success. Is this where code goes for external displays that use get and post reside? 3. which if any of 1or2 above use the Tools/Visualization directory. Is that the convention for the return of a Get or Post external displays? 4. I create my own datatype in data_config_type.xml with a <datatype> tag 5. in data_config_type.xml you only use the <display> record which -- internal or external displays? 6. I added a class in the binary.py file with the name of my new datatype. 7. I added a tool.mydatatype section and file with my xml to input and then output my datatype by specifying format=mydatatype not format=binary.mydatatype Plus I added this into Tools_conf.xml Are there other xml specific items I need in my tool xml for an internal visualization 8. By clicking the visualization icon. what does it call (hopefully 1or2 above) so i can add specific code to display my datatype internal "visualization?" (preferred) in galaxy? ----- Original Message -----From: Eric Rasche <esr@tamu.edu>To: rbrown1422@comcast.netCc: galaxy-dev <galaxy-dev@lists.bx.psu.edu>Sent: Tue, 08 Dec 2015 02:34:47 -0000 (UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? Hi Bob, Yes, galaxy supports output formats of HTML+JS, though the tool producing them may need to be whitelisted (grep for 'whitelist' in your galaxy.ini) for display to work properly. If you wish to write your own visualization plugin for a specific datatype, there are assorted pieces of documentation on the wiki that might help you (though I usually just look at one of the other existing visualization plugins and copy+paste). Cheers, Eric 2015-12-07 20:08 GMT-06:00 <rbrown1422@comcast.net>: Good evening Galaxy team, Is it possible to run JavaScript or HTML as a tool or a display module? I see support for URLs, e.g. UCSC Genome Browser, but I would like to process a specific datatype to allow it to be viewed in Galaxy -- but the application is a combination of HTML and JavaScript. Does Galaxy support this? Thank you, Bob ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Eric Rasche Programmer II Center for Phage Technology Rm 312A, BioBio Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu

Thank you Carl for the excellent write-up. It could serve as a visualization plugin tutorial. However, I am still not able to get the visualization icon to appear, see the attachment 2. I feel if I get the icon to show it will all come together. A couple questions followed by file dumps and startup info 1. my /api/datatypes/mapping dump it shows registry instead of binary near the top "galaxy.datatypes.genetics.RexpBase": {"galaxy.datatypes.images.Html": true, "galaxy.datatypes.data.Data": true, "galaxy.datatypes.data.Text": true, "galaxy.datatypes.genetics.RexpBase": true}, "galaxy.datatypes.registry.MDA_zip": {"galaxy.datatypes.registry.Binary": true, "galaxy.datatypes.data.Data": true, "galaxy.datatypes.binary.Binary": true, "galaxy.datatypes.registry.MDA_zip": true}, near the bottom in lower case "galaxy.datatypes.registry.Text", "mda_zip": "galaxy.datatypes.registry.MDA_zip", "chips": "galaxy.datatypes.registry.Text", "gg": "galaxy.datatypes.genetics.GenomeGraphs", "syco": "galaxy.datatypes.registry.Text", 2. I also added a snapshot of the visualization/tool/ sub-directories. It shows my extension because I have a format=type in the tool.xml output file defition but shows file type 'binary' at the bottom of the file history. Is this correct, or should it be my extension type there? Comment -- It appears the only link between the file extension and the visualization plugin is via the test line in the /conf/toolxml file. Is that correct? My xml in my visualize directory /tool/config/tool.xml (formatting is messed up here) attachment 1 <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE visualization SYSTEM "../../visualization.dtd"> <visualization name="MDAheatmap"> <data_sources> <data_source> <model_class>HistoryDatasetAssociation</model_class> <test type="isinstance" test_attr="datatype" result_type="datatype">binary.MDA_zip</test> <to_param param_attr="id">dataset_id</to_param> </data_source> </data_sources> <params> <param type="dataset" var_name_in_template="hda" required="true">dataset_id</param> </params> <entry_point entry_point_type="mako">MDAheatmap.mako</entry_point> </visualization> My templates/tool.mako file (hello world) <!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN"> <HTML> <HEAD> <TITLE> A Small Hello </TITLE> </HEAD> <BODY> <H1>Hi</H1> <P>This is very minimal "hello world" HTML document.</P> </BODY> </HTML> startup info seems successful galaxy.web.base.pluginframework INFO 2015-12-14 13:20:09,631 VisualizationsRegistry, loaded plugin: MDAheatmap x.x.10.42 - - [14/Dec/2015:14:11:50 -0400] "PUT /api/histories/a799d38679e985db/contents/datasets/fb85969571388350 HTTP/1.1" 200 - "http://xx.xx.10.4:8082/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" no skipping error and no other apparent errors in startup here is my Datatypes.conf.xml 2 additions <datatype extension="MDA_zip" type="galaxy.datatypes.binary:MDA_zip" mimetype="application/octet-stream" display_in_upload="true" subclass="False"> and sniffer <sniffer type="galaxy.datatypes.binary:MDA_zip"/> my binary.py addition class MDA_zip( Binary ): """Class describing an MDA Heatmap binary zip file""" file_ext = "MDA_zip" def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "Binary MDA_zip sequence file" dataset.blurb = nice_size( dataset.get_size() ) else: dataset.peek = 'file MDA_zip does not exist' dataset.blurb = 'file MDA_zip purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "Binary MDA_zip heatmap file (%s)" % ( nice_size( dataset.get_size() ) ) Binary.register_sniffable_binary_format("MDA_zip","MDA_zip",MDA_zip) my dataset.py addition class MDA_zipDataProvider( base.DataProvider ): def __init__( self, dataset, **kwards ): self.dataset = json.loads( "".join( dataset.readlines() )) """ def __init__( self, dataset, **kwards ): raise NotImplementedError() super( MDA_zipDataProvider, self ).__init__( dataset, **kwargs ) My tool in tools dir <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE visualization SYSTEM "../../visualization.dtd"> <tool id="MDAheatmap" name="MDA HeatMap" version="1.3.0"> <description>Display HeatMap generated by MDAnderson HeatMap tools</description> <command interpreter="python">MDAheatmap.py $input $out_file </command> <inputs> <param format="text" name="input" type="data" label="doesnt_matter" help="input any file and a MDA zipped file pulled off disk"/> </inputs> <outputs> <data format="MDA_zip" name="out_file"/> </outputs> </tool> ----- Original Message -----From: Carl Eberhard <carlfeberhard@gmail.com>To: rbrown1422@comcast.netCc: Eric Rasche <esr@tamu.edu>, galaxy-dev <galaxy-dev@lists.bx.psu.edu>Sent: Mon, 14 Dec 2015 16:40:34 -0000 (UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? the complete message ignore previous Hi, Bob There are a number of ways to visualize data in Galaxy, so let's settle on some terminology first for the three major ways: * display applications: these are definitions of external websites that can fetch galaxy datasets and display them in their visualization applications. Examples are the UCSC genome browser, GBrowse, and IGV. * tools: although the tool framework generally produces 'raw' data like tabular or binary files, it can also produce the html (and js even) that can display data for visualization. An example would be 'Boxplot of quality statistics'. * visualization plugins: these are python/mako/js files that are meant to be more interactive visualizations in order to explore data It sounds to me like you'd like to use the visualization plugin system, so: * since they're for external code or websites, let's take display applications off the table entirely. * since they can be a viable alternative to the plugin system, so I'd keep tools as another option but keep our focus on the plugin system It also sounds like you have two elements you'd like to incorporate: * a datatype * a plugin to display datasets of that datatype The datatype I'm more familiar with the plugin part of this process, but for the datatype it sounds like you want to subclass the binary datatype and add it to your galaxy installation. I'd start here (I imagine you've already seen it, tho): * https://wiki.galaxyproject.org/Admin/Datatypes * https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes Note: to see if your datatype was loaded successfully, with your server running - you can go to: /api/datatypes/mapping. You should be able to see your datatype listed at the first level of the json map, generally beginning with 'galaxy.datatypes.' and the python module name you added it to (like a python import namespace): e.g. galaxy.datatypes.binary.H5 or galaxy.datatypes.binary.mydatatype To simplify: 1. Add a definition to your datatypes_conf file 2. Add a sniffer for your datatype 3. Add a subclass/class for your datatype The plugin For the plugin, I also imagine you've seen this, but it's a (hopefully) good place to start: https://wiki.galaxyproject.org/VisualizationsRegistry The filesystem layout Here's a simplified process for creating an outline for a visualization plugin project: 1. in the filesystem, start at <your galaxy>/config/plugins/visualizations 2. think of an id for your plugin. This can be any (filesystem permissible) file name and is only used as an id by the registry - the users never see it. It should be unique from any other plugins in that directory. (e.g. myplugin) 3. create a main directory using your id. (e.g. <your galaxy>/config/plugins/visualizations/myplugin) 4. inside that directory, create three more directories: config, templates, static. This is where the plugin configuration, the mako templates, and any (optional) javascript or static files are kept respectively. 5. the configuration file should use the same id you used above for the directory: e.g. config/myplugin.xml. Most people copy and rename a simple config file like the one in config/plugins/visualizations/scatterplot/config/scatterplot.xml. We'll change the datatype it applies to later, but there's more at: https://wiki.galaxyproject.org/VisualizationsRegistry#The_visualization_conf... and https://wiki.galaxyproject.org/VisualizationsRegistry/Configuration 6. a mako template file should go into the templates directory: e.g. templates/myplugin.mako. This template file is the entry point for your visualization and is loaded first. You don't have to do any major coding here and can instead just launch javascript to render the visualization. https://wiki.galaxyproject.org/VisualizationsRegistry#Creating_the_code_and_... At this point, your config/plugins/visualizations/myplugin directory should look like one of the two trees displayed here: https://wiki.galaxyproject.org/VisualizationsRegistry#Configuring_your_visua... The configuration Now, we'll configure the plugin by editing the config/plugins/visualizations/myplugin/config/myplugin.xml file. Let's assume you've copied the scatterplot config file (https://github.com/galaxyproject/galaxy/blob/dev/config/plugins/visualizatio...): 1. change the name displayed to what you'd like users to see in the dataset visualizations dropdown menu: l<visualization name="My Visualization Plugin"> 2. change the configuration for your visualization to test for your datatype. For example, if you have mydatatype, change: <data_sources> <data_source> <model_class>HistoryDatasetAssociation</model_class> <test type="isinstance" test_attr="datatype" result_type="datatype">binary.MyDatatype</test> <to_param param_attr="id">dataset_id</to_param> </data_source> </data_sources> The above is basically saying, if an object is a) a HistoryDatasetAssociation (a dataset in a history) and b) it's a subclass or instance of binary.MyDatatype, then put a link: * in the dataset visualization dropdown menu * that will start my visualization * pass the encoded id of the dataset in the link using the parameter 'dataset_id' Note: the 'binary.MyDatatype' is essentially 'module.class' and is also the last half of the id given in the 'api/datatypes/mapping' mentioned previously. 3. change the template used as the entry point to reflect the name you gave it: <entry_point entry_point_type="mako">myplugin.mako</entry_point> Testing and troubleshooting Finally, you should be at a spot that: * your visualization will appear in the visualization dropdown menu of datasets that are of the mydatatype class * if that menu link is clicked, galaxy will go to visualization/show/<your plugin id>. E.g. /visualization/show/myplugin?dataset_id=0b1d0715f8366ea8 * whatever code is in myplugin.mako will be executed (you should be able to use print in python or alert in javascript to test) Troubleshooting * If your server is running when you've made changes to your config file, you'll need to restart it (changes to mako and static files do not need a restart) * Make sure your id is used in the proper places above and that they all match * If galaxy is trying to load or loading your visualization plugin, you should see an entry like this in your server logs: galaxy.web.base.pluginframework INFO 2015-12-14 08:46:44,788 VisualizationsRegistry, loaded plugin: myplugin * errors will also display there * it can help to change 'debug = True' in your config/galaxy.ini file (only do this on a separate development server - not you public shared instances) I know that's a lot of info, so let me know if I can add or clarify on any of it. Carl On Sat, Dec 12, 2015 at 9:57 AM, <rbrown1422@comcast.net> wrote: Thank you Eric, But I want to be clear after spending 5 days, reading many Galaxy wiki links, looking at samples, and with help from others trying to get this to work. I have written many tools but never tried to display a unique datatype before. Below is a series of questions and steps I followed trying the suggestions from early posts. My goals are: -- To get the visualization icon to show in the bottom left of my history output file? -- click the visualization icon to display my data (that I will process to display) inside Galaxy. The steps below are in order of urgency:) Thank you for you patience. 1. /Config/plugin/visualization is where code goes to have a Galaxy internal module display a user defined data type? -- mine is binary Without getting to in the weeds under this directory I have a mydatatype dir with subdirectories /config w xml, /static w .js, /templates w mako made of html 2. Under /lib/galaxy/datatypes/display_applications I tried the same approach with no success. Is this where code goes for external displays that use get and post reside? 3. which if any of 1or2 above use the Tools/Visualization directory. Is that the convention for the return of a Get or Post external displays? 4. I create my own datatype in data_config_type.xml with a <datatype> tag 5. in data_config_type.xml you only use the <display> record which -- internal or external displays? 6. I added a class in the binary.py file with the name of my new datatype. 7. I added a tool.mydatatype section and file with my xml to input and then output my datatype by specifying format=mydatatype not format=binary.mydatatype Plus I added this into Tools_conf.xml Are there other xml specific items I need in my tool xml for an internal visualization 8. By clicking the visualization icon. what does it call (hopefully 1or2 above) so i can add specific code to display my datatype internal "visualization?" (preferred) in galaxy? ----- Original Message -----From: Eric Rasche <esr@tamu.edu>To: rbrown1422@comcast.netCc: galaxy-dev <galaxy-dev@lists.bx.psu.edu>Sent: Tue, 08 Dec 2015 02:34:47 -0000 (UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? Hi Bob, Yes, galaxy supports output formats of HTML+JS, though the tool producing them may need to be whitelisted (grep for 'whitelist' in your galaxy.ini) for display to work properly. If you wish to write your own visualization plugin for a specific datatype, there are assorted pieces of documentation on the wiki that might help you (though I usually just look at one of the other existing visualization plugins and copy+paste). Cheers, Eric 2015-12-07 20:08 GMT-06:00 <rbrown1422@comcast.net>: Good evening Galaxy team, Is it possible to run JavaScript or HTML as a tool or a display module? I see support for URLs, e.g. UCSC Genome Browser, but I would like to process a specific datatype to allow it to be viewed in Galaxy -- but the application is a combination of HTML and JavaScript. Does Galaxy support this? Thank you, Bob ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Eric Rasche Programmer II Center for Phage Technology Rm 312A, BioBio Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Give this a try: <test type="isinstance" test_attr="datatype" result_type="datatype"> *registry.MDA_zip*</test> And if that doesn't work: <test type="isinstance" test_attr="datatype" result_type="datatype"> *MDA_zip*</test> Finally, try this (which won't return true for subclasses of MDA_zip): <test test_attr="extension">*MDA_zip*</test> (Sorry - initially sent this to Bob only) On Mon, Dec 14, 2015 at 4:14 PM, <rbrown1422@comcast.net> wrote:

Thank you Carl for the excellent write-up. It could serve as a visualization plugin tutorial.

However, I am still not able to get the visualization icon to appear, see the attachment 2.

I feel if I get the icon to show it will all come together.

A couple questions followed by file dumps and startup info

*1. my /api/datatypes/mapping dump it shows registry instead of binary*

*near the top*

"galaxy.datatypes.genetics.RexpBase": {"galaxy.datatypes.images.Html": true, "galaxy.datatypes.data.Data": true, "galaxy.datatypes.data.Text": true, "galaxy.datatypes.genetics.RexpBase": true}, "galaxy.datatypes. *registry*.MDA_zip": {"galaxy.datatypes.*registry*.Binary": true, "galaxy.datatypes.data.Data": true, "galaxy.datatypes.binary.Binary": true, "galaxy.datatypes.*registry*.MDA_zip": true},

*near the bottom in lower case*

"galaxy.datatypes.registry.Text", "mda_zip": "galaxy.datatypes.registry.MDA_zip", "chips": "galaxy.datatypes.registry.Text", "gg": "galaxy.datatypes.genetics.GenomeGraphs", "syco": "galaxy.datatypes.registry.Text",

2. I also added a snapshot of the visualization/tool/ sub-directories. It shows my extension because I have a format=type in the tool.xml output file defition but shows file type 'binary' at the bottom of the file history. Is this correct, or should it be my extension type there?

Comment -- It appears the only link between the file extension and the visualization plugin is via the test line in the /conf/toolxml file. Is that correct?

*My xml in my visualize directory /tool/config/tool.xml (formatting is messed up here) attachment 1*

<?xml version="1.0" encoding="UTF-8" ?>

<!DOCTYPE visualization SYSTEM "../../visualization.dtd">

<visualization name="MDAheatmap">

<data_sources>

<data_source>

<model_class>HistoryDatasetAssociation</model_class>

<test type="isinstance" test_attr="datatype" result_type="datatype">binary.MDA_zip</test>

<to_param param_attr="id">dataset_id</to_param>

</data_source>

</data_sources>

<params>

<param type="dataset" var_name_in_template="hda" required="true">dataset_id</param>

</params>

<entry_point entry_point_type="mako">MDAheatmap.mako</entry_point>

</visualization>

*My templates/tool.mako file (hello world)*

<!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">

<HTML>

<HEAD>

<TITLE>

A Small Hello

</TITLE>

</HEAD>

<BODY>

<H1>Hi</H1>

<P>This is very minimal "hello world" HTML document.</P>

</BODY>

</HTML>

*startup info seems successful*

galaxy.web.base.pluginframework INFO 2015-12-14 13:20:09,631 VisualizationsRegistry, loaded plugin: MDAheatmap

x.x.10.42 - - [14/Dec/2015:14:11:50 -0400] "PUT /api/histories/a799d38679e985db/contents/datasets/fb85969571388350 HTTP/1.1" 200 - "http://xx.xx.10.4:8082/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"

*no skipping error and no other apparent errors in startup*

*here is my Datatypes.conf.xml 2 additions*

<datatype extension="MDA_zip" type="galaxy.datatypes.binary:MDA_zip" mimetype="application/octet-stream" display_in_upload="true" subclass="False">

*and sniffer*

<sniffer type="galaxy.datatypes.binary:MDA_zip"/>

*my binary.py addition*

class MDA_zip( Binary ):

"""Class describing an MDA Heatmap binary zip file"""

file_ext = "MDA_zip"

def set_peek( self, dataset, is_multi_byte=False ):

if not dataset.dataset.purged:

dataset.peek = "Binary MDA_zip sequence file"

dataset.blurb = nice_size( dataset.get_size() )

else:

dataset.peek = 'file MDA_zip does not exist'

dataset.blurb = 'file MDA_zip purged from disk'

def display_peek( self, dataset ):

try:

return dataset.peek

except:

return "Binary MDA_zip heatmap file (%s)" % ( nice_size( dataset.get_size() ) )

Binary.register_sniffable_binary_format("MDA_zip","MDA_zip",MDA_zip)

*my dataset.py addition*

class MDA_zipDataProvider( base.DataProvider ):

def __init__( self, dataset, **kwards ):

self.dataset = json.loads( "".join( dataset.readlines() ))

"""

def __init__( self, dataset, **kwards ):

raise NotImplementedError()

super( MDA_zipDataProvider, self ).__init__( dataset, **kwargs )

*My tool in tools dir*

<?xml version="1.0" encoding="UTF-8" ?>

<!DOCTYPE visualization SYSTEM "../../visualization.dtd">

<tool id="MDAheatmap" name="MDA HeatMap" version="1.3.0">

<description>Display HeatMap generated by MDAnderson HeatMap tools</description>

<command interpreter="python">MDAheatmap.py $input $out_file </command>

<inputs>

<param format="text" name="input" type="data" label="doesnt_matter" help="input any file and a MDA zipped file pulled off disk"/>

</inputs>

<outputs>

<data format="MDA_zip" name="out_file"/>

</outputs>

</tool>

----- Original Message ----- From: Carl Eberhard <carlfeberhard@gmail.com> To: rbrown1422@comcast.net Cc: Eric Rasche <esr@tamu.edu>, galaxy-dev <galaxy-dev@lists.bx.psu.edu> Sent: Mon, 14 Dec 2015 16:40:34 -0000 (UTC) Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? the complete message ignore previous Hi, Bob There are a number of ways to visualize data in Galaxy, so let's settle on some terminology first for the three major ways: * *display applications*: these are definitions of external websites that can fetch galaxy datasets and display them in their visualization applications. Examples are the UCSC genome browser, GBrowse, and IGV. * *tools*: although the tool framework generally produces 'raw' data like tabular or binary files, it can also produce the html (and js even) that can display data for visualization. An example would be 'Boxplot of quality statistics'. * *visualization plugins*: these are python/mako/js files that are meant to be more interactive visualizations in order to explore data It sounds to me like you'd like to use the visualization plugin system, so: * since they're for external code or websites, *let's take display applications off the table entirely*. * since they can be a viable alternative to the plugin system, so *I'd keep tools as another option but keep our focus on the plugin system* It also sounds like you have two elements you'd like to incorporate: * a *datatype* * a *plugin* to display datasets of that datatype *The datatype* I'm more familiar with the plugin part of this process, but for the datatype it sounds like you want to subclass the binary datatype and add it to your galaxy installation. I'd start here (I imagine you've already seen it, tho): * https://wiki.galaxyproject.org/Admin/Datatypes * https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes Note: to see if your datatype was loaded successfully, with your server running - you can go to: /api/datatypes/mapping. You should be able to see your datatype listed at the first level of the json map, generally beginning with 'galaxy.datatypes.' and the python module name you added it to (like a python import namespace): e.g. galaxy.datatypes.binary.H5 or galaxy.datatypes.binary.mydatatype To simplify: 1. Add a definition to your datatypes_conf file 2. Add a sniffer for your datatype 3. Add a subclass/class for your datatype *The plugin* For the plugin, I also imagine you've seen this, but it's a (hopefully) good place to start: https://wiki.galaxyproject.org/VisualizationsRegistry *The filesystem layout* Here's a simplified process for creating an outline for a visualization plugin project: 1. in the filesystem, start at *<your galaxy>/config/plugins/visualizations* 2. *think of an id for your plugin*. This can be any (filesystem permissible) file name and is only used as an id by the registry - the users never see it. It should be unique from any other plugins in that directory. (e.g. myplugin) 3. *create a main directory using your id*. (e.g. <your galaxy>/config/plugins/visualizations/myplugin) 4. inside that directory, *create three more directories: config, templates, static*. This is where the plugin configuration, the mako templates, and any (optional) javascript or static files are kept respectively. 5. *the configuration file should use the same id* you used above for the directory: e.g. config/myplugin.xml. Most people copy and rename a simple config file like the one in config/plugins/visualizations/scatterplot/config/scatterplot.xml. We'll change the datatype it applies to later, but there's more at: https://wiki.galaxyproject.org/VisualizationsRegistry#The_visualization_conf... and https://wiki.galaxyproject.org/VisualizationsRegistry/Configuration 6. *a mako template file should go into the templates directory*: e.g. templates/myplugin.mako. This template file is the entry point for your visualization and is loaded first. You don't have to do any major coding here and can instead just launch javascript to render the visualization. https://wiki.galaxyproject.org/VisualizationsRegistry#Creating_the_code_and_... At this point, your config/plugins/visualizations/myplugin directory should look like one of the two trees displayed here:

https://wiki.galaxyproject.org/VisualizationsRegistry#Configuring_your_visua... *The configuration* Now, we'll configure the plugin by editing the config/plugins/visualizations/myplugin/config/myplugin.xml file. Let's assume you've copied the scatterplot config file ( https://github.com/galaxyproject/galaxy/blob/dev/config/plugins/visualizatio... ): 1. change the name displayed to what you'd like users to see in the dataset visualizations dropdown menu: l<visualization name="*My Visualization Plugin*"> 2. change the configuration for your visualization to test for your datatype. For example, if you have mydatatype, change: <data_sources> <data_source> <model_class>HistoryDatasetAssociation</model_class> <test type="isinstance" test_attr="datatype" result_type="datatype">*binary.MyDatatype*</test> <to_param param_attr="id">dataset_id</to_param> </data_source> </data_sources>

The above is basically saying, if an object is a) a HistoryDatasetAssociation (a dataset in a history) and b) it's a subclass or instance of binary.MyDatatype, then put a link: * in the dataset visualization dropdown menu * that will start my visualization * pass the encoded id of the dataset in the link using the parameter 'dataset_id' Note: the 'binary.MyDatatype' is essentially 'module.class' and is also the last half of the id given in the 'api/datatypes/mapping' mentioned previously. 3. change the template used as the entry point to reflect the name you gave it: <entry_point entry_point_type="mako">*myplugin.mako*</entry_point> *Testing and troubleshooting* Finally, you should be at a spot that: * your visualization will appear in the visualization dropdown menu of datasets that are of the mydatatype class * if that menu link is clicked, galaxy will go to visualization/show/<your plugin id>. E.g. /visualization/show/myplugin?dataset_id=0b1d0715f8366ea8 * whatever code is in myplugin.mako will be executed (you should be able to use print in python or alert in javascript to test) *Troubleshooting* * If your server is running when you've made changes to your config file, you'll need to restart it (changes to mako and static files do not need a restart) * Make sure your id is used in the proper places above and that they all match * If galaxy is trying to load or loading your visualization plugin, you should see an entry like this in your server logs: galaxy.web.base.pluginframework INFO 2015-12-14 08:46:44,788 VisualizationsRegistry, loaded plugin: myplugin * errors will also display there * it can help to change 'debug = True' in your config/galaxy.ini file (only do this on a separate development server - not you public shared instances) I know that's a lot of info, so let me know if I can add or clarify on any of it. Carl

On Sat, Dec 12, 2015 at 9:57 AM, <rbrown1422@comcast.net> wrote:

...

Thank you Eric, But I want to be clear after spending 5 days, reading many Galaxy wiki links, looking at samples, and with help from others trying to get this to work. I have written many tools but never tried to display a unique datatype before. Below is a series of questions and steps I followed trying the suggestions from early posts. My goals are: -- To get the visualization icon to show in the bottom left of my history output file? -- click the visualization icon to display my data (that I will process to display) inside Galaxy. The steps below are in order of urgency:) Thank you for you patience. 1. /Config/plugin/visualization is where code goes to have a Galaxy *internal* module display a user defined data type? -- mine is binary Without getting to in the weeds under this directory I have a mydatatype dir with subdirectories /config w xml, /static w .js, /templates w mako made of html 2. Under /lib/galaxy/datatypes/display_applications I tried the same approach with no success. Is this where code goes for external displays that use get and post reside? 3. which if any of 1or2 above use the Tools/Visualization directory. Is that the convention for the return of a Get or Post external displays? 4. I create my own datatype in data_config_type.xml with a <datatype> tag 5. in data_config_type.xml you only use the <display> record which -- internal or external displays? 6. I added a class in the binary.py file with the name of my new datatype. 7. I added a tool.mydatatype section and file with my xml to input and then output my datatype by specifying format=mydatatype not format=binary.mydatatype Plus I added this into Tools_conf.xml Are there other xml specific items I need in my tool xml for an internal visualization 8. By clicking the visualization icon. what does it call (hopefully 1or2 above) so i can add specific code to display my datatype internal "visualization?" (preferred) in galaxy? ----- Original Message ----- From: Eric Rasche <esr@tamu.edu> To: rbrown1422@comcast.net Cc: galaxy-dev <galaxy-dev@lists.bx.psu.edu> Sent: Tue, 08 Dec 2015 02:34:47 -0000 (UTC) Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? Hi Bob,

Yes, galaxy supports output formats of HTML+JS, though the tool producing them may need to be whitelisted (grep for 'whitelist' in your galaxy.ini) for display to work properly.

If you wish to write your own visualization plugin for a specific datatype, there are assorted pieces of documentation on the wiki <https://wiki.galaxyproject.org/?action=fullsearch&context=180&value=visualization&titlesearch=Titles> that might help you (though I usually just look at one of the other existing visualization plugins and copy+paste).

Cheers,

Eric

2015-12-07 20:08 GMT-06:00 <rbrown1422@comcast.net>:

...

Good evening Galaxy team, Is it possible to run JavaScript or HTML as a tool or a display module? I see support for URLs, e.g. UCSC Genome Browser, but I would like to process a specific datatype to allow it to be viewed in Galaxy -- but the application is a combination of HTML and JavaScript. Does Galaxy support this? Thank you, Bob

___________________________________________________________

Please keep all replies on the list by using "reply all"

in your mail client. To manage your subscriptions to this

and other Galaxy lists, please use the interface at:

https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:

http://galaxyproject.org/search/mailinglists/

-- Eric Rasche Programmer II Center for Phage Technology Rm 312A, BioBio Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu

___________________________________________________________

Please keep all replies on the list by using "reply all"

in your mail client. To manage your subscriptions to this

and other Galaxy lists, please use the interface at:

https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:

http://galaxyproject.org/search/mailinglists/

To afternoon team, I have a plugin that is visualization a static output. I want to retrieve the dataset id for the file I want to visualize when the lower left visualize icon is clicked. I see 'hda' is passed around in other plugins including. hda.name. Is hda an object? What attributes does it have. Or is there another way to retrieve the dataset id. I seem to see an encoded Id when the mako is called but cannot decipher it. Thanks Bob 1. I added a snapshot of the visualization/tool/ sub-directories. It shows my extension because I have a format=type in the tool.xml output file defition but shows file type 'binary' at the bottom of the file history. Is this correct, or should it be my extension type there? Comment -- It appears the only link between the file extension and the visualization plugin is via the test line in the /conf/toolxml file. Is that correct? My xml in my visualize directory /tool/config/tool.xml (formatting is messed up here) attachment 1 <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE visualization SYSTEM "../../visualization.dtd"> <visualization name="MDAheatmap"> <data_sources> <data_source> <model_class>HistoryDatasetAssociation</model_class> <test type="isinstance" test_attr="extension" result_type="datatype">registry.MDA_zip</test> <to_param param_attr="id">dataset_id</to_param> </data_source> </data_sources> <params> <param type="dataset" var_name_in_template="hda" required="true">dataset_id</param> </params> <entry_point entry_point_type="mako">MDAheatmap.mako</entry_point> </visualization> My templates/tool.mako file (hello world) <!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN"> <HTML> <HEAD> <TITLE> A Small Hello </TITLE> </HEAD> <BODY> <H1>Hi</H1> <P>This is very minimal "hello world" HTML document.</P> </BODY> </HTML> startup info seems successful galaxy.web.base.pluginframework INFO 2015-12-14 13:20:09,631 VisualizationsRegistry, loaded plugin: MDAheatmap x.x.10.42 - - [14/Dec/2015:14:11:50 -0400] "PUT /api/histories/a799d38679e985db/contents/datasets/fb85969571388350 HTTP/1.1" 200 - "http://xx.xx.10.4:8082/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" no skipping error and no other apparent errors in startup here is my Datatypes.conf.xml 2 additions <datatype extension="MDA_zip" type="galaxy.datatypes.binary:MDA_zip" mimetype="application/octet-stream" display_in_upload="true" subclass="False"> and sniffer <sniffer type="galaxy.datatypes.binary:MDA_zip"/> my binary.py addition class MDA_zip( Binary ): """Class describing an MDA Heatmap binary zip file""" file_ext = "MDA_zip" def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "Binary MDA_zip sequence file" dataset.blurb = nice_size( dataset.get_size() ) else: dataset.peek = 'file MDA_zip does not exist' dataset.blurb = 'file MDA_zip purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "Binary MDA_zip heatmap file (%s)" % ( nice_size( dataset.get_size() ) ) Binary.register_sniffable_binary_format("MDA_zip","MDA_zip",MDA_zip) my dataset.py addition class MDA_zipDataProvider( base.DataProvider ): def __init__( self, dataset, **kwards ): self.dataset = json.loads( "".join( dataset.readlines() )) """ def __init__( self, dataset, **kwards ): raise NotImplementedError() super( MDA_zipDataProvider, self ).__init__( dataset, **kwargs ) My tool in tools dir <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE visualization SYSTEM "../../visualization.dtd"> <tool id="MDAheatmap" name="MDA HeatMap" version="1.3.0"> <description>Display HeatMap generated by MDAnderson HeatMap tools</description> <command interpreter="python">MDAheatmap.py $input $out_file </command> <inputs> <param format="text" name="input" type="data" label="doesnt_matter" help="input any file and a MDA zipped file pulled off disk"/> </inputs> <outputs> <data format="MDA_zip" name="out_file"/> </outputs> </tool> ----- Original Message -----From: Carl Eberhard <carlfeberhard@gmail.com>To: rbrown1422@comcast.netCc: Eric Rasche <esr@tamu.edu>, galaxy-dev <galaxy-dev@lists.bx.psu.edu>Sent: Mon, 14 Dec 2015 16:40:34 -0000 (UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? the complete message ignore previous Hi, Bob There are a number of ways to visualize data in Galaxy, so let's settle on some terminology first for the three major ways: * display applications: these are definitions of external websites that can fetch galaxy datasets and display them in their visualization applications. Examples are the UCSC genome browser, GBrowse, and IGV. * tools: although the tool framework generally produces 'raw' data like tabular or binary files, it can also produce the html (and js even) that can display data for visualization. An example would be 'Boxplot of quality statistics'. * visualization plugins: these are python/mako/js files that are meant to be more interactive visualizations in order to explore data It sounds to me like you'd like to use the visualization plugin system, so: * since they're for external code or websites, let's take display applications off the table entirely. * since they can be a viable alternative to the plugin system, so I'd keep tools as another option but keep our focus on the plugin system It also sounds like you have two elements you'd like to incorporate: * a datatype * a plugin to display datasets of that datatype The datatype I'm more familiar with the plugin part of this process, but for the datatype it sounds like you want to subclass the binary datatype and add it to your galaxy installation. I'd start here (I imagine you've already seen it, tho): * https://wiki.galaxyproject.org/Admin/Datatypes * https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes Note: to see if your datatype was loaded successfully, with your server running - you can go to: /api/datatypes/mapping. You should be able to see your datatype listed at the first level of the json map, generally beginning with 'galaxy.datatypes.' and the python module name you added it to (like a python import namespace): e.g. galaxy.datatypes.binary.H5 or galaxy.datatypes.binary.mydatatype To simplify: 1. Add a definition to your datatypes_conf file 2. Add a sniffer for your datatype 3. Add a subclass/class for your datatype The plugin For the plugin, I also imagine you've seen this, but it's a (hopefully) good place to start: https://wiki.galaxyproject.org/VisualizationsRegistry The filesystem layout Here's a simplified process for creating an outline for a visualization plugin project: 1. in the filesystem, start at <your galaxy>/config/plugins/visualizations 2. think of an id for your plugin. This can be any (filesystem permissible) file name and is only used as an id by the registry - the users never see it. It should be unique from any other plugins in that directory. (e.g. myplugin) 3. create a main directory using your id. (e.g. <your galaxy>/config/plugins/visualizations/myplugin) 4. inside that directory, create three more directories: config, templates, static. This is where the plugin configuration, the mako templates, and any (optional) javascript or static files are kept respectively. 5. the configuration file should use the same id you used above for the directory: e.g. config/myplugin.xml. Most people copy and rename a simple config file like the one in config/plugins/visualizations/scatterplot/config/scatterplot.xml. We'll change the datatype it applies to later, but there's more at: https://wiki.galaxyproject.org/VisualizationsRegistry#The_visualization_conf... and https://wiki.galaxyproject.org/VisualizationsRegistry/Configuration 6. a mako template file should go into the templates directory: e.g. templates/myplugin.mako. This template file is the entry point for your visualization and is loaded first. You don't have to do any major coding here and can instead just launch javascript to render the visualization. https://wiki.galaxyproject.org/VisualizationsRegistry#Creating_the_code_and_... At this point, your config/plugins/visualizations/myplugin directory should look like one of the two trees displayed here: https://wiki.galaxyproject.org/VisualizationsRegistry#Configuring_your_visua... The configuration Now, we'll configure the plugin by editing the config/plugins/visualizations/myplugin/config/myplugin.xml file. Let's assume you've copied the scatterplot config file (https://github.com/galaxyproject/galaxy/blob/dev/config/plugins/visualizatio...): 1. change the name displayed to what you'd like users to see in the dataset visualizations dropdown menu: l<visualization name="My Visualization Plugin"> 2. change the configuration for your visualization to test for your datatype. For example, if you have mydatatype, change: <data_sources> <data_source> <model_class>HistoryDatasetAssociation</model_class> <test type="isinstance" test_attr="datatype" result_type="datatype">binary.MyDatatype</test> <to_param param_attr="id">dataset_id</to_param> </data_source> </data_sources> The above is basically saying, if an object is a) a HistoryDatasetAssociation (a dataset in a history) and b) it's a subclass or instance of binary.MyDatatype, then put a link: * in the dataset visualization dropdown menu * that will start my visualization * pass the encoded id of the dataset in the link using the parameter 'dataset_id' Note: the 'binary.MyDatatype' is essentially 'module.class' and is also the last half of the id given in the 'api/datatypes/mapping' mentioned previously. 3. change the template used as the entry point to reflect the name you gave it: <entry_point entry_point_type="mako">myplugin.mako</entry_point> Testing and troubleshooting Finally, you should be at a spot that: * your visualization will appear in the visualization dropdown menu of datasets that are of the mydatatype class * if that menu link is clicked, galaxy will go to visualization/show/<your plugin id>. E.g. /visualization/show/myplugin?dataset_id=0b1d0715f8366ea8 * whatever code is in myplugin.mako will be executed (you should be able to use print in python or alert in javascript to test) Troubleshooting * If your server is running when you've made changes to your config file, you'll need to restart it (changes to mako and static files do not need a restart) * Make sure your id is used in the proper places above and that they all match * If galaxy is trying to load or loading your visualization plugin, you should see an entry like this in your server logs: galaxy.web.base.pluginframework INFO 2015-12-14 08:46:44,788 VisualizationsRegistry, loaded plugin: myplugin * errors will also display there * it can help to change 'debug = True' in your config/galaxy.ini file (only do this on a separate development server - not you public shared instances) I know that's a lot of info, so let me know if I can add or clarify on any of it. Carl On Sat, Dec 12, 2015 at 9:57 AM, <rbrown1422@comcast.net> wrote: Thank you Eric, But I want to be clear after spending 5 days, reading many Galaxy wiki links, looking at samples, and with help from others trying to get this to work. I have written many tools but never tried to display a unique datatype before. Below is a series of questions and steps I followed trying the suggestions from early posts. My goals are: -- To get the visualization icon to show in the bottom left of my history output file? -- click the visualization icon to display my data (that I will process to display) inside Galaxy. The steps below are in order of urgency:) Thank you for you patience. 1. /Config/plugin/visualization is where code goes to have a Galaxy internal module display a user defined data type? -- mine is binary Without getting to in the weeds under this directory I have a mydatatype dir with subdirectories /config w xml, /static w .js, /templates w mako made of html 2. Under /lib/galaxy/datatypes/display_applications I tried the same approach with no success. Is this where code goes for external displays that use get and post reside? 3. which if any of 1or2 above use the Tools/Visualization directory. Is that the convention for the return of a Get or Post external displays? 4. I create my own datatype in data_config_type.xml with a <datatype> tag 5. in data_config_type.xml you only use the <display> record which -- internal or external displays? 6. I added a class in the binary.py file with the name of my new datatype. 7. I added a tool.mydatatype section and file with my xml to input and then output my datatype by specifying format=mydatatype not format=binary.mydatatype Plus I added this into Tools_conf.xml Are there other xml specific items I need in my tool xml for an internal visualization 8. By clicking the visualization icon. what does it call (hopefully 1or2 above) so i can add specific code to display my datatype internal "visualization?" (preferred) in galaxy? ----- Original Message -----From: Eric Rasche <esr@tamu.edu>To: rbrown1422@comcast.netCc: galaxy-dev <galaxy-dev@lists.bx.psu.edu>Sent: Tue, 08 Dec 2015 02:34:47 -0000 (UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? Hi Bob, Yes, galaxy supports output formats of HTML+JS, though the tool producing them may need to be whitelisted (grep for 'whitelist' in your galaxy.ini) for display to work properly. If you wish to write your own visualization plugin for a specific datatype, there are assorted pieces of documentation on the wiki that might help you (though I usually just look at one of the other existing visualization plugins and copy+paste). Cheers, Eric 2015-12-07 20:08 GMT-06:00 <rbrown1422@comcast.net>: Good evening Galaxy team, Is it possible to run JavaScript or HTML as a tool or a display module? I see support for URLs, e.g. UCSC Genome Browser, but I would like to process a specific datatype to allow it to be viewed in Galaxy -- but the application is a combination of HTML and JavaScript. Does Galaxy support this? Thank you, Bob ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Eric Rasche Programmer II Center for Phage Technology Rm 312A, BioBio Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/