December 2014 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] galaxy with torque
by remy.d1＠gmail.com 05 Dec '14

05 Dec '14

1 a drmaa d Envoyé depuis mon HTC ----- Reply message ----- De : "Fernandez Edgar" <edgar.fernandez(a)umontreal.ca> Pour : "John Chilton" <jmchilton(a)gmail.com> Cc : "galaxy-dev(a)bx.psu.edu" <galaxy-dev(a)bx.psu.edu> Objet : [galaxy-dev] galaxy with torque Date : mer., nov. 26, 2014 19:09 Hi John, First, thank you very much for your prompt answer. It's extremely appreciated. Secondly, I have some other questions: whatever answers you can provide me with, will be greatly helpful. Please, forgive my beginner level understanding of an DRM system. 1. Once I compile, make and install the Torque Submit Node code against the server running Galaxy, what exactly is the purpose for an DRMAA ? 2. What is exactly the PBS step describe here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster galaxy_user@galaxy_server% LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python What does it do exactly ? 3. All servers which includes my torque server, torque compute nodes and torque submit node (a.k.a galaxy server) have the galaxy user defines and its home shared on all of them. This means all of them have access (via NFS) of the installation directory of galaxy. But what about the MySQL server access. Does the torque server or the compute nodes need access to that service? I hope I'm not sending too many emails/questions... Thank you very much! Cordialement / Regards, Edgar Fernandez -----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : November-25-14 12:07 PM À : Fernandez Edgar Cc : galaxy-dev(a)bx.psu.edu Objet : Re: [galaxy-dev] galaxy with torque I am not sure we have a walkthrough for Torque specifically - but if you have Galaxy up and running and you can qsub commands to torque - hopefully you have done all of the hard parts. You will need a DRMAA library for your torque setup - https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster suggests compiling pbs_drmaa and outlines how to set it up. After that you just need to add a plugin and default destination to your job_conf.xml file - also outlined on that wiki page. Other good resources to consult if you are scaling up your Galaxy this way are: https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer https://wiki.galaxyproject.org/Events/GCC2014/TrainingDay/AdminWalkthrough Good luck and let us know if you encounter any problems. -John On Fri, Nov 21, 2014 at 2:30 PM, Fernandez Edgar <edgar.fernandez(a)umontreal.ca> wrote: > Hello all, > > > > My name is Edgar Fernandez. I’m a sys. admin. at University of Montreal. > > I’ve contacted you a while back about installing galaxy and I’ve > successfully done it on a redhat 6 server. > > > > I see myself in a situation where I need to utilise all my redhat > servers (who are identical to the server running the galaxy website). > > > > I’ve also successfully installed a server torque with compute notes > and clients nodes. > > > > What are the last step to make the link between galaxy and torque? > > Also, once that connection is made, how will galaxy keep track of the > jobs sent? > > I mean who will it know this job that just finished is for this user > and not another ? > > > > Also, my torque installation is so that my server running galaxy is a > submit node and a client node. > > I hope this is not a problem. > > > > Please help! > > > > Cordialement / Regards, > > > > Edgar Fernandez > > System Administrator (Linux) > > Direction Générale des Technologies de l'Information et de la > Communication > > ( Bur. : 1-514-343-6111 poste 16568 > > > > Université de Montréal > > PAVILLON ROGER-GAUDRY, bureau X-218 > > > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other > Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

5 13

tool with conditional parameter broken on release 2014.08.11
by Anne Pajon 05 Dec '14

05 Dec '14

Dear, I am looking for help after updating our Galaxy production server to release 2014.08.11, one of our in-house tool does not load in Galaxy anymore. I do think the problem is related to the conditional parameter because when I comment this section out Galaxy does display it. Any ideas where the problem could be? Here is the xml file: <tool id="hm" name="heatmap-public"> <description>Creates a heatmap from public datasets</description> <command interpreter="R --vanilla --slave -f"> #silent sys.stderr.write("!!!! Cheetah Template Variables !!!!\n") #for k,v in $searchList[2].items() #silent sys.stderr.write(" %s = %s\n" % (str(k), str(v) )) #end for ## path to the dataset selected extracted from the 'cri_datasets' data table (tool-data/cri/datasets.loc) #import os #set $dataset_relative_path = filter( lambda x: str( x[0] ) == str( $dataset.name ), $__app__.tool_data_tables[ 'cri_datasets' ].get_fields() )[0][-1] #set $dataset_path = os.path.join($__tool_data_path__, $dataset_relative_path) hm.R --args $dataset.name $dataset_path $input $col $output_html #if not $varExists('$output_pdf') none #else $output_pdf #end if $output_html.files_path $dataset.list $gene $dataset.label $dataset.sample ## optional arguments set to 'none' when not relevant to a particular dataset #if not $varExists('$dataset.blocks') none #else $dataset.blocks #end if #if not $varExists('$dataset.group') none #else $dataset.group #end if $scaling $coloring.coloring ## optional arguments set to '0' when not relevant to a colour option #if not $varExists('$coloring.limit') 0 #else $coloring.limit #end if $cexrow $cexcol 2>stderr || cat stderr 1>&2 </command> <inputs> <param name="input" type="data" format="tabular,txt" label="Gene List" help="Tab delimited text file containing a list of gene symbols"/> <param name="col" type="data_column" data_ref="input" numerical="False" label="Column" help="Column containing gene symbols (only available for tabular input file)" /> <conditional name="dataset"> <param name="name" type="select" label="Dataset name"> <options from_data_table="cri_datasets"> <column name="name" index="1"/> <column name="value" index="0"/> <filter type="multiple_splitter" column="4" separator="," /> <filter type="static_value" value="heatmap" column="4" /> </options> </param> <when value="glinsky"> <param name="list" type="select" label="Input type" help="Type of input (Genes or Affymetrix probe IDs)"> <option value="genes" selected="true">Genes</option> <option value="probes">Probes</option> </param> <param name="sample" type="select" label="Sample ordering" help="Cluster samples or order samples by Recurrence, Pathological Gleason grade or Surgical margin status"> <option value="cluster" selected="true">Cluster</option> <option value="recurrence">Recurrence</option> <option value="gleason">Gleason grade</option> <option value="sms">Surgical margin status</option> </param> <param name="blocks" type="select" label="Add coloured blocks" help="Add coloured blocks for Recurrence, Pathological Gleason grade, Surgical margin status (Negative/Positive), all three or none"> <option value="recurrence" selected="true">Recurrence</option> <option value="gleason">Gleason grade</option> <option value="sms">Surgical margin status</option> <option value="all">All three</option> <option value="none">No blocks</option> </param> <param name="label" type="select" label="Gene labels" help="Label rows of heatmap with Probe IDs, Gene symbols or both"> <option value="both" selected="true">Both</option> <option value="gene">Gene</option> <option value="probe">Probe</option> </param> </when> <when value="varambally"> <param name="list" type="select" label="Input type" help="Type of input (Genes or Affymetrix probe IDs)"> <option value="genes" selected="true">Genes</option> <option value="probes">Probes</option> </param> <param name="sample" type="select" label="Sample ordering" help="Cluster samples or order by Group"> <option value="cluster" selected="true">Cluster</option> <option value="group">Group</option> </param> <param name="label" type="select" label="Gene labels" help="Label rows of heatmap with Affymetrix Probe IDs, Gene symbols or both"> <option value="both" selected="true">Both</option> <option value="gene">Gene</option> <option value="probe">Probe</option> </param> </when> <when value="taylor"> <param name="list" type="select" label="Input type" help="Type of input (Genes or Refseq Accession number)"> <option value="genes" selected="true">Genes</option> <option value="access">Accession number</option> </param> <param name="sample" type="select" label="Sample ordering" help="Cluster samples or order samples by Recurrence, Gleason Grade, Surgical Margin Status, Group or Pairs (when matched pairs selected)"> <option value="cluster" selected="true">Cluster</option> <option value="recurrence">Recurrence</option> <option value="gleason">Gleason grade</option> <option value="sms">Surgical margin status</option> <option value="group">Group</option> <option value="pairs">Pairs</option> </param> <param name="blocks" type="select" label="Add coloured blocks" help="Add coloured blocks for Recurrence (N/Y), Pathological Gleason grade (Low/High), Surgical margin status (Negative/Positive), Group (Benign/Primary/Met/Cell line), All four or None"> <option value="recurrence" selected="true">Recurrence</option> <option value="gleason">Gleason grade</option> <option value="sms">Surgical margin status</option> <option value="group">Group</option> <option value="all">All four</option> <option value="none">No blocks</option> </param> <param name="group" type="select" label="Select groups" help="Select groups included in the plot, All, No cell line, Matched benign primary pairs, Cancer samples (Primary and Metastatic samples)"> <option value="all" selected="true">All</option> <option value="nocellline">No cell line</option> <option value="pairs">Matched pairs</option> <option value="cancer">Cancer Samples</option> </param> <param name="label" type="select" label="Gene labels" help="Label rows of heatmap with Refseq Accession number, Gene symbols or both"> <option value="both" selected="true">Both</option> <option value="gene">Gene</option> <option value="access">Accession number</option> </param> </when> </conditional> <param name="gene" type="select" label="Gene clustering" help="Cluster genes or to retain original ordering"> <option value="no" selected="true">No</option> <option value="yes">Yes</option> </param> <param name="scaling" type="select" label="Heatmap scaling" help="Chose how probes are scaled, median scaled (minus median), robust z-score (minus median divide median absolute deviation), or z-score (minus mean divide standard deviation"> <option value="median" selected="true">Median scaled</option> <option value="robustz">Robust z-score</option> <option value="zscore">Z-score</option> </param> <conditional name="coloring"> <param name="coloring" type="select" label="Heatmap colouring" help="How colouring is chosen, by heatmap i.e. colouring is specific for heatmap, or fixed i.e. the same probes in different heatmaps will have the same colour"> <option value="heatmap" selected="true">Heatmap</option> <option value="fixed">Fixed</option> </param> <when value="fixed"> <param name="limit" type="float" value="3" label="Fixed limits" help="When chosing the fixed colour option, this value is the limit for choosing colours" /> </when> </conditional> <param name="cexrow" type="float" label="Scale for Gene (row) Labels" min="0" value="0.9" help="1 is normal, use a lower value if the gene names are overlapping, a higher value if they are too small" /> <param name="cexcol" type="float" label="Scale for Sample (column) Labels" min="0" value="1.0" help="1 is normal, use a lower value if the sample names are overlapping, a higher value if they are too small" /> <param name="pdf" type="boolean" label="PDF output" help="Tick if PDF output is required in addition to html output" /> </inputs> <outputs> <data format="html" name="output_html" /> <data format="pdf" name="output_pdf"> <filter>pdf is True</filter> </data> </outputs> </tool> Here is the error message: Error - <type 'exceptions.ValueError'>: ('No case matched value:', 'dataset', None) URL: http://uk-cri-lbio06/galaxy/tool_runner?tool_id=hm File '/opt/local/home/webapp/galaxy/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/opt/local/home/webapp/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/opt/local/home/webapp/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/opt/local/home/webapp/galaxy/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/opt/local/home/webapp/galaxy/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/opt/local/home/webapp/galaxy/lib/galaxy/webapps/galaxy/controllers/tool_runner.py', line 90 in index template, vars = tool.handle_input( trans, params.__dict__ ) File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2116 in handle_input state, state_new = self.__fetch_state( trans, expanded_incoming, history, all_pages=all_pages ) File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2230 in __fetch_state state = self.new_state( trans, history=history, all_pages=all_pages ) File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2015 in new_state self.fill_in_new_state( trans, inputs, state.inputs, history=history ) File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2024 in fill_in_new_state state[ input.name ] = input.get_initial_value( trans, context, history=history ) File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/parameters/grouping.py', line 527 in get_initial_value current_case = self.get_current_case( test_value, trans ) File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/parameters/grouping.py', line 480 in get_current_case raise ValueError( "No case matched value:", self.name, str_value ) ValueError: ('No case matched value:', 'dataset', None) Many thanks. Kind regards, Anne. -- Dr Anne Pajon - Bioinformatics Core Cancer Research UK - Cambridge Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon(a)cruk.cam.ac.uk | +44 (0)1223 769 631

1 0

Excessive memory consumption on local galaxy
by Langhorst, Brad 05 Dec '14

05 Dec '14

I’m running hg id 2092948937ac+ (stable) release_2014.10.06 After a few days my web runners start to use excessive memory… This was not the case in previous releases. I did upgrade a few toolshed items around the same time and tried to add some reference genomes via the data manager tools. Anybody else seeing this problem? Any hints on the best way to track this down? I do not relish a daily restart to paper over this issue… (htop screenshot attached) Brad [cid:B1B04705-D365-4342-B0A7-472917ADA184@neb.com] -- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist

2 1

No API way to delete a galaxy data library folder?
by Dooley, Damion 05 Dec '14

05 Dec '14

I just wanted to verify that this is the case? I've put in a Trello card requesting this feature for the API. My app creates and manages the content of some data library folders for the versioned data project. Regards, Damion Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

2 2

Workflow Variables
by Adhemar 03 Dec '14

03 Dec '14

Hi, The workflow variables are awesome! https://wiki.galaxyproject.org/Learn/AdvancedWorkflow/VariablesEdit It's really nice to have the output files properly named after running several workflow steps. It would be great to use them in other fields like tophat's 'read groups' for instance. Thanks! Adhemar

2 1

how to use api to log into Galaxy?
by xlwang 03 Dec '14

03 Dec '14

Hi, I will integrate Galaxy with another system. When I logged into this system, and clicked a link jump to Galaxy, I hope this user has already logged into Galaxy. How to use api to deliver a session or somthing else to Galaxy? Now, I can use api to create a new user in this system and also in Galaxy.

2 1

Building reference genome with Data Manager socket.error: [Errno 110] Connection timed out
by Vova Korniychuk 03 Dec '14

03 Dec '14

Hello, I am a new Galaxy learner. I try to master data managers. I have tried to create sacCer2 reference genome following instruction on video http://vimeo.com/74265510. In Data Manager History, it gives a following error: Traceback (most recent call last): File "/home/uladzimir/shed_tools/ toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/2e…", line 350, in <module> if __name__ == "__main__": main() File "/home/uladzimir/shed_tools/ toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/2e…", line 345, in main REFERENCE_SOURCE_TO_DOWNLOAD[ params['param_dict']['reference_source']['reference_source_selector'] ]( data_manager_dict, params, target_directory, dbkey, sequence_id, sequence_name ) File "/home/uladzimir/shed_tools/ toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/2e…", line 182, in download_from_ucsc ftp = FTP( UCSC_FTP_SERVER ) File "/usr/lib/python2.7/ftplib.py", line 120, in __init__ self.connect(host) File "/usr/lib/python2.7/ftplib.py", line 135, in connect self.sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err socket.error: [Errno 110] Connection timed out Can you please help me? Thanks Uladzimir

1 0

no reference genome list in variant_selector tool
by Evan Bollig PhD 03 Dec '14

03 Dec '14

Devteam: The "cached" reference genome option on the variant_selector tool (Main Toolshed) does not properly load reference options into the drop down. Please comment out the following line to get it to work (the same edit was already done in Unified Genotyper tool): ########### diff -r 135e8721ffc5 variant_select.xml --- a/variant_select.xml Tue Apr 01 10:49:48 2014 -0400 +++ b/variant_select.xml Wed Dec 03 13:53:46 2014 +0000 @@ -106,7 +106,7 @@ <param name="input_variant" type="data" format="vcf" label="Variant file to select" help="-V,--variant &lt;variant&gt;" /> <param name="ref_file" type="select" label="Using reference genome" help="-R,--reference_sequence &lt;reference_sequence&gt;"> <options from_data_table="gatk_picard_indexes"> - <filter type="data_meta" key="dbkey" ref="input_variant" column="dbkey"/> +  </options> <validator type="no_options" message="A built-in reference genome is not available for the build associated with the selected input file"/> </param> ########### Can I rely on one of you folks in devteam to post a new revision of the tool? I'd rather not maintain a full fork. Cheers, -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 evan(a)msi.umn.edu boll0107(a)umn.edu

1 1

Page generated by a Galaxy behind HTTPS(Nginx) makes HTTP calls
by Maxime Lévesque 03 Dec '14

03 Dec '14

I have followed the instructions here : https://wiki.galaxyproject.org/Admin/Config/nginxProxy to run a Galaxy behind an NGinx proxy under with HTTPS, The relevant part of my NGinx config has : proxy_set_header REMOTE_USER "$loginOnHost"; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-URL-SCHEME https; it is working well, with the exception that some pages are making external calls to http://genome.ucsc.edu, and not https://genome.ucsc.edu . I get a "mixed content" error (in chrome) : I verified that the URL that the page is trying to access *does* exist under HTTPS, so it's just a matter of telling galaxy to generate HTTPS links. I would have thought that this is what "proxy_set_header X-URL-SCHEME https;" did... https://galaxy-1hge11q-uvd.udes.genap.ca/' was loaded over HTTPS, but requested an insecure resource ' http://genome.ucsc.edu/cgi-bin/hgTables?GALAXY_URL=https%3A//galaxy-1hge11q…'. This request has been blocked; the content must be served over HTTPS. Thanks !

2 1

Hovering over a history dataset lights up the other datasets....?
by Hans-Rudolf Hotz 02 Dec '14

02 Dec '14

Hi all and just as a follow up to my previous mail: If I continue reading the NewsBrief, it says: "Hovering over a history dataset lights up the other datasets from which it was derived." This doesn't wotk for me. I also tried it on 'main' without success. Is there a special trick I need to do? Thanks for your help Hans-Rudolf -- Hans-Rudolf Hotz, PhD Bioinformatics Support Friedrich Miescher Institute for Biomedical Research Maulbeerstrasse 66 4058 Basel/Switzerland

2 1