Re: [galaxy-dev] galaxy with torque
by remy.d1＠gmail.com
1 a drmaa d
Envoyé depuis mon HTC
----- Reply message -----
De : "Fernandez Edgar" <edgar.fernandez(a)umontreal.ca>
Pour : "John Chilton" <jmchilton(a)gmail.com>
Cc : "galaxy-dev(a)bx.psu.edu" <galaxy-dev(a)bx.psu.edu>
Objet : [galaxy-dev] galaxy with torque
Date : mer., nov. 26, 2014 19:09
Hi John,
First, thank you very much for your prompt answer.
It's extremely appreciated.
Secondly, I have some other questions: whatever answers you can provide me with, will be greatly helpful.
Please, forgive my beginner level understanding of an DRM system.
1. Once I compile, make and install the Torque Submit Node code against the server running Galaxy, what exactly is the purpose for an DRMAA ?
2. What is exactly the PBS step describe here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
galaxy_user@galaxy_server% LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python
What does it do exactly ?
3. All servers which includes my torque server, torque compute nodes and torque submit node (a.k.a galaxy server) have the galaxy user defines and its home shared on all of them. This means all of them have access (via NFS) of the installation directory of galaxy. But what about the MySQL server access. Does the torque server or the compute nodes need access to that service?
I hope I'm not sending too many emails/questions...
Thank you very much!
Cordialement / Regards,
Edgar Fernandez
-----Message d'origine-----
De : John Chilton [mailto:jmchilton@gmail.com]
Envoyé : November-25-14 12:07 PM
À : Fernandez Edgar
Cc : galaxy-dev(a)bx.psu.edu
Objet : Re: [galaxy-dev] galaxy with torque
I am not sure we have a walkthrough for Torque specifically - but if you have Galaxy up and running and you can qsub commands to torque - hopefully you have done all of the hard parts.
You will need a DRMAA library for your torque setup - https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
suggests compiling pbs_drmaa and outlines how to set it up. After that you just need to add a plugin and default destination to your job_conf.xml file - also outlined on that wiki page.
Other good resources to consult if you are scaling up your Galaxy this way are:
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer
https://wiki.galaxyproject.org/Events/GCC2014/TrainingDay/AdminWalkthrough
Good luck and let us know if you encounter any problems.
-John
On Fri, Nov 21, 2014 at 2:30 PM, Fernandez Edgar <edgar.fernandez(a)umontreal.ca> wrote:
> Hello all,
>
>
>
> My name is Edgar Fernandez. I’m a sys. admin. at University of Montreal.
>
> I’ve contacted you a while back about installing galaxy and I’ve
> successfully done it on a redhat 6 server.
>
>
>
> I see myself in a situation where I need to utilise all my redhat
> servers (who are identical to the server running the galaxy website).
>
>
>
> I’ve also successfully installed a server torque with compute notes
> and clients nodes.
>
>
>
> What are the last step to make the link between galaxy and torque?
>
> Also, once that connection is made, how will galaxy keep track of the
> jobs sent?
>
> I mean who will it know this job that just finished is for this user
> and not another ?
>
>
>
> Also, my torque installation is so that my server running galaxy is a
> submit node and a client node.
>
> I hope this is not a problem.
>
>
>
> Please help!
>
>
>
> Cordialement / Regards,
>
>
>
> Edgar Fernandez
>
> System Administrator (Linux)
>
> Direction Générale des Technologies de l'Information et de la
> Communication
>
> ( Bur. : 1-514-343-6111 poste 16568
>
>
>
> Université de Montréal
>
> PAVILLON ROGER-GAUDRY, bureau X-218
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this and other
> Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
5 years, 2 months
tool with conditional parameter broken on release 2014.08.11
by Anne Pajon
Dear,
I am looking for help after updating our Galaxy production server to release 2014.08.11, one of our in-house tool does not load in Galaxy anymore. I do think the problem is related to the conditional parameter because when I comment this section out Galaxy does display it. Any ideas where the problem could be?
Here is the xml file:
<tool id="hm" name="heatmap-public">
<description>Creates a heatmap from public datasets</description>
<command interpreter="R --vanilla --slave -f">
#silent sys.stderr.write("!!!! Cheetah Template Variables !!!!\n")
#for k,v in $searchList[2].items()
#silent sys.stderr.write(" %s = %s\n" % (str(k), str(v) ))
#end for
## path to the dataset selected extracted from the 'cri_datasets' data table (tool-data/cri/datasets.loc)
#import os
#set $dataset_relative_path = filter( lambda x: str( x[0] ) == str( $dataset.name ), $__app__.tool_data_tables[ 'cri_datasets' ].get_fields() )[0][-1]
#set $dataset_path = os.path.join($__tool_data_path__, $dataset_relative_path)
hm.R --args
$dataset.name
$dataset_path
$input
$col
$output_html
#if not $varExists('$output_pdf')
none
#else
$output_pdf
#end if
$output_html.files_path
$dataset.list
$gene
$dataset.label
$dataset.sample
## optional arguments set to 'none' when not relevant to a particular dataset
#if not $varExists('$dataset.blocks')
none
#else
$dataset.blocks
#end if
#if not $varExists('$dataset.group')
none
#else
$dataset.group
#end if
$scaling
$coloring.coloring
## optional arguments set to '0' when not relevant to a colour option
#if not $varExists('$coloring.limit')
0
#else
$coloring.limit
#end if
$cexrow
$cexcol
2>stderr || cat stderr 1>&2
</command>
<inputs>
<param name="input" type="data" format="tabular,txt" label="Gene List" help="Tab delimited text file containing a list of gene symbols"/>
<param name="col" type="data_column" data_ref="input" numerical="False" label="Column" help="Column containing gene symbols (only available for tabular input file)" />
<conditional name="dataset">
<param name="name" type="select" label="Dataset name">
<options from_data_table="cri_datasets">
<column name="name" index="1"/>
<column name="value" index="0"/>
<filter type="multiple_splitter" column="4" separator="," />
<filter type="static_value" value="heatmap" column="4" />
</options>
</param>
<when value="glinsky">
<param name="list" type="select" label="Input type" help="Type of input (Genes or Affymetrix probe IDs)">
<option value="genes" selected="true">Genes</option>
<option value="probes">Probes</option>
</param>
<param name="sample" type="select" label="Sample ordering" help="Cluster samples or order samples by Recurrence, Pathological Gleason grade or Surgical margin status">
<option value="cluster" selected="true">Cluster</option>
<option value="recurrence">Recurrence</option>
<option value="gleason">Gleason grade</option>
<option value="sms">Surgical margin status</option>
</param>
<param name="blocks" type="select" label="Add coloured blocks" help="Add coloured blocks for Recurrence, Pathological Gleason grade, Surgical margin status (Negative/Positive), all three or none">
<option value="recurrence" selected="true">Recurrence</option>
<option value="gleason">Gleason grade</option>
<option value="sms">Surgical margin status</option>
<option value="all">All three</option>
<option value="none">No blocks</option>
</param>
<param name="label" type="select" label="Gene labels" help="Label rows of heatmap with Probe IDs, Gene symbols or both">
<option value="both" selected="true">Both</option>
<option value="gene">Gene</option>
<option value="probe">Probe</option>
</param>
</when>
<when value="varambally">
<param name="list" type="select" label="Input type" help="Type of input (Genes or Affymetrix probe IDs)">
<option value="genes" selected="true">Genes</option>
<option value="probes">Probes</option>
</param>
<param name="sample" type="select" label="Sample ordering" help="Cluster samples or order by Group">
<option value="cluster" selected="true">Cluster</option>
<option value="group">Group</option>
</param>
<param name="label" type="select" label="Gene labels" help="Label rows of heatmap with Affymetrix Probe IDs, Gene symbols or both">
<option value="both" selected="true">Both</option>
<option value="gene">Gene</option>
<option value="probe">Probe</option>
</param>
</when>
<when value="taylor">
<param name="list" type="select" label="Input type" help="Type of input (Genes or Refseq Accession number)">
<option value="genes" selected="true">Genes</option>
<option value="access">Accession number</option>
</param>
<param name="sample" type="select" label="Sample ordering" help="Cluster samples or order samples by Recurrence, Gleason Grade, Surgical Margin Status, Group or Pairs (when matched pairs selected)">
<option value="cluster" selected="true">Cluster</option>
<option value="recurrence">Recurrence</option>
<option value="gleason">Gleason grade</option>
<option value="sms">Surgical margin status</option>
<option value="group">Group</option>
<option value="pairs">Pairs</option>
</param>
<param name="blocks" type="select" label="Add coloured blocks" help="Add coloured blocks for Recurrence (N/Y), Pathological Gleason grade (Low/High), Surgical margin status (Negative/Positive), Group (Benign/Primary/Met/Cell line), All four or None">
<option value="recurrence" selected="true">Recurrence</option>
<option value="gleason">Gleason grade</option>
<option value="sms">Surgical margin status</option>
<option value="group">Group</option>
<option value="all">All four</option>
<option value="none">No blocks</option>
</param>
<param name="group" type="select" label="Select groups" help="Select groups included in the plot, All, No cell line, Matched benign primary pairs, Cancer samples (Primary and Metastatic samples)">
<option value="all" selected="true">All</option>
<option value="nocellline">No cell line</option>
<option value="pairs">Matched pairs</option>
<option value="cancer">Cancer Samples</option>
</param>
<param name="label" type="select" label="Gene labels" help="Label rows of heatmap with Refseq Accession number, Gene symbols or both">
<option value="both" selected="true">Both</option>
<option value="gene">Gene</option>
<option value="access">Accession number</option>
</param>
</when>
</conditional>
<param name="gene" type="select" label="Gene clustering" help="Cluster genes or to retain original ordering">
<option value="no" selected="true">No</option>
<option value="yes">Yes</option>
</param>
<param name="scaling" type="select" label="Heatmap scaling" help="Chose how probes are scaled, median scaled (minus median), robust z-score (minus median divide median absolute deviation), or z-score (minus mean divide standard deviation">
<option value="median" selected="true">Median scaled</option>
<option value="robustz">Robust z-score</option>
<option value="zscore">Z-score</option>
</param>
<conditional name="coloring">
<param name="coloring" type="select" label="Heatmap colouring" help="How colouring is chosen, by heatmap i.e. colouring is specific for heatmap, or fixed i.e. the same probes in different heatmaps will have the same colour">
<option value="heatmap" selected="true">Heatmap</option>
<option value="fixed">Fixed</option>
</param>
<when value="fixed">
<param name="limit" type="float" value="3" label="Fixed limits" help="When chosing the fixed colour option, this value is the limit for choosing colours" />
</when>
</conditional>
<param name="cexrow" type="float" label="Scale for Gene (row) Labels" min="0" value="0.9" help="1 is normal, use a lower value if the gene names are overlapping, a higher value if they are too small" />
<param name="cexcol" type="float" label="Scale for Sample (column) Labels" min="0" value="1.0" help="1 is normal, use a lower value if the sample names are overlapping, a higher value if they are too small" />
<param name="pdf" type="boolean" label="PDF output" help="Tick if PDF output is required in addition to html output" />
</inputs>
<outputs>
<data format="html" name="output_html" />
<data format="pdf" name="output_pdf">
<filter>pdf is True</filter>
</data>
</outputs>
</tool>
Here is the error message:
Error - <type 'exceptions.ValueError'>: ('No case matched value:', 'dataset', None)
URL: http://uk-cri-lbio06/galaxy/tool_runner?tool_id=hm
File '/opt/local/home/webapp/galaxy/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__
app_iter = self.application(environ, sr_checker)
File '/opt/local/home/webapp/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__
return self.application(environ, start_response)
File '/opt/local/home/webapp/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__
return self.application(environ, start_response)
File '/opt/local/home/webapp/galaxy/lib/galaxy/web/framework/base.py', line 132 in __call__
return self.handle_request( environ, start_response )
File '/opt/local/home/webapp/galaxy/lib/galaxy/web/framework/base.py', line 190 in handle_request
body = method( trans, **kwargs )
File '/opt/local/home/webapp/galaxy/lib/galaxy/webapps/galaxy/controllers/tool_runner.py', line 90 in index
template, vars = tool.handle_input( trans, params.__dict__ )
File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2116 in handle_input
state, state_new = self.__fetch_state( trans, expanded_incoming, history, all_pages=all_pages )
File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2230 in __fetch_state
state = self.new_state( trans, history=history, all_pages=all_pages )
File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2015 in new_state
self.fill_in_new_state( trans, inputs, state.inputs, history=history )
File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py', line 2024 in fill_in_new_state
state[ input.name ] = input.get_initial_value( trans, context, history=history )
File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/parameters/grouping.py', line 527 in get_initial_value
current_case = self.get_current_case( test_value, trans )
File '/opt/local/home/webapp/galaxy/lib/galaxy/tools/parameters/grouping.py', line 480 in get_current_case
raise ValueError( "No case matched value:", self.name, str_value )
ValueError: ('No case matched value:', 'dataset', None)
Many thanks.
Kind regards,
Anne.
--
Dr Anne Pajon - Bioinformatics Core
Cancer Research UK - Cambridge Institute
Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE
anne.pajon(a)cruk.cam.ac.uk | +44 (0)1223 769 631
5 years, 2 months
Excessive memory consumption on local galaxy
by Langhorst, Brad
I’m running
hg id
2092948937ac+ (stable) release_2014.10.06
After a few days my web runners start to use excessive memory…
This was not the case in previous releases.
I did upgrade a few toolshed items around the same time and tried to add some reference genomes via the data manager tools.
Anybody else seeing this problem?
Any hints on the best way to track this down?
I do not relish a daily restart to paper over this issue…
(htop screenshot attached)
Brad
[cid:B1B04705-D365-4342-B0A7-472917ADA184@neb.com]
--
Bradley W. Langhorst, Ph.D.
Applications and Product Development Scientist
5 years, 2 months
No API way to delete a galaxy data library folder?
by Dooley, Damion
I just wanted to verify that this is the case? I've put in a Trello card requesting this feature for the API. My app creates and manages the content of some data library folders for the versioned data project.
Regards,
Damion
Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
5 years, 2 months
how to use api to log into Galaxy?
by xlwang
Hi,
I will integrate Galaxy with another system. When I logged into this system, and clicked a link jump to Galaxy, I hope this user has already logged into Galaxy. How to use api to deliver a session or somthing else to Galaxy?
Now, I can use api to create a new user in this system and also in Galaxy.
5 years, 2 months
Building reference genome with Data Manager socket.error: [Errno 110] Connection timed out
by Vova Korniychuk
Hello,
I am a new Galaxy learner. I try to master data managers. I have tried to
create sacCer2 reference genome following instruction on video
http://vimeo.com/74265510. In Data Manager History, it gives a following
error:
Traceback (most recent call last):
File "/home/uladzimir/shed_tools/
toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/...",
line 350, in <module>
if __name__ == "__main__": main()
File "/home/uladzimir/shed_tools/
toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/...",
line 345, in main
REFERENCE_SOURCE_TO_DOWNLOAD[
params['param_dict']['reference_source']['reference_source_selector'] ](
data_manager_dict, params, target_directory, dbkey, sequence_id,
sequence_name )
File "/home/uladzimir/shed_tools/
toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/...",
line 182, in download_from_ucsc
ftp = FTP( UCSC_FTP_SERVER )
File "/usr/lib/python2.7/ftplib.py", line 120, in __init__
self.connect(host)
File "/usr/lib/python2.7/ftplib.py", line 135, in connect
self.sock = socket.create_connection((self.host, self.port),
self.timeout)
File "/usr/lib/python2.7/socket.py", line 571, in create_connection
raise err
socket.error: [Errno 110] Connection timed out
Can you please help me?
Thanks
Uladzimir
5 years, 2 months
no reference genome list in variant_selector tool
by Evan Bollig PhD
Devteam:
The "cached" reference genome option on the variant_selector tool
(Main Toolshed) does not properly load reference options into the drop
down. Please comment out the following line to get it to work (the
same edit was already done in Unified Genotyper tool):
###########
diff -r 135e8721ffc5 variant_select.xml
--- a/variant_select.xml Tue Apr 01 10:49:48 2014 -0400
+++ b/variant_select.xml Wed Dec 03 13:53:46 2014 +0000
@@ -106,7 +106,7 @@
<param name="input_variant" type="data" format="vcf"
label="Variant file to select" help="-V,--variant
&lt;variant&gt;" />
<param name="ref_file" type="select" label="Using reference
genome" help="-R,--reference_sequence
&lt;reference_sequence&gt;">
<options from_data_table="gatk_picard_indexes">
- <filter type="data_meta" key="dbkey" ref="input_variant"
column="dbkey"/>
+ <!-- <filter type="data_meta" key="dbkey"
ref="input_variant" column="dbkey"/> -->
</options>
<validator type="no_options" message="A built-in reference
genome is not available for the build associated with the selected
input file"/>
</param>
###########
Can I rely on one of you folks in devteam to post a new revision of
the tool? I'd rather not maintain a full fork.
Cheers,
-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
evan(a)msi.umn.edu
boll0107(a)umn.edu
5 years, 2 months
Hovering over a history dataset lights up the other datasets....?
by Hans-Rudolf Hotz
Hi all
and just as a follow up to my previous mail:
If I continue reading the NewsBrief, it says:
"Hovering over a history dataset lights up the other datasets from which
it was derived."
This doesn't wotk for me. I also tried it on 'main' without success.
Is there a special trick I need to do?
Thanks for your help
Hans-Rudolf
--
Hans-Rudolf Hotz, PhD
Bioinformatics Support
Friedrich Miescher Institute for Biomedical Research
Maulbeerstrasse 66
4058 Basel/Switzerland
5 years, 2 months
