Thanks a lot Alban for these explanations ,
Everything is OK.

++,
Sarah

alermine a écrit :
Hi all, 

* For the tool FilterControl, our cluster is configured to kill jobs that use more than 4Go of memory.
I don't achieve to modify qsub options in my Galaxy instance, so I'he changed this option -Xmx6g in  "-Xmx4g". Maybe some treatment won't run by lake of memory ..
The memory that you specify with -Xmx6g is the memory used by Java and NOT the memory used for the job launched by Galaxy on the cluster (see http://www.auditmypc.com/java-memory-xmx512.asp).
So you have to specify in your java configuration the min and max memory usable. 
If you have any idea on how to add option on Galaxy qsub, could you please help me ?
I would like to add these options to qsub : qsub -l mem=6G -l h_vmem=8G
Try by specifying in universe_wsgi.ini runner:///queue/-l nodes=1:ppn=1,mem=6gb,h_vmem=8gb/

If it's not understood by Galaxy,To add new qsub options, you have to make some modifications in the source code:

1. Identify your scheduler (pbs, drmaa, sge)
2. Edit the python script which creates jobs for your scheduler:
GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/pbs.py
GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/sge.py

3. Search in the script the function which parse your scheduler options (ex: for pbs.py, the function is named "def determine_pbs_options( self, url ):")

4. Modify the parsing step to make this function understand the h_vmem option

Hope it's help you,

++,

Alban

--
Alban Lermine
Unité 900: INSERM - Mines ParisTech - Institut Curie
" Bioinformatics and Computational Systems Biology of Cancer"
11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
Tel: +33 (0) 1 56 24 69 84



Le 5 févr. 2013 à 14:53, Sarah Maman <sarah.maman@toulouse.inra.fr> a écrit :

Hi Alban, Marie-Stephane and Bjoern,


* For IntersectBed tool, running is OK. I've just deleted ";" in xml file and your tool runs. My cluster add one ";" so a double ";;" gives an error.

I thing that this new eroor is due to files tested :

Differing number of VCF fields encountered at line: 38.  Exiting...

*For RepeatMasker tool, thanks to Marie-Stephane (CONGRATULATIONS MARIE-STEPHANE !), we need to specify that bash commands are included iwith Python. So xml file have been modify in this way :
- Specify "perl" before RepeatMasker command.
- Specify RepeatMasker path
- Include all bash comman in a Python script :  #os.system("cp $gff_file $output_gff;")  instead of  cp $gff_file $output_gff;
So the code is :

    <command>
## The command is a Cheetah template which allows some Python based syntax.
## Lines starting hash hash are comments. Galaxy will turn newlines into spaces
## create temp directory
#import tempfile, os
#set $dirname = os.path.abspath(tempfile.mkdtemp())
#set $input_filename = os.path.split(str($query))[-1]
#set $output_basename = os.path.join($dirname, $input_filename)
perl /usr/local/bioinfo/bin/RepeatMasker -parallel 8 $nolow $noint $norna
#if str($species)!="all":
    $species
#end if
-dir $dirname
#if $adv_opts.adv_opts_selector=="advanced":
    #if str($adv_opts.gc)!="0":
        -gc $adv_opts.gc
    #end if
    $adv_opts.gccalc
    #set $output_files_list = str($adv_opts.output_files).split(',')
    #if "gff" in $output_files_list:
        -gff
    #end if
    #if "html" in $output_files_list:
        -html
    #end if
    $adv_opts.slow_search
    $adv_opts.quick_search
    $adv_opts.rush_search
    $adv_opts.only_alus
    $adv_opts.is_only
#else:
    ## Set defaults
    -gff
## End of advanced options:
#end if
$query
/dev/null 2> /dev/null;

## Copy the output files to galaxy
#if $adv_opts.adv_opts_selector=="advanced":
    #if "summary" in $output_files_list:
        ## Write out the summary file (default)
        #set $summary_file = $output_basename + '.tbl'
        #os.system("cp $summary_file $output_summary;")
    #end if
    #if "gff" in $output_files_list:
        ## Write out the gff file (default)
        #set $gff_file = $output_basename + '.out.gff'
        #os.system("cp $gff_file $output_gff");
    #end if
    #if "html" in $output_files_list:
        ## Write out the html file
        #set $html_file = $output_basename + '.out.html'
        #os.system("cp $html_file $output_html;")
    #end if
#else:

    ## Write out the summary file (default)
    #set $summary_file = $output_basename + '.tbl'
    #os.system("cp $summary_file $output_summary;")
    ## Write out the gff file (default)
    #set $gff_file = $output_basename + '.out.gff'
    #os.system("cp $gff_file $output_gff;")
## End of advanced options:
#end if
## Write out mask sequence file
#set $mask_sequence_file = $output_basename + '.masked'
#os.system("cp $mask_sequence_file $output_mask;")
## Write out standard file (default)
## The default '.out' file from RepeatMasker has a 3-line header and spaces rather
## than tabs. Remove the header and replace the whitespaces with tab
#set $standard_file = $output_basename + '.out'
#os.system("tail -n +4 $standard_file | tr -s ' ' '\t' > $output_std;")
## Delete all temporary files
#os.system("rm $dirname -r;" 


* For the tool FilterControl, our cluster is configured to kill jobs that use more than 4Go of memory.
I don't achieve to modify qsub options in my Galaxy instance, so I'he changed this option -Xmx6g in  "-Xmx4g". Maybe some treatment won't run by lake of memory ..
If you have any idea on how to add option on Galaxy qsub, could you please help me ?
I would like to add these options to qsub : qsub -l mem=6G -l h_vmem=8G


*  ChIPMunk :Sorry, but all ChIPMunk files (xml, pl; sh) didn't have execution rights... So I just do "chmod a+x" on these files and ChIPMunk tool is OK in my Galaxy instance....

Thanks a lot for all your explanations, Alban, Marie-Stephane and Bjoern .

Thanks in advance for qsub and IntersectBed tool (test file),

Sarah






alermine a écrit :
Hi Sarah,

I'll try to debug, point per point:

- Here is the error I get when running the tool FilterControl 
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 ***

I think here you have a misconfiguration of the memory of your java install (according to the need of the tool)

If you look at the FilterControlPeaks.sh file, the java is called with the option -Xmx6g. So your java install have to be allowed to use 6G as memory (by default it's 1024M)

- Here is the error I get when running the tool IntersectBed 

/work/galaxy/database/pbs/galaxy_4129.sh: line 13: Erreur de syntaxe
près du symbole inattendu « ;; »
/work/galaxy/database/pbs/galaxy_4129.sh: line 13: `bedtools intersect
-f 0.05

I don't understand this one, The intersectBed tool is only composed of a xml which simply call bedtools..

Check the command by typing 'bedtools intersect' in a terminal, if it sends you back the option list, the Galaxy tool should work.. If not, you have to check your bedtools install

- Here is the error I get when running the tool ChIPmunk (thanks Alban
for you help (ChIPMunk v2 is used and all librairies are found) but
here is a new error, maybe a problem of "dirname" ?)

mv: impossible d'évaluer
« /work/galaxy/database/job_working_directory/004/4132/galaxy_dataset_5992.dat_0.xml.png »: Aucun fichier ou dossier de ce type
mv: impossible d'évaluer « /work/gala
Here, I think that the error is due to the ChIPMunk execution which doesn't work.

You can try the ChIPMunk execution this way:

Find the line containing:

ruby $CHIPMUNK_BIN_DIR/run_chiphorde.rb $LOG $motifs $MODE yes 1.0 s:$inputfile 100 10 1 4 >> /dev/null 2>> /dev/null

Add a line with:

echo "ruby $CHIPMUNK_BIN_DIR/run_chiphorde.rb $LOG $motifs $MODE yes 1.0 s:$inputfile 100 10 1 4 >> /dev/null 2>> /dev/null" > ~/ChIPMunk.log

Then try to launch the command in ~/ChIPMunk.log in a terminal and look at the output.

If ChIPMunk has run, the error comes from next step (i.e. the parseChipmunkOutput.pl script).

Else, try to debug ChIPMunk, and then the tool will work.
- Here is the error I get when running the tool PeaksToBed 
readline() on closed filehandle FILE at
	/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 124 (#1)
    (W closed) The filehandle you're reading from got itself closed sometime
    before now.  Check your control flow.
    
Use of uninitialized value $fields[1] in concatenation (.) or string at

For this one, edit the PeaksToBed.pl script this way:

The 3 lines:

use strict;
use warnings;
use diagnostics;

becomes:

#use strict;
#use warnings;
#use diagnostics;

Galaxy doesn't really love to have the return of a script (even if it's not an error). Here the perl scripts gives you warnings that are understood as errors by Galaxy.


Hope It will help you, don't hesitate if you need more..

++,

Alban



--
Alban Lermine
Unité 900: INSERM - Mines ParisTech - Institut Curie
" Bioinformatics and Computational Systems Biology of Cancer"
11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
Tel: +33 (0) 1 56 24 69 84



Le 1 févr. 2013 à 14:41, Sarah Maman <sarah.maman@toulouse.inra.fr> a écrit :

Thanks Bjoern,

In your xml file, I 've specified the interpreter :
<command> replace by <command interpreter="bash"> but the problem is not solved (";" is supported by our cluster) because the command is runned with an error ";;" (double ;) instead of ";" (simple ;) :
`bash /my/path/to/RepeatMasker -parallel 8         -species fungi   -dir /work/tmp/tmprRrJh2      -gff   /my/path/to/galaxy/database/files/005/dataset_5979.dat   > /dev/null 2> /dev/null;      
cp /my/path/to/work/tmp/tmprRrJh2/dataset_5979.dat.tbl /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5999.dat;      
cp /work/tmp/tmprRrJh2/dataset_5979.dat.out.gff None;   
cp /work/tmp/tmprRrJh2/dataset_5979.dat.masked None; 
tail -n +4 /work/tmp/tmprRrJh2/dataset_5979.dat.out | tr -s ' ' '\t' > 
/my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5998.dat;  
rm /work/tmp/tmprRrJh2 -r;; 
cd /my/path/to/galaxy/galaxy-dist;
/my/path/to/galaxy/galaxy-dist/set_metadata.sh /my/path/to/galaxy/database/files /my/path/to/galaxy/database/job_working_directory/004/4135 . /my/path/to/galaxy/galaxy-dist/universe_wsgi.ini 
/work/tmp/tmpi0u8TZ /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy.json /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_in_HistoryDatasetAssociation_6156_Lp_2X3,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_kwds_HistoryDatasetAssociation_6156_emJ62i,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_out_HistoryDatasetAssociation_6156_Dgw3la,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_results_HistoryDatasetAssociation_6156_aTCjOU,
/my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5999.dat,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_override_HistoryDatasetAssociation_6156_I6enyB 
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_in_HistoryDatasetAssociation_6155_LRm2gT,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_kwds_HistoryDatasetAssociation_6155_4xAztH,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_out_HistoryDatasetAssociation_6155_14AYvT,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_results_HistoryDatasetAssociation_6155_l_1gJ2,
/my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5998.dat,
/my/path/to/galaxy/database/job_working_directory/004/4135/metadata_override_HistoryDatasetAssociation_6155_NufEns'
So the error is ";;"
rm /work/tmp/tmprRrJh2 -r;;
instead of ";"
rm /work/tmp/tmprRrJh2 -r;
I've tried to replace ";" by nothing but this generate others errors (files not generated) :
cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.tbl »: Aucun fichier ou dossier de ce type
cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.out.gff »: Aucun fichier ou dossier de ce type
cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.masked »: Aucun fichier ou dossier de ce type
tail: impossible d'ouvrir « /work/tmp/tmpga_QOo/dataset_5979.dat.out » en lecture: Aucun fichier ou dossier de ce type
Do you have any idea ?
Thanks in advance,
Sarah




Björn Grüning a écrit : 
Hi Sarah,

i looks like something is wrong with the PBS system.
The errors for bedtools and repeatmasker are the same. I can't
understand french but i guess that ";" are not supported by PBS or in
your configuration? Repeatmasker and bedtools creating more than one
shell command and concatenating them with ';'.

Hope that helps,
Bjoern

  
Hello,

I've installed NEBULA tools and "RepeatMasker" tool on our local
Galaxy instance.

For some tools (others work), I get the following errors:
- Here is the error I get when running the tool FilterControl 
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 ***
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 ***
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 ***
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 ***
/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/FilterControlPeaks/FilterControlPeaks.sh: line 80: 61310 Abandon                 java -classpath $LOCAL_PATH/ -Xmx6g FilterPeaks -f $inputfile -c $controlfile -t $minHeight -v $minRatio -o $output > /dev/null
*** glibc detected *** java: double free or corruption (!prev): 0x00007f1f5800ecd0 ***
/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/FilterControlPeaks/FilterControlPeaks.sh: line 81: 61351 Abandon      

- Here is the error I get when running the tool IntersectBed 

/work/galaxy/database/pbs/galaxy_4129.sh: line 13: Erreur de syntaxe
près du symbole inattendu « ;; »
/work/galaxy/database/pbs/galaxy_4129.sh: line 13: `bedtools intersect
-f 0.05

- Here is the error I get when running the tool ChIPmunk (thanks Alban
for you help (ChIPMunk v2 is used and all librairies are found) but
here is a new error, maybe a problem of "dirname" ?)

mv: impossible d'évaluer
« /work/galaxy/database/job_working_directory/004/4132/galaxy_dataset_5992.dat_0.xml.png »: Aucun fichier ou dossier de ce type
mv: impossible d'évaluer « /work/gala


- Here is the error I get when running the tool PeaksToBed 
readline() on closed filehandle FILE at
	/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 124 (#1)
    (W closed) The filehandle you're reading from got itself closed sometime
    before now.  Check your control flow.
    
Use of uninitialized value $fields[1] in concatenation (.) or string at
	/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 154, <FILE> line 1 (#2)
    (W uninitialized) An undefined value was used as if it were already
    defined.  It was interpreted as a "" or a 0, but maybe it was a mistake.
    To suppress this warning assign a defined value to your variables.
    
    To help you figure out what was undefined, perl will try to tell you the
    name of the variable (if any) that was undefined. In some cases it cannot
    do this, so it also tells you what operation you used the undefined value
    in.  Note, however, that perl optimizes your program and the operation
    displayed in the warning may not necessarily appear literally in your
    program.  For example, "that $foo" is usually optimized into "that "
    . $foo, and the warning will refer to the concatenation (.) operator,
    even though there is no . in your program.
    
Use of uninitialized value in concatenation (.) or string at
	/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 154, <FILE> line 1 (#2)
Use of uninitialized value in numeric ge (>=) at
	/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 156, <FILE> line 1 (#2)
......
	/usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 156, <FILE> line 63 (#2)

Use of uninitialized value $fields[1] in conc


I dowloaded tool "RepeatMasker" from Galaxy Tool Shed,and RepeatMasker
is installed on our cluster.
Here is the error I get when running the tool RepeatMasker:

An error occurred running this job: Epilog : job finished at jeu.
janv. 31 14:39:16 CET 2013
/work/galaxy/database/pbs/galaxy_4134.sh: line 13: Erreur de syntaxe
près du symbole inattendu « ;; »
/work/galaxy/database/pbs/galaxy_4134.sh: line 13: `RepeatMasker
-parallel 8 -speci


Do you have any ideas ?
Thank you in advance for your help,
Sarah Maman 
-- 
          --*--
Sarah Maman
INRA - LGC - SIGENAE
http://www.sigenae.org/
Chemin de Borde-Rouge - Auzeville - BP 52627
31326 Castanet-Tolosan cedex - FRANCE
Tel:   +33(0)5.61.28.57.08
Fax:   +33(0)5.61.28.57.53 
         --*--
    

  


-- 
          --*--
Sarah Maman
INRA - LGC - SIGENAE
http://www.sigenae.org/
Chemin de Borde-Rouge - Auzeville - BP 52627
31326 Castanet-Tolosan cedex - FRANCE
Tel:   +33(0)5.61.28.57.08
Fax:   +33(0)5.61.28.57.53 
         --*--



-- 
          --*--
Sarah Maman
INRA - LGC - SIGENAE
http://www.sigenae.org/
Chemin de Borde-Rouge - Auzeville - BP 52627
31326 Castanet-Tolosan cedex - FRANCE
Tel:   +33(0)5.61.28.57.08
Fax:   +33(0)5.61.28.57.53 
         --*--



-- 
          --*--
Sarah Maman
INRA - LGC - SIGENAE
http://www.sigenae.org/
Chemin de Borde-Rouge - Auzeville - BP 52627
31326 Castanet-Tolosan cedex - FRANCE
Tel:   +33(0)5.61.28.57.08
Fax:   +33(0)5.61.28.57.53 
         --*--