Hi all,
- For the tool FilterControl, our cluster is configured to kill jobs that use more than 4Go of memory.
I don't achieve to modify qsub options in my Galaxy instance, so I'he changed this option -Xmx6g in "-Xmx4g". Maybe some treatment won't run by lake of memory ..
The memory that you specify with -Xmx6g is the memory used by Java and NOT the memory used for the job launched by Galaxy on the cluster (see http://www.auditmypc.com/java-memory-xmx512.asp). So you have to specify in your java configuration the min and max memory usable.
If you have any idea on how to add option on Galaxy qsub, could you please help me ? I would like to add these options to qsub : qsub -l mem=6G -l h_vmem=8G
Try by specifying in universe_wsgi.ini runner:///queue/-l nodes=1:ppn=1,mem=6gb,h_vmem=8gb/
If it's not understood by Galaxy,To add new qsub options, you have to make some modifications in the source code:
1. Identify your scheduler (pbs, drmaa, sge) 2. Edit the python script which creates jobs for your scheduler: GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/pbs.py GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/sge.py
3. Search in the script the function which parse your scheduler options (ex: for pbs.py, the function is named "def determine_pbs_options( self, url ):")
4. Modify the parsing step to make this function understand the h_vmem option
Hope it's help you,
++,
Alban
-- Alban Lermine Unité 900: INSERM - Mines ParisTech - Institut Curie " Bioinformatics and Computational Systems Biology of Cancer" 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel: +33 (0) 1 56 24 69 84
Le 5 févr. 2013 à 14:53, Sarah Maman sarah.maman@toulouse.inra.fr a écrit :
Hi Alban, Marie-Stephane and Bjoern,
- For IntersectBed tool, running is OK. I've just deleted ";" in xml file and your tool runs. My cluster add one ";" so a double ";;" gives an error.
I thing that this new eroor is due to files tested :
Differing number of VCF fields encountered at line: 38. Exiting...
*For RepeatMasker tool, thanks to Marie-Stephane (CONGRATULATIONS MARIE-STEPHANE !), we need to specify that bash commands are included iwith Python. So xml file have been modify in this way :
- Specify "perl" before RepeatMasker command.
- Specify RepeatMasker path
- Include all bash comman in a Python script : #os.system("cp $gff_file $output_gff;") instead of cp $gff_file $output_gff;
So the code is : <command> ## The command is a Cheetah template which allows some Python based syntax. ## Lines starting hash hash are comments. Galaxy will turn newlines into spaces ## create temp directory #import tempfile, os #set $dirname = os.path.abspath(tempfile.mkdtemp()) #set $input_filename = os.path.split(str($query))[-1] #set $output_basename = os.path.join($dirname, $input_filename) perl /usr/local/bioinfo/bin/RepeatMasker -parallel 8 $nolow $noint $norna #if str($species)!="all": $species #end if -dir $dirname #if $adv_opts.adv_opts_selector=="advanced": #if str($adv_opts.gc)!="0": -gc $adv_opts.gc #end if $adv_opts.gccalc #set $output_files_list = str($adv_opts.output_files).split(',') #if "gff" in $output_files_list: -gff #end if #if "html" in $output_files_list: -html #end if $adv_opts.slow_search $adv_opts.quick_search $adv_opts.rush_search $adv_opts.only_alus $adv_opts.is_only #else: ## Set defaults -gff ## End of advanced options: #end if $query /dev/null 2> /dev/null; ## Copy the output files to galaxy #if $adv_opts.adv_opts_selector=="advanced": #if "summary" in $output_files_list: ## Write out the summary file (default) #set $summary_file = $output_basename + '.tbl' #os.system("cp $summary_file $output_summary;") #end if #if "gff" in $output_files_list: ## Write out the gff file (default) #set $gff_file = $output_basename + '.out.gff' #os.system("cp $gff_file $output_gff"); #end if #if "html" in $output_files_list: ## Write out the html file #set $html_file = $output_basename + '.out.html' #os.system("cp $html_file $output_html;") #end if #else:
## Write out the summary file (default) #set $summary_file = $output_basename + '.tbl' #os.system("cp $summary_file $output_summary;") ## Write out the gff file (default) #set $gff_file = $output_basename + '.out.gff' #os.system("cp $gff_file $output_gff;")## End of advanced options: #end if ## Write out mask sequence file #set $mask_sequence_file = $output_basename + '.masked' #os.system("cp $mask_sequence_file $output_mask;") ## Write out standard file (default) ## The default '.out' file from RepeatMasker has a 3-line header and spaces rather ## than tabs. Remove the header and replace the whitespaces with tab #set $standard_file = $output_basename + '.out' #os.system("tail -n +4 $standard_file | tr -s ' ' '\t' > $output_std;") ## Delete all temporary files #os.system("rm $dirname -r;"
- For the tool FilterControl, our cluster is configured to kill jobs that use more than 4Go of memory.
I don't achieve to modify qsub options in my Galaxy instance, so I'he changed this option -Xmx6g in "-Xmx4g". Maybe some treatment won't run by lake of memory .. If you have any idea on how to add option on Galaxy qsub, could you please help me ? I would like to add these options to qsub : qsub -l mem=6G -l h_vmem=8G
- ChIPMunk :Sorry, but all ChIPMunk files (xml, pl; sh) didn't have execution rights... So I just do "chmod a+x" on these files and ChIPMunk tool is OK in my Galaxy instance....
Thanks a lot for all your explanations, Alban, Marie-Stephane and Bjoern .
Thanks in advance for qsub and IntersectBed tool (test file),
Sarah
alermine a écrit :
Hi Sarah,
I'll try to debug, point per point:
- Here is the error I get when running the tool FilterControl
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 ***
I think here you have a misconfiguration of the memory of your java install (according to the need of the tool)
If you look at the FilterControlPeaks.sh file, the java is called with the option -Xmx6g. So your java install have to be allowed to use 6G as memory (by default it's 1024M)
- Here is the error I get when running the tool IntersectBed
/work/galaxy/database/pbs/galaxy_4129.sh: line 13: Erreur de syntaxe près du symbole inattendu « ;; » /work/galaxy/database/pbs/galaxy_4129.sh: line 13: `bedtools intersect -f 0.05
I don't understand this one, The intersectBed tool is only composed of a xml which simply call bedtools..
Check the command by typing 'bedtools intersect' in a terminal, if it sends you back the option list, the Galaxy tool should work.. If not, you have to check your bedtools install
- Here is the error I get when running the tool ChIPmunk (thanks Alban
for you help (ChIPMunk v2 is used and all librairies are found) but here is a new error, maybe a problem of "dirname" ?)
mv: impossible d'évaluer « /work/galaxy/database/job_working_directory/004/4132/galaxy_dataset_5992.dat_0.xml.png »: Aucun fichier ou dossier de ce type mv: impossible d'évaluer « /work/gala
Here, I think that the error is due to the ChIPMunk execution which doesn't work.
You can try the ChIPMunk execution this way:
Find the line containing:
ruby $CHIPMUNK_BIN_DIR/run_chiphorde.rb $LOG $motifs $MODE yes 1.0 s:$inputfile 100 10 1 4 >> /dev/null 2>> /dev/null
Add a line with:
echo "ruby $CHIPMUNK_BIN_DIR/run_chiphorde.rb $LOG $motifs $MODE yes 1.0 s:$inputfile 100 10 1 4 >> /dev/null 2>> /dev/null" > ~/ChIPMunk.log
Then try to launch the command in ~/ChIPMunk.log in a terminal and look at the output.
If ChIPMunk has run, the error comes from next step (i.e. the parseChipmunkOutput.pl script).
Else, try to debug ChIPMunk, and then the tool will work.
- Here is the error I get when running the tool PeaksToBed
readline() on closed filehandle FILE at /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 124 (#1) (W closed) The filehandle you're reading from got itself closed sometime before now. Check your control flow.
Use of uninitialized value $fields[1] in concatenation (.) or string at
For this one, edit the PeaksToBed.pl script this way:
The 3 lines:
use strict; use warnings; use diagnostics;
becomes:
#use strict; #use warnings; #use diagnostics;
Galaxy doesn't really love to have the return of a script (even if it's not an error). Here the perl scripts gives you warnings that are understood as errors by Galaxy.
Hope It will help you, don't hesitate if you need more..
++,
Alban
-- Alban Lermine Unité 900: INSERM - Mines ParisTech - Institut Curie " Bioinformatics and Computational Systems Biology of Cancer" 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel: +33 (0) 1 56 24 69 84
Le 1 févr. 2013 à 14:41, Sarah Maman sarah.maman@toulouse.inra.fr a écrit :
Thanks Bjoern,
In your xml file, I 've specified the interpreter : <command> replace by <command interpreter="bash"> but the problem is not solved (";" is supported by our cluster) because the command is runned with an error ";;" (double ;) instead of ";" (simple ;) : `bash /my/path/to/RepeatMasker -parallel 8 -species fungi -dir /work/tmp/tmprRrJh2 -gff /my/path/to/galaxy/database/files/005/dataset_5979.dat > /dev/null 2> /dev/null; cp /my/path/to/work/tmp/tmprRrJh2/dataset_5979.dat.tbl /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5999.dat; cp /work/tmp/tmprRrJh2/dataset_5979.dat.out.gff None; cp /work/tmp/tmprRrJh2/dataset_5979.dat.masked None; tail -n +4 /work/tmp/tmprRrJh2/dataset_5979.dat.out | tr -s ' ' '\t' > /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5998.dat; rm /work/tmp/tmprRrJh2 -r;; cd /my/path/to/galaxy/galaxy-dist; /my/path/to/galaxy/galaxy-dist/set_metadata.sh /my/path/to/galaxy/database/files /my/path/to/galaxy/database/job_working_directory/004/4135 . /my/path/to/galaxy/galaxy-dist/universe_wsgi.ini /work/tmp/tmpi0u8TZ /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy.json /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_in_HistoryDatasetAssociation_6156_Lp_2X3, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_kwds_HistoryDatasetAssociation_6156_emJ62i, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_out_HistoryDatasetAssociation_6156_Dgw3la, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_results_HistoryDatasetAssociation_6156_aTCjOU, /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5999.dat, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_override_HistoryDatasetAssociation_6156_I6enyB /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_in_HistoryDatasetAssociation_6155_LRm2gT, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_kwds_HistoryDatasetAssociation_6155_4xAztH, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_out_HistoryDatasetAssociation_6155_14AYvT, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_results_HistoryDatasetAssociation_6155_l_1gJ2, /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5998.dat, /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_override_HistoryDatasetAssociation_6155_NufEns'
So the error is ";;" rm /work/tmp/tmprRrJh2 -r;; instead of ";" rm /work/tmp/tmprRrJh2 -r; I've tried to replace ";" by nothing but this generate others errors (files not generated) : cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.tbl »: Aucun fichier ou dossier de ce type cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.out.gff »: Aucun fichier ou dossier de ce type cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.masked »: Aucun fichier ou dossier de ce type tail: impossible d'ouvrir « /work/tmp/tmpga_QOo/dataset_5979.dat.out » en lecture: Aucun fichier ou dossier de ce type Do you have any idea ? Thanks in advance, Sarah
Björn Grüning a écrit :
Hi Sarah,
i looks like something is wrong with the PBS system. The errors for bedtools and repeatmasker are the same. I can't understand french but i guess that ";" are not supported by PBS or in your configuration? Repeatmasker and bedtools creating more than one shell command and concatenating them with ';'.
Hope that helps, Bjoern
Hello,
I've installed NEBULA tools and "RepeatMasker" tool on our local Galaxy instance.
For some tools (others work), I get the following errors:
- Here is the error I get when running the tool FilterControl
*** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 *** *** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 *** *** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 *** *** glibc detected *** java: double free or corruption (!prev): 0x00007fe56800ecd0 *** /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/FilterControlPeaks/FilterControlPeaks.sh: line 80: 61310 Abandon java -classpath $LOCAL_PATH/ -Xmx6g FilterPeaks -f $inputfile -c $controlfile -t $minHeight -v $minRatio -o $output > /dev/null *** glibc detected *** java: double free or corruption (!prev): 0x00007f1f5800ecd0 *** /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/FilterControlPeaks/FilterControlPeaks.sh: line 81: 61351 Abandon
- Here is the error I get when running the tool IntersectBed
/work/galaxy/database/pbs/galaxy_4129.sh: line 13: Erreur de syntaxe près du symbole inattendu « ;; » /work/galaxy/database/pbs/galaxy_4129.sh: line 13: `bedtools intersect -f 0.05
- Here is the error I get when running the tool ChIPmunk (thanks Alban
for you help (ChIPMunk v2 is used and all librairies are found) but here is a new error, maybe a problem of "dirname" ?)
mv: impossible d'évaluer « /work/galaxy/database/job_working_directory/004/4132/galaxy_dataset_5992.dat_0.xml.png »: Aucun fichier ou dossier de ce type mv: impossible d'évaluer « /work/gala
- Here is the error I get when running the tool PeaksToBed
readline() on closed filehandle FILE at /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 124 (#1) (W closed) The filehandle you're reading from got itself closed sometime before now. Check your control flow.
Use of uninitialized value $fields[1] in concatenation (.) or string at /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 154, <FILE> line 1 (#2) (W uninitialized) An undefined value was used as if it were already defined. It was interpreted as a "" or a 0, but maybe it was a mistake. To suppress this warning assign a defined value to your variables.
To help you figure out what was undefined, perl will try to tell you the name of the variable (if any) that was undefined. In some cases it cannot do this, so it also tells you what operation you used the undefined value in. Note, however, that perl optimizes your program and the operation displayed in the warning may not necessarily appear literally in your program. For example, "that $foo" is usually optimized into "that " . $foo, and the warning will refer to the concatenation (.) operator, even though there is no . in your program.Use of uninitialized value in concatenation (.) or string at /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 154, <FILE> line 1 (#2) Use of uninitialized value in numeric ge (>=) at /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 156, <FILE> line 1 (#2) ...... /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl line 156, <FILE> line 63 (#2)
Use of uninitialized value $fields[1] in conc
I dowloaded tool "RepeatMasker" from Galaxy Tool Shed,and RepeatMasker is installed on our cluster. Here is the error I get when running the tool RepeatMasker:
An error occurred running this job: Epilog : job finished at jeu. janv. 31 14:39:16 CET 2013 /work/galaxy/database/pbs/galaxy_4134.sh: line 13: Erreur de syntaxe près du symbole inattendu « ;; » /work/galaxy/database/pbs/galaxy_4134.sh: line 13: `RepeatMasker -parallel 8 -speci
Do you have any ideas ? Thank you in advance for your help, Sarah Maman -- --*-- Sarah Maman INRA - LGC - SIGENAE http://www.sigenae.org/ Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53 --*--
-- --*-- Sarah Maman INRA - LGC - SIGENAE http://www.sigenae.org/ Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53 --*--
-- --*-- Sarah Maman INRA - LGC - SIGENAE http://www.sigenae.org/ Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53 --*--