- galaxy-user - lists.galaxyproject.org

Flagstat crashes in non-linear workflows with TORQUE
by Andrew Warren 31 Aug '11

31 Aug '11

>Sorry this is old but I tried recompiling torque after setting the >NCONNECTS to 20 and the issue's still there. > >But there's more: It doesn't affect only flagstat but other non-linear >workflows. One of the two jobs that are submitted when their "father" >stopped running triggers the same error: >PBS error 15033: No free connections > >And the best part is that it works fine sometimes. But when it crashes, >it's always the same job that crashes. > >Does anyone have a clue? > >Cheers, >L-A I was getting the same behavior as you on asynchronous workflows on a multicore computer that is acting as both head and compute node for the torque system. Even after recompiling with a higher NCONNECTS I was getting the same error. I suspect that this is due to galaxy opening up multiple connections to check the status of currently running jobs. Because there can be many status checks in an asynchronous workflow the pbs system is randomly busy depending on when the job submission comes in. To deal with this I modified the lib/galaxy/jobs/runners/pbs.py script to make multiple attempts at submitting in the following way: @@ -286,6 +286,12 @@ class PBSJobRunner( BaseJobRunner ): log.debug("(%s) submitting file %s" % ( galaxy_job_id, job_file ) ) log.debug("(%s) command is: %s" % ( galaxy_job_id, command_line ) ) job_id = pbs.pbs_submit(c, job_attrs, job_file, pbs_queue_name, None) + ##Modified to give ten tries for qsubbing a job + num_try=0 + while(not job_id and num_try<10): + job_id = pbs.pbs_submit(c, job_attrs, job_file, pbs_queue_name, None) + num_try+=1 + pbs.pbs_disconnect(c) # check to see if it submitted I haven't had any problems since. Cheers, Andrew >* Louise-Amélie Schmitt wrote:*>>* Hello everyone*>>**>>* I observed an issue when flagstat is incorporated in a workflow in which*>>* the BAM file it works on is also used by another program (generate*>>* pileup for instance) and is NOT the input dataset (generated by sam to*>>* bam within the workflow).*>>**>>* I tested it with the local job runner and with TORQUE (with the pbs*>>* scheduler and Maui).*>>**>>* - With the local job runner, it works just fine*>>**>>* - With TORQUE I get the following error message:*>>* pbs_submit failed, PBS error 15033: No free connections*>* Hi,*>**>* This can most likely be fixed by increasing the value of NCONNECTS in*>* the TORQUE source, in src/include/libpbs.h, and recompiling on your*>* TORQUE server. I haven't seen a problem after increasing the value to*>* 20.*>**>* --nate*>** >>* Surprisingly, other non-linear workflows work fine. I only observed this*>>* error with flagstat. Moreover, when flagstat is in a linear workflow, it*>>* works fine too. Ad if it is non-linear but the input dataset is the bam*>>* file flagstat works on, it works fine too.*>>**>>* Please find attached one of the test workflow where I found the error.*>>* The input dataset is a sam file.*>>**>>* Any clue?*>>**>>* Cheers,*>>* LA*>>* {*>>* "a_galaxy_workflow": "true",*>>* "annotation": "to see if it fails if not forked",*>>* "format-version": "0.1",*>>* "name": "test flagstat",*>>* "steps": {*>>* "0": {*>>* "annotation": "",*>>* "id": 0,*>>* "input_connections": {},*>>* "inputs": [*>>* {*>>* "description": "",*>>* "name": "Input Dataset"*>>* }*>>* ],*>>* "name": "Input dataset",*>>* "outputs": [],*>>* "position": {*>>* "left": 200,*>>* "top": 200*>>* },*>>* "tool_errors": null,*>>* "tool_id": null,*>>* "tool_state": "{\"name\": \"Input Dataset\"}",*>>* "tool_version": null,*>>* "type": "data_input",*>>* "user_outputs": []*>>* },*>>* "1": {*>>* "annotation": "",*>>* "id": 1,*>>* "input_connections": {*>>* "source|input1": {*>>* "id": 0,*>>* "output_name": "output"*>>* }*>>* },*>>* "inputs": [],*>>* "name": "SAM-to-BAM",*>>* "outputs": [*>>* {*>>* "name": "output1",*>>* "type": "bam"*>>* }*>>* ],*>>* "position": {*>>* "left": 274.5,*>>* "top": 307*>>* },*>>* "tool_errors": null,*>>* "tool_id": "sam_to_bam",*>>* "tool_state": "{\"source\": \"{\\\"index_source\\\": \\\"cached\\\", \\\"input1\\\": null, \\\"__current_case__\\\": 0}\", \"__page__\": 0}",*>>* "tool_version": "1.1.1",*>>* "type": "tool",*>>* "user_outputs": []*>>* },*>>* "2": {*>>* "annotation": "",*>>* "id": 2,*>>* "input_connections": {*>>* "input1": {*>>* "id": 1,*>>* "output_name": "output1"*>>* }*>>* },*>>* "inputs": [],*>>* "name": "flagstat",*>>* "outputs": [*>>* {*>>* "name": "output1",*>>* "type": "txt"*>>* }*>>* ],*>>* "position": {*>>* "left": 396.5,*>>* "top": 445*>>* },*>>* "tool_errors": null,*>>* "tool_id": "samtools_flagstat",*>>* "tool_state": "{\"__page__\": 0, \"input1\": \"null\"}",*>>* "tool_version": "1.0.0",*>>* "type": "tool",*>>* "user_outputs": []*>>* },*>>* "3": {*>>* "annotation": "",*>>* "id": 3,*>>* "input_connections": {*>>* "refOrHistory|input1": {*>>* "id": 1,*>>* "output_name": "output1"*>>* }*>>* },*>>* "inputs": [],*>>* "name": "Generate pileup",*>>* "outputs": [*>>* {*>>* "name": "output1",*>>* "type": "tabular"*>>* }*>>* ],*>>* "position": {*>>* "left": 519,*>>* "top": 340*>>* },*>>* "tool_errors": null,*>>* "tool_id": "sam_pileup",*>>* "tool_state": "{\"__page__\": 0, \"c\": \"{\\\"consensus\\\": \\\"no\\\", \\\"__current_case__\\\": 0}\", \"indels\": \"\\\"no\\\"\", \"refOrHistory\": \"{\\\"input1\\\": null, \\\"reference\\\": \\\"indexed\\\", \\\"__current_case__\\\": 0}\", \"lastCol\": \"\\\"no\\\"\", \"mapCap\": \"\\\"60\\\"\"}",*>>* "tool_version": "1.1.1",*>>* "type": "tool",*>>* "user_outputs": []*>>* }*>>* }*>>* }*

1 0

install tools on cluster
by Wei,Xintao 30 Aug '11

30 Aug '11

Hi, I am learning how to install and configure Galaxy on our computer cluster. Many third party tools, for example, bowtie, samtools, R and so on, have to be installed and configured in order to install galaxy. Do I have to convert the third party tools to MPI version in order to run the tools on a computer cluster? Or do I just install the "regular" version of the tools on the "shared" folder of our cluster? Thank you! Xintao

1 0

aaChanges
by David Joly 30 Aug '11

30 Aug '11

Hi everybody, Maybe a dumb question but does someone know if we can use the aaChanges tool on a custom genome? I have my SNP dataset and my gene dataset but I can only assign them to one of the genome build already in UCSC... I am using the web server of galaxy. Do I need to install my own local instance of Galaxy in order to add a custom genome build? Thanks, David

2 3

rna-seq mutation detection
by Richard Mark White 30 Aug '11

30 Aug '11

Hi, I am trying to look at differences between two RNA-seq samples to see if there are mutations in one of them relative to the other (i.e. not in comparison to a reference genome). Does anyone know of a way to do this within galaxy? Any help is appreciated! rich

2 3

Galaxy is Hiring
by Dave Clements 29 Aug '11

29 Aug '11

Hello all The Galaxy Project is growing and has open positions in both the Penn State and Emory groups (http://wiki.g2.bx.psu.edu/News/Galaxy%20is%20Hiring) *Penn State: System administrators/analysts* The Nekrutenko Lab <http://www.bx.psu.edu/%7Eanton/> at the Huck Institutes of Life Sciences <http://www.huck.psu.edu/> at Penn State <http://psu.edu/>is currently recruiting system analysts/administrators with experience in building and maintaining complex performance compute environments. The areas of immediate need include: - Storage balancing and tiered storage - Virtualization - Schedulers - Deployment of Galaxy instances and dependence management - Relational databases and query optimization - User management A minimum of 5 year experience with UNIX/Linux system administration is required. Applicants should submit a CV and list of references to jobs(a)galaxyproject.org. <http://bx.mathcs.emory.edu/joining/> *Emory: Software Engineers and Post-Docs* The Taylor Lab <http://bx.mathcs.emory.edu/> in the Biology<http://www.biology.emory.edu/>and Mathematics & Computer Science <http://www.mathcs.emory.edu/> at Emory University<http://emory.edu/>is looking for software engineers <http://bx.mathcs.emory.edu/joining/sw/> and postdoctoral scholars<http://bx.mathcs.emory.edu/joining/postdocs/>to work on the Galaxy project. We are seeking software engineers <http://bx.mathcs.emory.edu/joining/sw/>with expertise in distributed computing and systems programming, web-based visualization and visual analytics, informatics and data analysis and integration, and bioinformatics application areas such as re-sequencing, de novo assembly, metagenomics, transcriptome analysis and epigenetics. These are full time positions located in Atlanta, GA. See the official posting<http://bx.mathcs.emory.edu/joining/sw/>for full details. Postdoctoral applicants <http://bx.mathcs.emory.edu/joining/postdocs/>should have expertise in Bioinformatics and Computational Biology and research interests that complement but extend the lab's current interests<http://bx.mathcs.emory.edu/research/>: The Galaxy project; distributed and high-performance computing for data intensive science; vertebrate functional genomics; and genomics and epigenomic mechanisms of gene regulation, the role of transcription factors and chromatin structure in global gene expression, development, and differentiation. See the announcement<http://bx.mathcs.emory.edu/joining/postdocs/>for full details. If any of these positions describe you then please consider applying. Thanks, Dave C. -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://galaxyproject.org/wiki/

1 0

Cufflinks quartile normalization
by David Joly 27 Aug '11

27 Aug '11

Can someone help me understanding the quartile normalization in Cufflinks? I read different threads in which they reported that the FPKM values were inflated after normalization (-N) but most people didn't report their values so I don't know how big the inflation should be... In my case, the difference is huge! The FPKM values for the four first genes without normalization are in the range of [61 - 184] while after normalization, they are in the range of [2.4e+6 - 7.4e+6]. Even though this inflation does not seem to affect the calculation of the gene expression changes [ log (FPKM2/FPKM1) ], I'm wondering if something is wrong with my dataset. Is it was I should expect? Is it always better to use the normalization? Thanks, David

2 1

load data from galaxy to IGV
by Jennifer Jackson 27 Aug '11

27 Aug '11

Hi Jen, is anyway I can directly load the analyzed data from galaxy to IGV (Integrative genome viewer) without downloading data to my computer first? Thanks, tao

2 1

Error when running Generate Pileup
by Alison Gardner 26 Aug '11

26 Aug '11

Hello, I am aligning Illumina paired end sequence using BWA, then run SAM-to-BAM resulting in a 1.1 Gb file. But when I then generate pileup on this file column 4 (consensus bases) are all N's and the file is now 8.4 Gb. Can someone tell me what might be going on? Any help much appreciated. Thanks Alison Gardner Neurogenetics Research Program, SA Pathology (at Women's and Children's Hospital), Level 9, Clarence Rieger Bldg., 72 King William Road, North Adelaide, S.A., Australia, 5006

2 1

galaxy with cuffdiff output
by 姚晨 26 Aug '11

26 Aug '11

Dear all, I use cuffdiff in galaxy net version. The output are isoform.diff , gene.diff ...etc, but I want to find the differential expressed non-coding RNAs. How can I get these information? Thank you Chen Yao

3 2

line estimation for pileup generation
by Austin Paul 26 Aug '11

26 Aug '11

Hello, I am curious if the line estimation shown in the history window for pileup generation is at all accurate. I am using the pileup files to generate expression data from bwa mapping for looking at differential expression, but I am having some trouble understanding the line estimates. For example, for one pileup file, when I cut the reference id column and the number of hits column (columns 1 and 4), the number of lines in the cut file is about 25% that of the pileup file, and for another file it will be 5000%. How can the number of lines grow 50x when I am just cutting columns from the file? Shouldnt the line estimate be the same? Thanks, Austin

2 3