I would think the correct approach would be to allow any bam file and if a tool requires a coordinate sorted bam and one is not provided it would make sense for the tool to fail with an error.

Does galaxy itself require a coordinate sorted bam or this this so the majority of tools can function properly?

Sent from my iPhone

On Jul 17, 2015, at 11:08 AM, John Chilton jmchilton@gmail.com wrote:

I'm not the right person to respond to this but since no one else has I will explain my limited (probably incorrect) understanding of the problem and what needs to be done. I believe Galaxy assumes all bam files are coordinated sorted by default - so Galaxy's bam datatype would be better thought of as a coordinate-sorted bam. Certain people do not want to allow non-coordinate sorted bam files because these would be allowed by Galaxy as valid as inputs to tools that expect sorted bams - and things would break. For the same reason I imagine these people would argue the bai file needs to be generated because certain tools will depend on its existence.

I believe the proposed solution is to add an unsorted bam datatype to Galaxy. I don't know if there is any work in progress on this but it seems to be needed by many researchers.

Hope this clarifies things somewhat, -John

...

On Mon, Jun 29, 2015 at 11:06 AM, julie dubois dubjulie@gmail.com wrote: Hi all,

My galaxy is the last update and I work with samtools 1.2. I have a bug with the sorting and the upload of bam file. This bug appears only when I try to upload bam sorted by name or when I try to sort by name. When I work with bam sorted by coordinate, there is no problem.

So, when I try to sort a bam file by name, this error appears under the "bug icon" :

Tool execution generated the following error message: [bam_sort_core] merging from 20 files... *** Error in `python': double free or corruption (!prev): 0x0000000002897990 *** Aborted (core dumped)

The tool produced the following additional output:

[E::hts_idx_push] chromosome blocks not continuous Galaxy attempted to build the BAM index with samtools 1.0+ but failed: [Errno 2] No such file or directory: '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai'

And the same type of error appears when I try to upload bam sorted by name.

What I understand is that the bam can be sorted properly (The command line to sort is ok in a terminal with no bug and I can follow this step in the job_working_directory during the run) but Galaxy seems to fail to create a bai.

For our analysis pipeline, I think the bai is not crucial, is it possible to avoid this creation of bai during uploading and sorting of bam file by name ?

Is there another problem than this creation of bai ?

Thanks.

Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

We also encountered this issue. As I understood it: if Galaxy gets a BAM file (either through upload or generated by a tool), it will generate metadata and indexing is part of that. This only works if the BAM file is coordinate sorted, otherwise it fails (although there is nothing wrong with the BAM file).

As a lot of tools (and visualisation) require sorted BAM files, this avoids a lot of problems for users (but does create this issue).

An ‘unsorted bam’ (or namesorted) file type could be a solution although this would cause the illogic situation where a ‘bam' is hierarchically not above ‘unsorted bam’ as is the case for e.g. the types of fastq. I guess changing what is now ‘bam’ to ‘sorted bam’ will cause problems for a lot of users...

Frederik

On 17 Jul 2015, at 21:29, Alexander Vowinkel vowinkel.alexander@gmail.com wrote:

Indeed, this would be the right approach. But so far the current strategy is quite imputed also in tools. E.g. BWA from devteam automatically sorts the bam afterwards - wanted or not.

2015-07-17 11:59 GMT-05:00 Ryan G <ngsbioinformatics@gmail.com mailto:ngsbioinformatics@gmail.com>: I would think the correct approach would be to allow any bam file and if a tool requires a coordinate sorted bam and one is not provided it would make sense for the tool to fail with an error.

Does galaxy itself require a coordinate sorted bam or this this so the majority of tools can function properly?

Sent from my iPhone

...

On Jul 17, 2015, at 11:08 AM, John Chilton <jmchilton@gmail.com mailto:jmchilton@gmail.com> wrote:

I'm not the right person to respond to this but since no one else has I will explain my limited (probably incorrect) understanding of the problem and what needs to be done. I believe Galaxy assumes all bam files are coordinated sorted by default - so Galaxy's bam datatype would be better thought of as a coordinate-sorted bam. Certain people do not want to allow non-coordinate sorted bam files because these would be allowed by Galaxy as valid as inputs to tools that expect sorted bams - and things would break. For the same reason I imagine these people would argue the bai file needs to be generated because certain tools will depend on its existence.

I believe the proposed solution is to add an unsorted bam datatype to Galaxy. I don't know if there is any work in progress on this but it seems to be needed by many researchers.

Hope this clarifies things somewhat, -John

...

On Mon, Jun 29, 2015 at 11:06 AM, julie dubois <dubjulie@gmail.com mailto:dubjulie@gmail.com> wrote: Hi all,

My galaxy is the last update and I work with samtools 1.2. I have a bug with the sorting and the upload of bam file. This bug appears only when I try to upload bam sorted by name or when I try to sort by name. When I work with bam sorted by coordinate, there is no problem.

So, when I try to sort a bam file by name, this error appears under the "bug icon" :

Tool execution generated the following error message: [bam_sort_core] merging from 20 files... *** Error in `python': double free or corruption (!prev): 0x0000000002897990 *** Aborted (core dumped)

The tool produced the following additional output:

[E::hts_idx_push] chromosome blocks not continuous Galaxy attempted to build the BAM index with samtools 1.0+ but failed: [Errno 2] No such file or directory: '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai'

And the same type of error appears when I try to upload bam sorted by name.

What I understand is that the bam can be sorted properly (The command line to sort is ok in a terminal with no bug and I can follow this step in the job_working_directory during the run) but Galaxy seems to fail to create a bai.

For our analysis pipeline, I think the bai is not crucial, is it possible to avoid this creation of bai during uploading and sorting of bam file by name ?

Is there another problem than this creation of bai ?

Thanks.

Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ http://galaxyproject.org/search/mailinglists/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/