Updating BLAST+ wrapper
Hi all, The NCBI have just released BLAST 2.2.25+ which includes some interesting new stuff of interest to the tabular output, i.e. "Added support for query and subject length to tabular output" I would therefore like to update my BLAST+ wrappers in Galaxy to add these two columns to the 'extended tabular' output option. They are going to be very helpful as you can now calculate things like the percentage identity (or similarity) compared to the query or sequence. If there are no objections I propose to add these as columns 23 and 24 (i.e. at the end), so minimise disruption to anyone already using the tabular output. I would hope to have a branch ready for merging next week... Peter
On Thu, Mar 31, 2011 at 6:28 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,
The NCBI have just released BLAST 2.2.25+ which includes some interesting new stuff of interest to the tabular output, i.e.
"Added support for query and subject length to tabular output"
I would therefore like to update my BLAST+ wrappers in Galaxy to add these two columns to the 'extended tabular' output option. They are going to be very helpful as you can now calculate things like the percentage identity (or similarity) compared to the query or sequence.
If there are no objections I propose to add these as columns 23 and 24 (i.e. at the end), so minimise disruption to anyone already using the tabular output. I would hope to have a branch ready for merging next week...
One reason why I want to move to BLAST 2.2.25+ on our local machine is they fixed the subject IDs in tabular output, so that I can now use BLAST to compare two FASTA files against each other for a reciprocal best hits (RBH) search. I've just pushed the first draft of this to my bit-bucket, https://bitbucket.org/peterjc/galaxy-central/changeset/198bf927ca30 The idea behind this tool is to allow RBH to be done using any suitable tabular input - including but not limited to the BLAST tabular output (which can also be mimicked by other tools such as Bill Pearson's FASTA). I'm hoping this will be accepted into Galaxy later on (it is not ready yet), but I will need some help with this issue first: http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-March/004799.html https://bitbucket.org/galaxy/galaxy-central/issue/507/ Thanks, Peter
On Fri, Apr 1, 2011 at 12:01 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Mar 31, 2011 at 6:28 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,
The NCBI have just released BLAST 2.2.25+ which includes some interesting new stuff of interest to the tabular output, ...
One reason why I want to move to BLAST 2.2.25+ on our local machine is they fixed the subject IDs in tabular output, ...
Hi Kanwei, Could you transplant or merge this commit please? https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec It updates the test output files to work with BLAST 2.2.25+, which I will be using locally due to several important bug fixes. As you don't offer BLAST+ on the Penn State server I don't expect this to cause you any problems. Also (and unrelated to this change), do you see this from the test suite? $ ./run_functional_tests.sh -id ncbi_blastp_wrapper ... ====================================================================== ERROR: NCBI BLAST+ blastp ( ncbi_blastp_wrapper ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 155, in test_tool self.do_it( td ) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 71, in do_it page_inputs = self.__expand_grouping(testdef.tool.inputs_by_page[0], all_inputs) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 122, in __expand_grouping elif isinstance(declared_inputs[value.name], str): KeyError: 'blast_type' Thanks, Peter
On Mon, Apr 4, 2011 at 12:19 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Fri, Apr 1, 2011 at 12:01 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Mar 31, 2011 at 6:28 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,
The NCBI have just released BLAST 2.2.25+ which includes some interesting new stuff of interest to the tabular output, ...
One reason why I want to move to BLAST 2.2.25+ on our local machine is they fixed the subject IDs in tabular output, ...
Hi Kanwei,
Could you transplant or merge this commit please? https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec
It updates the test output files to work with BLAST 2.2.25+, which I will be using locally due to several important bug fixes. As you don't offer BLAST+ on the Penn State server I don't expect this to cause you any problems.
Also (and unrelated to this change), do you see this from the test suite?
$ ./run_functional_tests.sh -id ncbi_blastp_wrapper ... ====================================================================== ERROR: NCBI BLAST+ blastp ( ncbi_blastp_wrapper ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 155, in test_tool self.do_it( td ) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 71, in do_it page_inputs = self.__expand_grouping(testdef.tool.inputs_by_page[0], all_inputs) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 122, in __expand_grouping elif isinstance(declared_inputs[value.name], str): KeyError: 'blast_type'
Solved, it seems I can't omit any parameters in the <test> and expect the defaults to be used. This doesn't seem desirable (new bug?), but the quick fix is to add a few lines to the test. Could you commit this too: https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5 Thanks, Peter
Hi Peter, I think you need to merge your commits as I cannot seem to transplant or import the patches properly. K On Mon, Apr 4, 2011 at 9:46 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Apr 4, 2011 at 12:19 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Fri, Apr 1, 2011 at 12:01 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Mar 31, 2011 at 6:28 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,
The NCBI have just released BLAST 2.2.25+ which includes some interesting new stuff of interest to the tabular output, ...
One reason why I want to move to BLAST 2.2.25+ on our local machine is they fixed the subject IDs in tabular output, ...
Hi Kanwei,
Could you transplant or merge this commit please? https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec
It updates the test output files to work with BLAST 2.2.25+, which I will be using locally due to several important bug fixes. As you don't offer BLAST+ on the Penn State server I don't expect this to cause you any problems.
Also (and unrelated to this change), do you see this from the test suite?
$ ./run_functional_tests.sh -id ncbi_blastp_wrapper ... ====================================================================== ERROR: NCBI BLAST+ blastp ( ncbi_blastp_wrapper ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 155, in test_tool self.do_it( td ) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 71, in do_it page_inputs = self.__expand_grouping(testdef.tool.inputs_by_page[0], all_inputs) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 122, in __expand_grouping elif isinstance(declared_inputs[value.name], str): KeyError: 'blast_type'
Solved, it seems I can't omit any parameters in the <test> and expect the defaults to be used. This doesn't seem desirable (new bug?), but the quick fix is to add a few lines to the test. Could you commit this too:
https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5
Thanks,
Peter
On Mon, Apr 4, 2011 at 3:07 PM, Kanwei Li <kanwei@gmail.com> wrote:
Hi Peter,
I think you need to merge your commits as I cannot seem to transplant or import the patches properly.
K
Hi Kanwei, I'm not sure why they won't transplant - they are both on this fresh branch from the trunk, with just these two commits so far: https://bitbucket.org/peterjc/galaxy-central/src/blast25 Can you just merge that "blast25" branch? Alternatively, do the raw changesets help? First: https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec/raw/gala... Then: https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5/raw/gala... Peter
I tried both ways, no dice (you can try yourself with a clean galaxy-central base) On Mon, Apr 4, 2011 at 10:58 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Apr 4, 2011 at 3:07 PM, Kanwei Li <kanwei@gmail.com> wrote:
Hi Peter,
I think you need to merge your commits as I cannot seem to transplant or import the patches properly.
K
Hi Kanwei,
I'm not sure why they won't transplant - they are both on this fresh branch from the trunk, with just these two commits so far:
https://bitbucket.org/peterjc/galaxy-central/src/blast25
Can you just merge that "blast25" branch?
Alternatively, do the raw changesets help? First:
https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec/raw/gala...
Then:
https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5/raw/gala...
Peter
On Mon, Apr 4, 2011 at 4:03 PM, Kanwei Li <kanwei@gmail.com> wrote:
I tried both ways, no dice (you can try yourself with a clean galaxy-central base)
I'm a bit puzzled and not an hg expert. Was there an error message? And for plan (B), what went wrong with using patch and the raw changes? Surely that should work? https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec/raw/gala... https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5/raw/gala... Peter
I think it's because the parent of those changesets were before your merge, so the branches got messed up On Mon, Apr 4, 2011 at 11:07 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Apr 4, 2011 at 4:03 PM, Kanwei Li <kanwei@gmail.com> wrote:
I tried both ways, no dice (you can try yourself with a clean galaxy-central base)
I'm a bit puzzled and not an hg expert. Was there an error message?
And for plan (B), what went wrong with using patch and the raw changes? Surely that should work? https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec/raw/gala... https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5/raw/gala...
Peter
On Mon, Apr 4, 2011 at 4:10 PM, Kanwei Li <kanwei@gmail.com> wrote:
I think it's because the parent of those changesets were before your merge, so the branches got messed up
But I haven't done any merge in this branch's history... It starts here, with James' latest commit to the trunk: https://bitbucket.org/galaxy/galaxy-central/changeset/086c9c2c52b9 https://bitbucket.org/peterjc/galaxy-central/changeset/086c9c2c52b9 Then I update the test files, https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec Then I update missing arguments in one test: https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5 This seems to be confirmed by hg log, $ hg log --branch blast25 changeset: 5586:124e4556f3c5 branch: blast25 tag: tip user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 14:44:52 2011 +0100 summary: Include optional parameters in BLASTP test to avoid KeyError changeset: 5585:ab40f95393ec branch: blast25 parent: 5336:086c9c2c52b9 user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 12:12:11 2011 +0100 summary: Update tests for NCBI BLAST 2.2.25+ which fixed subject ID in tabular output Do you definitely have the latest code from galaxy-central in your repository? If you local copy is out of date that might explain things. Peter
dhcp243253:galaxy-central kanwei$ hg transplant -s ~/peter/galaxy-central/ -b blast25 searching for changes changeset: 5585:ab40f95393ec branch: blast25 parent: 5583:086c9c2c52b9 user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 12:12:11 2011 +0100 summary: Update tests for NCBI BLAST 2.2.25+ which fixed subject ID in tabular output apply changeset? [ynmpcq?]: y changeset: 5586:124e4556f3c5 branch: blast25 user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 14:44:52 2011 +0100 summary: Include optional parameters in BLASTP test to avoid KeyError apply changeset? [ynmpcq?]: y searching for changes adding changesets adding manifests adding file changes added 2 changesets with 4 changes to 4 files dhcp243253:galaxy-central kanwei$ hg tip changeset: 5338:124e4556f3c5 branch: blast25 tag: tip user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 14:44:52 2011 +0100 summary: Include optional parameters in BLASTP test to avoid KeyError dhcp243253:galaxy-central kanwei$ hg push pushing to https://kanwei@bitbucket.org/galaxy/galaxy-central/ searching for changes abort: push creates new remote branches: blast25! (use 'hg push --new-branch' to create new remote branches) dhcp243253:galaxy-central kanwei$ hg merge blast25 abort: merging with a working directory ancestor has no effect Importing the diffs gives merge error On Mon, Apr 4, 2011 at 11:23 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Apr 4, 2011 at 4:10 PM, Kanwei Li <kanwei@gmail.com> wrote:
I think it's because the parent of those changesets were before your merge, so the branches got messed up
But I haven't done any merge in this branch's history... It starts here, with James' latest commit to the trunk:
https://bitbucket.org/galaxy/galaxy-central/changeset/086c9c2c52b9 https://bitbucket.org/peterjc/galaxy-central/changeset/086c9c2c52b9
Then I update the test files,
https://bitbucket.org/peterjc/galaxy-central/changeset/ab40f95393ec
Then I update missing arguments in one test:
https://bitbucket.org/peterjc/galaxy-central/changeset/124e4556f3c5
This seems to be confirmed by hg log,
$ hg log --branch blast25 changeset: 5586:124e4556f3c5 branch: blast25 tag: tip user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 14:44:52 2011 +0100 summary: Include optional parameters in BLASTP test to avoid KeyError
changeset: 5585:ab40f95393ec branch: blast25 parent: 5336:086c9c2c52b9 user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 12:12:11 2011 +0100 summary: Update tests for NCBI BLAST 2.2.25+ which fixed subject ID in tabular output
Do you definitely have the latest code from galaxy-central in your repository? If you local copy is out of date that might explain things.
Peter
On Mon, Apr 4, 2011 at 4:36 PM, Kanwei Li <kanwei@gmail.com> wrote:
dhcp243253:galaxy-central kanwei$ hg transplant -s ~/peter/galaxy-central/ -b blast25 searching for changes changeset: 5585:ab40f95393ec branch: blast25 parent: 5583:086c9c2c52b9 user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 12:12:11 2011 +0100 summary: Update tests for NCBI BLAST 2.2.25+ which fixed subject ID in tabular output
apply changeset? [ynmpcq?]: y changeset: 5586:124e4556f3c5 branch: blast25 user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 14:44:52 2011 +0100 summary: Include optional parameters in BLASTP test to avoid KeyError
apply changeset? [ynmpcq?]: y searching for changes adding changesets adding manifests adding file changes added 2 changesets with 4 changes to 4 files dhcp243253:galaxy-central kanwei$ hg tip changeset: 5338:124e4556f3c5 branch: blast25 tag: tip user: peterjc <p.j.a.cock@googlemail.com> date: Mon Apr 04 14:44:52 2011 +0100 summary: Include optional parameters in BLASTP test to avoid KeyError
The above is strange - why have you got a blast25 branch? Maybe the transplant isn't working as expected...
dhcp243253:galaxy-central kanwei$ hg push pushing to https://kanwei@bitbucket.org/galaxy/galaxy-central/ searching for changes abort: push creates new remote branches: blast25! (use 'hg push --new-branch' to create new remote branches) dhcp243253:galaxy-central kanwei$ hg merge blast25 abort: merging with a working directory ancestor has no effect
It looks like you are on the blast25 branch, try checking out default then merging the new blast25 branch. It should be a fast-forward merge (just the two commits). Then you can push default back to galaxy-central on bitbucket. I think.
Importing the diffs gives merge error
Are you using a hg command here? What I meant for the "Plan B" was just using the Unix command patch to apply my changes "by hand" to the default branch. Then commit it under your name. It won't be associated with my bitbucket account, but it should just work. Peter
On Mon, Apr 4, 2011 at 5:55 PM, Kanwei Li <kanwei@gmail.com> wrote:
Done the manual way
Thanks. I was hoping to have finished this today, but it turns out there are some subtle but annoying changes to how BLAST records the hit id/name/description in their XML file which make the script to convert from XML to tabular unhappy. I'm starting to think I should just deprecate my conversion tool, and instead add BLAST's ASN.1 output to Galaxy as a new format, and wrap the NCBI blast_formatter tool to allow the user convert this into XML, tabular, etc. Peter
On Mon, Apr 4, 2011 at 6:07 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Apr 4, 2011 at 5:55 PM, Kanwei Li <kanwei@gmail.com> wrote:
Done the manual way
Thanks.
I was hoping to have finished this today, but it turns out there are some subtle but annoying changes to how BLAST records the hit id/name/description in their XML file which make the script to convert from XML to tabular unhappy.
Hi Kanwei, I think I'm nearly done now, but I am still working on the blast25 branch you had trouble merging/transplanting from. I guess I will need to make a new clean branch from the galaxy tip for submission. Do you prefer several small atomic commits (which I understand is the recommend practice in git), or a few large commits (which seems quite common on the Galaxy hg history)? Thanks, Peter
participants (2)
-
Kanwei Li -
Peter Cock