Liftover tool installation woes
Hi Everyone, I installed a local instance of galaxy and some tools using the command line, and for some reason liftover will work in the command line but won't work in galaxy. Here's what I've done so far: - Downloaded liftover standalone executable from the tool-dependencies page in the Gwiki. - Changed permissions on liftover executable to make it usable in the command prompt (terminal, macosx) -> otherwise I get permission denied. - Copied executable over to /usr/local/bin likewise to use in command prompt and in Galaxy. - Modified liftover.loc to include all the conversion types and directories of chain.gz files I downloaded to use with the tool (I believe liftover can make use of both .chain and .chain.gz files). After that I tried to run liftover in Galaxy and it didn't work, giving me some error that said hg19ToMm9 mapping is currently not available. It seems to work fine using the command line though. Since my goal is to setup galaxy locally, I really want to get this working correctly. Can anyone guide me through what's gone wrong? Also, there must be a better way of installing all these tools than manually downloading every single one and using the command line, surely? I had to type out the format of liftover.loc file manually for around 130 chain files... I'm convinced there has to be a better way. Thanks in advance, SJ
Hello Kwon, Most of this sounds correct. The only issue is with compressed files. Galaxy uses uncompressed liftOver data at our site and this is how we instruct that it be set up (see galaxy-central/tool-data/liftOver.loc.sample). Would you be able to run a test to see if uncompressing resolves the issue? Uncompress one of the over.chain files, modify the .loc file so that the name also reflects the uncompressed naming, then try to use the UI version of liftOver with this uncompressed data. Please let us know how this works out (cc-ing the list, so that others can learn of the confirmed dependency). For now, there is no automated way to enter the .loc data entries. There is some development activity going on right now to address data setup (including liftOver), but nothing is ready yet to be shared. When complete, any new tools will be announced (loudly - we know that they are highly desired). Thanks! Jen Galaxy team On 7/3/12 10:21 PM, Kwon SoonJae wrote:
Hi Everyone,
I installed a local instance of galaxy and some tools using the command line, and for some reason liftover will work in the command line but won't work in galaxy. Here's what I've done so far:
- Downloaded liftover standalone executable from the tool-dependencies page in the Gwiki.
- Changed permissions on liftover executable to make it usable in the command prompt (terminal, macosx) -> otherwise I get permission denied.
- Copied executable over to /usr/local/bin likewise to use in command prompt and in Galaxy.
- Modified liftover.loc to include all the conversion types and directories of chain.gz files I downloaded to use with the tool (I believe liftover can make use of both .chain and .chain.gz files).
After that I tried to run liftover in Galaxy and it didn't work, giving me some error that said hg19ToMm9 mapping is currently not available. It seems to work fine using the command line though.
Since my goal is to setup galaxy locally, I really want to get this working correctly. Can anyone guide me through what's gone wrong? Also, there must be a better way of installing all these tools than manually downloading every single one and using the command line, surely? I had to type out the format of liftover.loc file manually for around 130 chain files... I'm convinced there has to be a better way.
Thanks in advance,
SJ
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
Hi Jen, Thanks for looking into this. I have unzipped all the .gz files so that the directory contains all .chain files. The liftover.loc file has also been modified to refer to all the .chain files instead, as such: http://pastebin.com/xSKGvUtG I still run into the same error if I try to run the tool in Galaxy, though (hg19ToMm9 mapping currently not available). The liftover command still works fine through the command line. I have a feeling the error has to do with the issue with Galaxy's job runner returning anything written to stderr as an error, mentioned here: http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr Seems like it was an issue back in 2010. Has this been rectified? Again, thank you for your help, SJ On Wed, Jul 4, 2012 at 9:42 PM, Jennifer Jackson <jen@bx.psu.edu> wrote:
Hello Kwon,
Most of this sounds correct. The only issue is with compressed files. Galaxy uses uncompressed liftOver data at our site and this is how we instruct that it be set up (see galaxy-central/tool-data/liftOver.loc.sample).
Would you be able to run a test to see if uncompressing resolves the issue? Uncompress one of the over.chain files, modify the .loc file so that the name also reflects the uncompressed naming, then try to use the UI version of liftOver with this uncompressed data. Please let us know how this works out (cc-ing the list, so that others can learn of the confirmed dependency).
For now, there is no automated way to enter the .loc data entries. There is some development activity going on right now to address data setup (including liftOver), but nothing is ready yet to be shared. When complete, any new tools will be announced (loudly - we know that they are highly desired).
Thanks!
Jen Galaxy team
On 7/3/12 10:21 PM, Kwon SoonJae wrote:
Hi Everyone,
I installed a local instance of galaxy and some tools using the command line, and for some reason liftover will work in the command line but won't work in galaxy. Here's what I've done so far:
- Downloaded liftover standalone executable from the tool-dependencies page in the Gwiki.
- Changed permissions on liftover executable to make it usable in the command prompt (terminal, macosx) -> otherwise I get permission denied.
- Copied executable over to /usr/local/bin likewise to use in command prompt and in Galaxy.
- Modified liftover.loc to include all the conversion types and directories of chain.gz files I downloaded to use with the tool (I believe liftover can make use of both .chain and .chain.gz files).
After that I tried to run liftover in Galaxy and it didn't work, giving me some error that said hg19ToMm9 mapping is currently not available. It seems to work fine using the command line though.
Since my goal is to setup galaxy locally, I really want to get this working correctly. Can anyone guide me through what's gone wrong? Also, there must be a better way of installing all these tools than manually downloading every single one and using the command line, surely? I had to type out the format of liftover.loc file manually for around 130 chain files... I'm convinced there has to be a better way.
Thanks in advance,
SJ
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jacksonhttp://galaxyproject.org
Hello Kwon, There are no known issues with the liftOver wrapper (tools / extract / liftOver_wrapper.py and liftOver_wrapper.xml). Three more troubleshooting ideas: 1 - This may sound simple, but have you restarted the database since you updated the .loc files? http://wiki.g2.bx.psu.edu/Admin/Data%20Integration (final step) 2 - Remove the relative path from the .loc file entries (the "~/")) and replace with an explicit full path. I personally have not seen this used before in .loc files. Perhaps others on the list can comment about whether they have been able to use this successfully and if any special configuration was necessary. 3- Confirm that the genomes are in your builds list (also covered in wiki above). Note that UCSC "dbkey" values have special capitalization rules depending on where they are used. Your example .loc file has this correct, but I recommend that you also check your builds list entries against the wiki examples and your actual data. If you make any changes, be sure to restart before testing. For example, the liftOver file pair: hg19ToDanRer6.over.chain & danRer6ToHg19.over.chain human dbkey = hg19 zebrafish dbkey = danRer6 Please let us know how this works out, Jen Galaxy team On 7/4/12 7:35 PM, Kwon SoonJae wrote:
Hi Jen,
Thanks for looking into this. I have unzipped all the .gz files so that the directory contains all .chain files. The liftover.loc file has also been modified to refer to all the .chain files instead, as such: http://pastebin.com/xSKGvUtG
I still run into the same error if I try to run the tool in Galaxy, though (hg19ToMm9 mapping currently not available). The liftover command still works fine through the command line.
I have a feeling the error has to do with the issue with Galaxy's job runner returning anything written to stderr as an error, mentioned here: http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr
Seems like it was an issue back in 2010. Has this been rectified?
Again, thank you for your help,
SJ
On Wed, Jul 4, 2012 at 9:42 PM, Jennifer Jackson <jen@bx.psu.edu <mailto:jen@bx.psu.edu>> wrote:
Hello Kwon,
Most of this sounds correct. The only issue is with compressed files. Galaxy uses uncompressed liftOver data at our site and this is how we instruct that it be set up (see galaxy-central/tool-data/liftOver.loc.sample).
Would you be able to run a test to see if uncompressing resolves the issue? Uncompress one of the over.chain files, modify the .loc file so that the name also reflects the uncompressed naming, then try to use the UI version of liftOver with this uncompressed data. Please let us know how this works out (cc-ing the list, so that others can learn of the confirmed dependency).
For now, there is no automated way to enter the .loc data entries. There is some development activity going on right now to address data setup (including liftOver), but nothing is ready yet to be shared. When complete, any new tools will be announced (loudly - we know that they are highly desired).
Thanks!
Jen Galaxy team
On 7/3/12 10:21 PM, Kwon SoonJae wrote:
Hi Everyone,
I installed a local instance of galaxy and some tools using the command line, and for some reason liftover will work in the command line but won't work in galaxy. Here's what I've done so far:
- Downloaded liftover standalone executable from the tool-dependencies page in the Gwiki.
- Changed permissions on liftover executable to make it usable in the command prompt (terminal, macosx) -> otherwise I get permission denied.
- Copied executable over to /usr/local/bin likewise to use in command prompt and in Galaxy.
- Modified liftover.loc to include all the conversion types and directories of chain.gz files I downloaded to use with the tool (I believe liftover can make use of both .chain and .chain.gz files).
After that I tried to run liftover in Galaxy and it didn't work, giving me some error that said hg19ToMm9 mapping is currently not available. It seems to work fine using the command line though.
Since my goal is to setup galaxy locally, I really want to get this working correctly. Can anyone guide me through what's gone wrong? Also, there must be a better way of installing all these tools than manually downloading every single one and using the command line, surely? I had to type out the format of liftover.loc file manually for around 130 chain files... I'm convinced there has to be a better way.
Thanks in advance,
SJ
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
-- Jennifer Jackson http://galaxyproject.org
Hi Jen! Definitely didn't forget to reset the server everytime I tried. The problem was solved when I fixed the relative path to the full path.. can't believe I was stuck forever because of such a small miss. Fixed! and everything working so far =) Thank you so much, SJ On Thu, Jul 5, 2012 at 4:43 PM, Jennifer Jackson <jen@bx.psu.edu> wrote:
Hello Kwon,
There are no known issues with the liftOver wrapper (tools / extract / liftOver_wrapper.py and liftOver_wrapper.xml).
Three more troubleshooting ideas:
1 - This may sound simple, but have you restarted the database since you updated the .loc files? http://wiki.g2.bx.psu.edu/Admin/Data%20Integration (final step)
2 - Remove the relative path from the .loc file entries (the "~/")) and replace with an explicit full path. I personally have not seen this used before in .loc files. Perhaps others on the list can comment about whether they have been able to use this successfully and if any special configuration was necessary.
3- Confirm that the genomes are in your builds list (also covered in wiki above). Note that UCSC "dbkey" values have special capitalization rules depending on where they are used. Your example .loc file has this correct, but I recommend that you also check your builds list entries against the wiki examples and your actual data. If you make any changes, be sure to restart before testing.
For example, the liftOver file pair: hg19ToDanRer6.over.chain & danRer6ToHg19.over.chain human dbkey = hg19 zebrafish dbkey = danRer6
Please let us know how this works out,
Jen Galaxy team
On 7/4/12 7:35 PM, Kwon SoonJae wrote:
Hi Jen,
Thanks for looking into this. I have unzipped all the .gz files so that the directory contains all .chain files. The liftover.loc file has also been modified to refer to all the .chain files instead, as such: http://pastebin.com/xSKGvUtG
I still run into the same error if I try to run the tool in Galaxy, though (hg19ToMm9 mapping currently not available). The liftover command still works fine through the command line.
I have a feeling the error has to do with the issue with Galaxy's job runner returning anything written to stderr as an error, mentioned here: http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr
Seems like it was an issue back in 2010. Has this been rectified?
Again, thank you for your help,
SJ
On Wed, Jul 4, 2012 at 9:42 PM, Jennifer Jackson <jen@bx.psu.edu> wrote:
Hello Kwon,
Most of this sounds correct. The only issue is with compressed files. Galaxy uses uncompressed liftOver data at our site and this is how we instruct that it be set up (see galaxy-central/tool-data/liftOver.loc.sample).
Would you be able to run a test to see if uncompressing resolves the issue? Uncompress one of the over.chain files, modify the .loc file so that the name also reflects the uncompressed naming, then try to use the UI version of liftOver with this uncompressed data. Please let us know how this works out (cc-ing the list, so that others can learn of the confirmed dependency).
For now, there is no automated way to enter the .loc data entries. There is some development activity going on right now to address data setup (including liftOver), but nothing is ready yet to be shared. When complete, any new tools will be announced (loudly - we know that they are highly desired).
Thanks!
Jen Galaxy team
On 7/3/12 10:21 PM, Kwon SoonJae wrote:
Hi Everyone,
I installed a local instance of galaxy and some tools using the command line, and for some reason liftover will work in the command line but won't work in galaxy. Here's what I've done so far:
- Downloaded liftover standalone executable from the tool-dependencies page in the Gwiki.
- Changed permissions on liftover executable to make it usable in the command prompt (terminal, macosx) -> otherwise I get permission denied.
- Copied executable over to /usr/local/bin likewise to use in command prompt and in Galaxy.
- Modified liftover.loc to include all the conversion types and directories of chain.gz files I downloaded to use with the tool (I believe liftover can make use of both .chain and .chain.gz files).
After that I tried to run liftover in Galaxy and it didn't work, giving me some error that said hg19ToMm9 mapping is currently not available. It seems to work fine using the command line though.
Since my goal is to setup galaxy locally, I really want to get this working correctly. Can anyone guide me through what's gone wrong? Also, there must be a better way of installing all these tools than manually downloading every single one and using the command line, surely? I had to type out the format of liftover.loc file manually for around 130 chain files... I'm convinced there has to be a better way.
Thanks in advance,
SJ
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jacksonhttp://galaxyproject.org
-- Jennifer Jacksonhttp://galaxyproject.org
Great news, very glad this worked. I'll add liftOver to our wiki with more specific instructions and add in the bit about explicit paths to the .loc file section as well. Should help others going forward. Thanks for taking the time to report back the solution to the list! Jen Galaxy team On 7/5/12 1:47 AM, Kwon SoonJae wrote:
Hi Jen!
Definitely didn't forget to reset the server everytime I tried. The problem was solved when I fixed the relative path to the full path.. can't believe I was stuck forever because of such a small miss. Fixed! and everything working so far =)
Thank you so much,
SJ
On Thu, Jul 5, 2012 at 4:43 PM, Jennifer Jackson <jen@bx.psu.edu <mailto:jen@bx.psu.edu>> wrote:
Hello Kwon,
There are no known issues with the liftOver wrapper (tools / extract / liftOver_wrapper.py and liftOver_wrapper.xml).
Three more troubleshooting ideas:
1 - This may sound simple, but have you restarted the database since you updated the .loc files? http://wiki.g2.bx.psu.edu/Admin/Data%20Integration (final step)
2 - Remove the relative path from the .loc file entries (the "~/")) and replace with an explicit full path. I personally have not seen this used before in .loc files. Perhaps others on the list can comment about whether they have been able to use this successfully and if any special configuration was necessary.
3- Confirm that the genomes are in your builds list (also covered in wiki above). Note that UCSC "dbkey" values have special capitalization rules depending on where they are used. Your example .loc file has this correct, but I recommend that you also check your builds list entries against the wiki examples and your actual data. If you make any changes, be sure to restart before testing.
For example, the liftOver file pair: hg19ToDanRer6.over.chain & danRer6ToHg19.over.chain human dbkey = hg19 zebrafish dbkey = danRer6
Please let us know how this works out,
Jen Galaxy team
On 7/4/12 7:35 PM, Kwon SoonJae wrote:
Hi Jen,
Thanks for looking into this. I have unzipped all the .gz files so that the directory contains all .chain files. The liftover.loc file has also been modified to refer to all the .chain files instead, as such: http://pastebin.com/xSKGvUtG
I still run into the same error if I try to run the tool in Galaxy, though (hg19ToMm9 mapping currently not available). The liftover command still works fine through the command line.
I have a feeling the error has to do with the issue with Galaxy's job runner returning anything written to stderr as an error, mentioned here: http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr
Seems like it was an issue back in 2010. Has this been rectified?
Again, thank you for your help,
SJ
On Wed, Jul 4, 2012 at 9:42 PM, Jennifer Jackson <jen@bx.psu.edu <mailto:jen@bx.psu.edu>> wrote:
Hello Kwon,
Most of this sounds correct. The only issue is with compressed files. Galaxy uses uncompressed liftOver data at our site and this is how we instruct that it be set up (see galaxy-central/tool-data/liftOver.loc.sample).
Would you be able to run a test to see if uncompressing resolves the issue? Uncompress one of the over.chain files, modify the .loc file so that the name also reflects the uncompressed naming, then try to use the UI version of liftOver with this uncompressed data. Please let us know how this works out (cc-ing the list, so that others can learn of the confirmed dependency).
For now, there is no automated way to enter the .loc data entries. There is some development activity going on right now to address data setup (including liftOver), but nothing is ready yet to be shared. When complete, any new tools will be announced (loudly - we know that they are highly desired).
Thanks!
Jen Galaxy team
On 7/3/12 10:21 PM, Kwon SoonJae wrote:
Hi Everyone,
I installed a local instance of galaxy and some tools using the command line, and for some reason liftover will work in the command line but won't work in galaxy. Here's what I've done so far:
- Downloaded liftover standalone executable from the tool-dependencies page in the Gwiki.
- Changed permissions on liftover executable to make it usable in the command prompt (terminal, macosx) -> otherwise I get permission denied.
- Copied executable over to /usr/local/bin likewise to use in command prompt and in Galaxy.
- Modified liftover.loc to include all the conversion types and directories of chain.gz files I downloaded to use with the tool (I believe liftover can make use of both .chain and .chain.gz files).
After that I tried to run liftover in Galaxy and it didn't work, giving me some error that said hg19ToMm9 mapping is currently not available. It seems to work fine using the command line though.
Since my goal is to setup galaxy locally, I really want to get this working correctly. Can anyone guide me through what's gone wrong? Also, there must be a better way of installing all these tools than manually downloading every single one and using the command line, surely? I had to type out the format of liftover.loc file manually for around 130 chain files... I'm convinced there has to be a better way.
Thanks in advance,
SJ
___________________________________________________________ Please keep all replies on the list by using"reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
-- Jennifer Jackson http://galaxyproject.org
-- Jennifer Jackson http://galaxyproject.org
participants (2)
-
Jennifer Jackson -
Kwon SoonJae