Hello,
I have a question which I have not seen specifically addressed in the online Galaxy wiki documentation about how to integrate tools (dependencies) into Galaxy. I have implemented a locally managed instance of Galaxy that my business is using with our cluster and now have a freshly installed and configured instance of Galaxy running. It is bare-bones right now and I did not use mercurial to sync any existing files/directory structures. I have seen the page on external tool dependencies ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies) needed for Galaxy, but I am somewhat unsure where to place the tools to utilize them as intended (other than through trial & error).
It appears that there are shell directories for the tools under ~/galaxy-dist/tools/ with basic wrapper scripts but without the corresponding executables (very few that I've noticed have the tools already in them). Is the intent to download the dependency tools and (building from source if necessary) take the binaries in those directories and copy them to their corresponding directory under ~/galaxy-dist/tools/? This seems to have worked with an error I first got when clipping a FASTQ file which reported that fastx_clipper was not a recognized command. So I downloaded the FASTX Toolkit, compiled the binaries, and copied only the binaries into the corresponding fastx tools directory. Would I do the same thing for TopHat and Cufflinks by taking all their binaries (combined) and copying them into ~/galaxy-dist/tools/ngs_rna/?
Even if that is the case though, I have occasionally gotten errors about tools missing in completely different directories. One was for the FASTQ Groomer. One user saw this error in their browser (which for now is the only way I know to figure out where tools are *expected* to be):
*File "/home/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py", line 141, in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##rgFastQC.py error - cannot find executable /home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc *
To fix this I downloaded the FastQC tar file from its webpage, unzipped it, and copied the fastqc binary/script to the "home/galaxy/galaxy -dist/tool-data/shared/jars/FastQC/*"* directory. I also had to mkdir FastQC/ under jars/ to place it there since it didn't already exist. Had I not been told the specific directory by the error I'm not sure how I would have intuitively known to place the binary there (unless I'm overlooking some critical documentation). And how do I know that other similar things are not missing which should be there? Can anyone shed some light on this please? Adding a brief page on the Galaxy wiki site under the Admin section about this would really help, even if it only showed an example for one or two specific tools.
Thanks, Josh Nielsen
Hello,
I have a question which I have not seen specifically addressed in the online Galaxy wiki documentation about how to integrate tools (dependencies) into Galaxy. I have implemented a locally managed instance of Galaxy that my business is using with our cluster and now have a freshly installed and configured instance of Galaxy running. It is bare-bones right now and I did not use mercurial to sync any existing files/directory structures. I have seen the page on external tool dependencies ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies) needed for Galaxy, but I am somewhat unsure where to place the tools to utilize them as intended (other than through trial & error).
It appears that there are shell directories for the tools under ~/galaxy-dist/tools/ with basic wrapper scripts but without the corresponding executables (very few that I've noticed have the tools already in them). Is the intent to download the dependency tools and (building from source if necessary) take the binaries in those directories and copy them to their corresponding directory under ~/galaxy-dist/tools/? This seems to have worked with an error I first got when clipping a FASTQ file which reported that fastx_clipper was not a recognized command. So I downloaded the FASTX Toolkit, compiled the binaries, and copied only the binaries into the corresponding fastx tools directory. Would I do the same thing for TopHat and Cufflinks by taking all their binaries (combined) and copying them into ~/galaxy-dist/tools/ngs_rna/?
Even if that is the case though, I have occasionally gotten errors about tools missing in completely different directories. One was for the FASTQ Groomer. One user saw this error in their browser (which for now is the only way I know to figure out where tools are *expected* to be):
*File "/home/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py", line 141, in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##rgFastQC.py error - cannot find executable /home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc *
To fix this I downloaded the FastQC tar file from its webpage, unzipped it, and copied the fastqc binary/script to the "home/galaxy/galaxy -dist/tool-data/shared/jars/FastQC/*"* directory. I also had to mkdir FastQC/ under jars/ to place it there since it didn't already exist. Had I not been told the specific directory by the error I'm not sure how I would have intuitively known to place the binary there (unless I'm overlooking some critical documentation). And how do I know that other similar things are not missing which should be there? Can anyone shed some light on this please? Adding a brief page on the Galaxy wiki site under the Admin section about this would really help, even if it only showed an example for one or two specific tools.
Thanks, Josh
On Dec 9, 2011, at 4:34 PM, Josh Nielsen wrote:
Hello,
I have a question which I have not seen specifically addressed in the online Galaxy wiki documentation about how to integrate tools (dependencies) into Galaxy. I have implemented a locally managed instance of Galaxy that my business is using with our cluster and now have a freshly installed and configured instance of Galaxy running. It is bare-bones right now and I did not use mercurial to sync any existing files/directory structures. I have seen the page on external tool dependencies (http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies) needed for Galaxy, but I am somewhat unsure where to place the tools to utilize them as intended (other than through trial & error).
It appears that there are shell directories for the tools under ~/galaxy-dist/tools/ with basic wrapper scripts but without the corresponding executables (very few that I've noticed have the tools already in them). Is the intent to download the dependency tools and (building from source if necessary) take the binaries in those directories and copy them to their corresponding directory under ~/galaxy-dist/tools/? This seems to have worked with an error I first got when clipping a FASTQ file which reported that fastx_clipper was not a recognized command. So I downloaded the FASTX Toolkit, compiled the binaries, and copied only the binaries into the corresponding fastx tools directory. Would I do the same thing for TopHat and Cufflinks by taking all their binaries (combined) and copying them into ~/galaxy-dist/tools/ngs_rna/?
Hi Josh,
There are two ways to do this. The simplest is to place the binaries into a directory on the Galaxy user's $PATH. The second is via the tool dependency system, which I need to write up documentation for to put in the wiki, which I'll do this week.
Even if that is the case though, I have occasionally gotten errors about tools missing in completely different directories. One was for the FASTQ Groomer. One user saw this error in their browser (which for now is the only way I know to figure out where tools are *expected* to be):
File "/home/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py", line 141, in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##rgFastQC.py error - cannot find executable /home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc
Java JARs are a special case, and FastQC has a unique way of locating its jars, which is why it is expected to be found in that directory. This needs to be documented.
To fix this I downloaded the FastQC tar file from its webpage, unzipped it, and copied the fastqc binary/script to the "home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/" directory. I also had to mkdir FastQC/ under jars/ to place it there since it didn't already exist. Had I not been told the specific directory by the error I'm not sure how I would have intuitively known to place the binary there (unless I'm overlooking some critical documentation). And how do I know that other similar things are not missing which should be there? Can anyone shed some light on this please? Adding a brief page on the Galaxy wiki site under the Admin section about this would really help, even if it only showed an example for one or two specific tools.
The list of external dependencies by tool is maintained here:
http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies
I'll update this page with links to the new documentation when I write it. I should also add that work is under way to make it possible to automatically install these dependencies as needed.
--nate
Thanks, Josh ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Thanks Nate! Updated documentation is always welcome and useful. I appreciate the clarifications.
-J
On Mon, Dec 12, 2011 at 10:18 AM, Nate Coraor nate@bx.psu.edu wrote:
On Dec 9, 2011, at 4:34 PM, Josh Nielsen wrote:
Hello,
I have a question which I have not seen specifically addressed in the
online Galaxy wiki documentation about how to integrate tools (dependencies) into Galaxy. I have implemented a locally managed instance of Galaxy that my business is using with our cluster and now have a freshly installed and configured instance of Galaxy running. It is bare-bones right now and I did not use mercurial to sync any existing files/directory structures. I have seen the page on external tool dependencies ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies) needed for Galaxy, but I am somewhat unsure where to place the tools to utilize them as intended (other than through trial & error).
It appears that there are shell directories for the tools under
~/galaxy-dist/tools/ with basic wrapper scripts but without the corresponding executables (very few that I've noticed have the tools already in them). Is the intent to download the dependency tools and (building from source if necessary) take the binaries in those directories and copy them to their corresponding directory under ~/galaxy-dist/tools/? This seems to have worked with an error I first got when clipping a FASTQ file which reported that fastx_clipper was not a recognized command. So I downloaded the FASTX Toolkit, compiled the binaries, and copied only the binaries into the corresponding fastx tools directory. Would I do the same thing for TopHat and Cufflinks by taking all their binaries (combined) and copying them into ~/galaxy-dist/tools/ngs_rna/?
Hi Josh,
There are two ways to do this. The simplest is to place the binaries into a directory on the Galaxy user's $PATH. The second is via the tool dependency system, which I need to write up documentation for to put in the wiki, which I'll do this week.
Even if that is the case though, I have occasionally gotten errors about
tools missing in completely different directories. One was for the FASTQ Groomer. One user saw this error in their browser (which for now is the only way I know to figure out where tools are *expected* to be):
File "/home/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py", line 141,
in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##rgFastQC.py error - cannot find executable /home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc
Java JARs are a special case, and FastQC has a unique way of locating its jars, which is why it is expected to be found in that directory. This needs to be documented.
To fix this I downloaded the FastQC tar file from its webpage, unzipped
it, and copied the fastqc binary/script to the "home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/" directory. I also had to mkdir FastQC/ under jars/ to place it there since it didn't already exist. Had I not been told the specific directory by the error I'm not sure how I would have intuitively known to place the binary there (unless I'm overlooking some critical documentation). And how do I know that other similar things are not missing which should be there? Can anyone shed some light on this please? Adding a brief page on the Galaxy wiki site under the Admin section about this would really help, even if it only showed an example for one or two specific tools.
The list of external dependencies by tool is maintained here:
http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies
I'll update this page with links to the new documentation when I write it. I should also add that work is under way to make it possible to automatically install these dependencies as needed.
--nate
Thanks, Josh ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Josh,
It appears that there are shell directories for the tools under ~/galaxy-dist/tools/ with basic wrapper scripts but without the corresponding executables (very few that I've noticed have the tools already in them). Is the intent to download the dependency tools and (building from source if necessary) take the binaries in those directories and copy them to their corresponding directory under ~/galaxy-dist/tools/? This seems to have worked with an error I first got when clipping a FASTQ file which reported that fastx_clipper was not a recognized command. So I downloaded the FASTX Toolkit, compiled the binaries, and copied only the binaries into the corresponding fastx tools directory. Would I do the same thing for TopHat and Cufflinks by taking all their binaries (combined) and copying them into ~/galaxy-dist/tools/ngs_rna/?
You'll want to read about Galaxy Tool files a bit to understand the files in ~/galaxy-dist/tools:
http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax#Admin.2BAC8-Too...
These are not shell directories; instead, they include tool config files + additional wrapper scripts to run a tool in Galaxy.
To answer your question, executables for tools need to be in your path but do not need to be in the config/wrapper directories. For example, in an SGE cluster, we suggest setting the PATH environment var in ~/.sge_request
Even if that is the case though, I have occasionally gotten errors about tools missing in completely different directories. One was for the FASTQ Groomer. One user saw this error in their browser (which for now is the only way I know to figure out where tools are *expected* to be):
File "/home/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py", line 141, in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##rgFastQC.py error - cannot find executable /home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc
The exception to the above is Java-based tools. For these tools, you'll need to use the ~/galaxy-dist/shared/jars directory. This is a limitation of Galaxy that will likely be addressed in the future.
Adding a brief page on the Galaxy wiki site under the Admin section about this would really help, even if it only showed an example for one or two specific tools.
I looked a bit but couldn't find it; I suspect it is out on the wiki somewhere, though clearly it needs to be easier to find.
Good luck, J.
Thanks Jeremy,
That does clear quite a few things up. I will probably take the route of just adding the directories that contain the tool executables to my path. And I guess I'll just pay attention to which tools are Java based and copy the executables under the jars directory. Thanks for looking on the wiki too. If you find (or create) a suitable wiki page sometime soon would you be so kind as to post a link to it here, for myself and posterity? That would be great. Thanks for your help!
P.S. Sorry for the 'double post'. I don't use mailing lists often.
-J
On Sat, Dec 10, 2011 at 8:33 AM, Jeremy Goecks jeremy.goecks@emory.eduwrote:
Josh,
It appears that there are shell directories for the tools under ~/galaxy-dist/tools/ with basic wrapper scripts but without the corresponding executables (very few that I've noticed have the tools already in them). Is the intent to download the dependency tools and (building from source if necessary) take the binaries in those directories and copy them to their corresponding directory under ~/galaxy-dist/tools/? This seems to have worked with an error I first got when clipping a FASTQ file which reported that fastx_clipper was not a recognized command. So I downloaded the FASTX Toolkit, compiled the binaries, and copied only the binaries into the corresponding fastx tools directory. Would I do the same thing for TopHat and Cufflinks by taking all their binaries (combined) and copying them into ~/galaxy-dist/tools/ngs_rna/?
You'll want to read about Galaxy Tool files a bit to understand the files in ~/galaxy-dist/tools:
http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax#Admin.2BAC8-Too...
These are not shell directories; instead, they include tool config files + additional wrapper scripts to run a tool in Galaxy.
To answer your question, executables for tools need to be in your path but do not need to be in the config/wrapper directories. For example, in an SGE cluster, we suggest setting the PATH environment var in ~/.sge_request
Even if that is the case though, I have occasionally gotten errors about tools missing in completely different directories. One was for the FASTQ Groomer. One user saw this error in their browser (which for now is the only way I know to figure out where tools are *expected* to be):
*File "/home/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py", line 141, in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##rgFastQC.py error - cannot find executable /home/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc
The exception to the above is Java-based tools. For these tools, you'll need to use the ~/galaxy-dist/shared/jars directory. This is a limitation of Galaxy that will likely be addressed in the future.
Adding a brief page on the Galaxy wiki site under the Admin section about this would really help, even if it only showed an example for one or two specific tools.
I looked a bit but couldn't find it; I suspect it is out on the wiki somewhere, though clearly it needs to be easier to find.
Good luck, J.
galaxy-dev@lists.galaxyproject.org
-
Jeremy Goecks -
Josh Nielsen -
Nate Coraor