I fixed the error. It have been the directory permissions.
Found in galaxy log:
tool_shed.util.hg_util DEBUG 2015-06-16 16:17:04,045 Error cloning repository: [Errno 13] Permission denied: '/mnt/galaxy/galaxy-app/gbs_tools/testtoolshed.g2.bx.psu.edu'
But the question with the error message is still active, I guess. 

2015-06-15 15:53 GMT-05:00 Alexander Vowinkel <vowinkel.alexander@gmail.com>:
Hi,

I have a custom tool_conf.xml in gbs_tools/gbs_tool_conf.xml
It is added to the config and loaded without problems.
"<toolbox tool_path="./gbs_tools">"

But when I want to install tools from a toolshed (test and main)
to it ("Shed tool configuration file"), it reports Error during installation.

Clicking on the tool, it says "This repository is not installed
correctly (see the Repository installation error below). [...]"
But there is no "Repository installation error".
So, how can I find out, what the problem is?
Installing in the default tool_shed_conf.xml works.

I also checked if it's a problem with the tool_path (I assume it
points to the galaxy root). So I created a directory gbs_tools
in gbs_tools, in case it is relative to the xml file directory.
No difference.

Thanks for help!
Alexander