Hi Peter, OK, I've worked out why the terminal did not start up galaxy. I have done that, performed a blastp search, and it failed with the usual error, but now I have the terminal output for that failure. Firstly the error report from galaxy: ---------------------------------------------------------------------- An error occurred running this job: blastp: 2.2.26+ Package: blast 2.2.26, build Aug 15 2012 17:48:54 BLAST Database error: No alias or index file found for protein database [/media/sf_mikeds_bioinf/haldb/HAL_collection] in search path [/var/lib/galaxy-server/database/job_working_directory/ ---------------------------------------------------------------------- Now, the log output in the terminal window for that galaxy session: (DEBUG flags are at the end) ---------------------------------------------------------------------------- 127.0.0.1 - - [01/May/2013:20:49:58 +1100] "POST /root/user_get_usage HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs.manager DEBUG 2013-05-01 20:49:59,237 (23) Job assigned to handler 'main' 127.0.0.1 - - [01/May/2013:20:50:02 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-05-01 20:50:04,403 (23) Working directory for job is: /usr/lib/galaxy-server/database/job_working_directory/000/23 galaxy.jobs.handler DEBUG 2013-05-01 20:50:04,404 dispatching job 23 to local runner galaxy.jobs.handler INFO 2013-05-01 20:50:04,523 (23) Job dispatched galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:04,727 executing: python /usr/lib/galaxy-server/tools/data_source/upload.py /usr/lib/galaxy-server /usr/lib/galaxy-server/database/tmp/tmpckvivx /usr/lib/galaxy-server/database/tmp/tmpED2TiJ 23:/usr/lib/galaxy-server/database/job_working_directory/000/23/dataset_23_files:/usr/lib/galaxy-server/database/files/000/dataset_23.dat galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:05,459 execution finished: python /usr/lib/galaxy-server/tools/data_source/upload.py /usr/lib/galaxy-server /usr/lib/galaxy-server/database/tmp/tmpckvivx /usr/lib/galaxy-server/database/tmp/tmpED2TiJ 23:/usr/lib/galaxy-server/database/job_working_directory/000/23/dataset_23_files:/usr/lib/galaxy-server/database/files/000/dataset_23.dat galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:05,592 executing external set_meta script for job 23: /usr/lib/galaxy-server/set_metadata.sh ./database/files /usr/lib/galaxy-server/database/job_working_directory/000/23 . /usr/lib/galaxy-server/universe_wsgi.ini /usr/lib/galaxy-server/database/tmp/tmpckvivx /usr/lib/galaxy-server/database/job_working_directory/000/23/galaxy.json /usr/lib/galaxy-server/database/job_working_directory/000/23/metadata_in_HistoryDatasetAssociation_23_mZBOP0,/usr/lib/galaxy-server/database/job_working_directory/000/23/metadata_kwds_HistoryDatasetAssociation_23_ogpRq8,/usr/lib/galaxy-server/database/job_working_directory/000/23/metadata_out_HistoryDatasetAssociation_23_3yS7Yh,/usr/lib/galaxy-server/database/job_working_directory/000/23/metadata_results_HistoryDatasetAssociation_23_71mkqj,,/usr/lib/galaxy-server/database/job_working_directory/000/23/metadata_override_HistoryDatasetAssociation_23_KhG0tM galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:06,549 execution of external set_meta for job 23 finished galaxy.jobs DEBUG 2013-05-01 20:50:06,626 The tool did not define exit code or stdio handling; checking stderr for success 127.0.0.1 - - [01/May/2013:20:50:06 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.datatypes.metadata DEBUG 2013-05-01 20:50:06,773 loading metadata from file for: HistoryDatasetAssociation 23 galaxy.jobs DEBUG 2013-05-01 20:50:06,905 job 23 ended galaxy.datatypes.metadata DEBUG 2013-05-01 20:50:06,905 Cleaning up external metadata files 127.0.0.1 - - [01/May/2013:20:50:10 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:10 +1100] "POST /root/history_get_disk_size HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:11 +1100] "GET /datasets/f09437b8822035f7/display/?preview=True HTTP/1.1" 200 - " http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:23 +1100] "GET /tool_runner?tool_id=ncbi_blastp_wrapper HTTP/1.1" 200 - " http://127.0.0.1:8080/root/tool_menu" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:36 +1100] "POST /tool_runner/index HTTP/1.1" 200 - " http://127.0.0.1:8080/tool_runner?tool_id=ncbi_blastp_wrapper" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:37 +1100] "GET /history HTTP/1.1" 200 - " http://127.0.0.1:8080/tool_runner/index" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:37 +1100] "POST /root/user_get_usage HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:41 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs.manager DEBUG 2013-05-01 20:50:41,334 (24) Job assigned to handler 'main' 127.0.0.1 - - [01/May/2013:20:50:45 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-05-01 20:50:46,447 (24) Working directory for job is: /usr/lib/galaxy-server/database/job_working_directory/000/24 galaxy.jobs.handler DEBUG 2013-05-01 20:50:46,447 dispatching job 24 to local runner galaxy.jobs.handler INFO 2013-05-01 20:50:46,580 (24) Job dispatched galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:46,772 executing: blastp -version &> /usr/lib/galaxy-server/database/tmp/GALAXY_VERSION_STRING_24; python /usr/lib/galaxy-server/tools/ncbi_blast_plus/hide_stderr.py blastp -query "/usr/lib/galaxy-server/database/files/000/dataset_23.dat" -db "/media/sf_mikeds_bioinf/haldb/HAL_collection" -task blastp -evalue 0.001 -out /usr/lib/galaxy-server/database/files/000/dataset_24.dat -outfmt 6 -num_threads 8 127.0.0.1 - - [01/May/2013:20:50:49 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:49,568 execution finished: blastp -version &> /usr/lib/galaxy-server/database/tmp/GALAXY_VERSION_STRING_24; python /usr/lib/galaxy-server/tools/ncbi_blast_plus/hide_stderr.py blastp -query "/usr/lib/galaxy-server/database/files/000/dataset_23.dat" -db "/media/sf_mikeds_bioinf/haldb/HAL_collection" -task blastp -evalue 0.001 -out /usr/lib/galaxy-server/database/files/000/dataset_24.dat -outfmt 6 -num_threads 8 galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:49,658 executing external set_meta script for job 24: /usr/lib/galaxy-server/set_metadata.sh ./database/files /usr/lib/galaxy-server/database/job_working_directory/000/24 . /usr/lib/galaxy-server/universe_wsgi.ini /usr/lib/galaxy-server/database/tmp/tmpckvivx /usr/lib/galaxy-server/database/job_working_directory/000/24/galaxy.json /usr/lib/galaxy-server/database/job_working_directory/000/24/metadata_in_HistoryDatasetAssociation_24_Z7Rv70,/usr/lib/galaxy-server/database/job_working_directory/000/24/metadata_kwds_HistoryDatasetAssociation_24_NGaXc0,/usr/lib/galaxy-server/database/job_working_directory/000/24/metadata_out_HistoryDatasetAssociation_24_0FT0mx,/usr/lib/galaxy-server/database/job_working_directory/000/24/metadata_results_HistoryDatasetAssociation_24_U1RFk9,,/usr/lib/galaxy-server/database/job_working_directory/000/24/metadata_override_HistoryDatasetAssociation_24_PI3NHu galaxy.jobs.runners.local DEBUG 2013-05-01 20:50:50,616 execution of external set_meta for job 24 finished galaxy.jobs DEBUG 2013-05-01 20:50:50,635 The tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2013-05-01 20:50:50,686 setting dataset state to ERROR galaxy.jobs DEBUG 2013-05-01 20:50:50,817 job 24 ended galaxy.datatypes.metadata DEBUG 2013-05-01 20:50:50,817 Cleaning up external metadata files 127.0.0.1 - - [01/May/2013:20:50:53 +1100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" 127.0.0.1 - - [01/May/2013:20:50:53 +1100] "POST /root/history_get_disk_size HTTP/1.1" 200 - "http://127.0.0.1:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0" ------------------------------------------------------------------------------ Regards, Mike On Wed, May 1, 2013 at 8:43 PM, Mike Dyall-Smith <mike.dyallsmith@gmail.com>wrote:
Hi Peter,
In biolinux, I start galaxy by a clicking on an application icon.
If I open the terminal and run the suggested command: $ sh run.sh | grep -i blast I get the following output (long):
$ sudo sh run.sh | grep -i blast python path is: /usr/lib/galaxy-server/eggs/numpy-1.6.0-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/boto-2.2.2-py2.7.egg, /usr/lib/galaxy-server/eggs/mercurial-2.1.2-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/Fabric-1.4.2-py2.7.egg, /usr/lib/galaxy-server/eggs/ssh-1.7.14-py2.7.egg, /usr/lib/galaxy-server/eggs/Whoosh-0.3.18-py2.7.egg, /usr/lib/galaxy-server/eggs/pycrypto-2.5-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/python_lzo-1.08_2.03_static-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/bx_python-0.7.1_7b95ff194725-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/amqplib-0.6.1-py2.7.egg, /usr/lib/galaxy-server/eggs/pexpect-2.4-py2.7.egg, /usr/lib/galaxy-server/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg, /usr/lib/galaxy-server/eggs/Babel-0.9.4-py2.7.egg, /usr/lib/galaxy-server/eggs/Mako-0.4.1-py2.7.egg, /usr/lib/galaxy-server/eggs/WebHelpers-0.2-py2.7.egg, /usr/lib/galaxy-server/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/wchartype-0.1-py2.7.egg, /usr/lib/galaxy-server/eggs/elementtree-1.2.6_20050316-py2.7.egg, /usr/lib/galaxy-server/eggs/docutils-0.7-py2.7.egg, /usr/lib/galaxy-server/eggs/WebOb-0.8.5-py2.7.egg, /usr/lib/galaxy-server/eggs/Routes-1.12.3-py2.7.egg, /usr/lib/galaxy-server/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg, /usr/lib/galaxy-server/eggs/PasteDeploy-1.3.3-py2.7.egg, /usr/lib/galaxy-server/eggs/PasteScript-1.7.3-py2.7.egg, /usr/lib/galaxy-server/eggs/Paste-1.6-py2.7.egg, /usr/lib/galaxy-server/lib, /usr/lib/python2.7, /usr/lib/python2.7/plat-linux2, /usr/lib/python2.7/lib-tk, /usr/lib/python2.7/lib-old, /usr/lib/python2.7/lib-dynload, /usr/local/lib/python2.7/dist-packages, /usr/lib/python2.7/dist-packages/PIL, /usr/lib/python2.7/dist-packages/gst-0.10, /usr/lib/python2.7/dist-packages/gtk-2.0, /usr/lib/pymodules/python2.7, /usr/lib/python2.7/dist-packages/ubuntu-sso-client, /usr/lib/python2.7/dist-packages galaxy.datatypes.registry DEBUG 2013-05-01 19:58:26,358 Loaded sniffer for datatype 'galaxy.datatypes.xml:BlastXml' galaxy.tools.data DEBUG 2013-05-01 19:58:26,371 Loaded tool data table 'blastdb' galaxy.tools.data DEBUG 2013-05-01 19:58:26,372 Loaded tool data table 'blastdb_p' galaxy.tools DEBUG 2013-05-01 19:58:30,808 Loading section: NCBI BLAST+ galaxy.tools DEBUG 2013-05-01 19:58:30,808 Loaded tool id: ncbi_blastn_wrapper, version: 0.0.11. galaxy.tools DEBUG 2013-05-01 19:58:30,808 Loaded tool id: ncbi_blastp_wrapper, version: 0.0.11. galaxy.tools DEBUG 2013-05-01 19:58:30,809 Loaded tool id: ncbi_blastx_wrapper, version: 0.0.11. galaxy.tools DEBUG 2013-05-01 19:58:30,809 Loaded tool id: ncbi_tblastn_wrapper, version: 0.0.11. galaxy.tools DEBUG 2013-05-01 19:58:30,809 Loaded tool id: ncbi_tblastx_wrapper, version: 0.0.11. galaxy.tools DEBUG 2013-05-01 19:58:30,809 Loaded tool id: blastxml_to_tabular, version: 0.0.8. galaxy.tools DEBUG 2013-05-01 19:58:30,813 Loaded tool id: megablast_wrapper, version: 1.2.0. galaxy.tools DEBUG 2013-05-01 19:58:30,813 Loaded tool id: megablast_xml_parser, version: 1.0.0. Traceback (most recent call last): File "./scripts/paster.py", line 34, in <module> command.run() File "/usr/lib/galaxy-server/eggs/PasteScript-1.7.3-py2.7.egg/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/usr/lib/galaxy-server/eggs/PasteScript-1.7.3-py2.7.egg/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/usr/lib/galaxy-server/eggs/PasteScript-1.7.3-py2.7.egg/paste/script/command.py", line 218, in run result = self.command() File "/usr/lib/galaxy-server/eggs/PasteScript-1.7.3-py2.7.egg/paste/script/serve.py", line 303, in command serve() File "/usr/lib/galaxy-server/eggs/PasteScript-1.7.3-py2.7.egg/paste/script/serve.py", line 287, in serve server(app) File "/usr/lib/galaxy-server/eggs/PasteDeploy-1.3.3-py2.7.egg/paste/deploy/loadwsgi.py", line 151, in server_wrapper **context.local_conf) File "/usr/lib/galaxy-server/eggs/PasteDeploy-1.3.3-py2.7.egg/paste/deploy/util/fixtypeerror.py", line 57, in fix_call val = callable(*args, **kw) File "/usr/lib/galaxy-server/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1314, in server_runner serve(wsgi_app, **kwargs) File "/usr/lib/galaxy-server/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1264, in serve threadpool_options=threadpool_options) File "/usr/lib/galaxy-server/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1114, in __init__ RequestHandlerClass, ssl_context) File "/usr/lib/galaxy-server/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1094, in __init__ RequestHandlerClass, ssl_context) File "/usr/lib/galaxy-server/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 358, in __init__ HTTPServer.__init__(self, server_address, RequestHandlerClass) File "/usr/lib/python2.7/SocketServer.py", line 408, in __init__ self.server_bind() File "/usr/lib/python2.7/BaseHTTPServer.py", line 108, in server_bind SocketServer.TCPServer.server_bind(self) File "/usr/lib/python2.7/SocketServer.py", line 419, in server_bind self.socket.bind(self.server_address) File "/usr/lib/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 98] Address already in use
---------------------------------------------------------------------------------------------- as might be expected from this output, it doesn't start up galaxy at localhost.
check is the read permissions on the BLAST database Well, from the terminal, I have access to all those files. The permissions are all the same, e.g. -rwxrwx--- 1 root vboxsf 15721925 Apr 10 18:55 nr.00.phd
Within galaxy, I can open them with the get data menu command.
... Galaxy log output How is the best way to get that?
In other news, I was able to get galaxy working on the host OS X system. This took a long time, but finally, within virtualenv, and with many repeated attempts to get all the python egg dependencies, I got it up and running. In the process, I seem to have deleted some of the old python files in the OS X system (trying to force it to use the macports py27 installation). Perhaps the OS X instance will be OK but it does not come with the blast tools that I want, and I do not see an easy way to install them (I don't see an admin button to automatically install them).
Now that I've played with linux, I can see that it is not so bad to use. Apple seems to have dropped the ball with facilitating scientific computing. From your advice, it seems that instead of spending any more of my time trying to set up galaxy on my laptop, I should look around for ssh access to an existing galaxy server.
Regards, Mike DS
On Wed, May 1, 2013 at 7:26 PM, Peter Cock <p.j.a.cock@googlemail.com>wrote:
Hi Mike,
On Tue, Apr 30, 2013 at 11:17 PM, Mike Dyall-Smith <mike.dyallsmith@gmail.com> wrote:
Thanks Peter. My answers are below:
What query sequences are you using? I have just been using one fasta protein sequence.
OK - this and the fact it works on the host machine is good to know.
Meanwhile monitor the system with top Top says only a max of 27% cpu usage, but the linux screen eventually freezes, and I have to restart. I am not sure how to read out RAM and disk IO from top.
Linux 'top' does list memory usage, both per process and for the system in the text at the top. There are other tools for monitoring IO like iotop - but you could also probably just watch the host Apple OS X System Monitor for this.
This does sound like the VM hasn't got enough RAM to run BLASTP against NR efficiently.
grep blastp paster.log Tried that, and it says there is no such file or directory
In the Galaxy folder? Maybe the default log filename has changed since I setup my machine... are you running Galaxy as a daemon, or running run.sh at the terminal directly? If the later, try something like this:
$ sh run.sh | grep -i blast
(If you don't get much output, adjust the logging level in the universe_wgsi.ini configuration file.)
Could you try running BLAST from the host Mac OX Yes. And it works fine! I get a good match in a relatively short time.
That's progress - the hardware itself is capable :)
I then made a very small protein database, checked it by commandline blastp in both host OS X and in guest linux, and it worked fine. Added it to the blastdb_p.loc file, restarted and saw it listed in galaxy. Tried to use it for a blastp, and got the same error as before with the huge NCBI nr database. So, it is not a matter of size....
There are clearly two separate issues, (1) getting BLAST to run nicely on your VM - which I think is running out of RAM, and (2) sorting out your Galaxy BLAST database configuration.
Something to check is the read permissions on the BLAST database files (which Linux user are you running Galaxy as, and can that user read the database files and their folder?).
I'm keen to see the Galaxy log output to see what exactly was the command line being used to run BLAST, which would help with debugging where the problem is.
Thanks for your comments about RAM and blast searches. It gives me hope that I can get galaxy running usefully. I only chose biolinux because of the suite of programs and the apparent ease of use. The other reason was that I could not install galaxy on OS X (10.6). I get errors that others have noted on the discussion lists but no-one seems to have a solution for.
I used to do my Galaxy tool development on Mac OS X, but it didn't work 100% right, and in any case many of the tools I wanted to wrap and run within Galaxy were Linux only - so now I just ssh into a Linux server to do Galaxy work. Given the main Galaxy development and the Penn state Galaxy server all happens on Linux, you'll have a much easier time under Linux than Mac OS X.
Regards,
Peter
-- _____________________________________ Mike Dyall-Smith, Ph.D.,
-- _____________________________________ Mike Dyall-Smith, Ph.D., Charles Sturt University, PO Box U183, Wagga Wagga Australia 2678 Tel: +61 (0)2 693 32029 email: mike.dyallsmith@gmail.com web: www.haloarchaea.com skype: mike.dyallsmith _____________________________________ On measuring microbial diversity: "This is like randomly sampling a bus load of people and then trying to infer the diversity of all people in the world. You would not expect to find many Lithuanians." - Curtis and Sloan, 2005