Re: [galaxy-dev] pbs_python can't be installed via scramble
Hello Will, The second command returns the container.hpp inside the directory of the torque source which i used to install it: find / -name container.hpp /home/user/torque-5.1.1.2-1_18e4a5f1/src/include/container.hpp I noticed there is also a log.h file there. the first command doesn't return anything (I've search manually too, nothing inside /usr/local/include/). How do I fix this? Shall I change the export command of PBS_PYTHON_INCLUDEDIR? Thank you, Makis Date: Thu, 24 Sep 2015 08:31:07 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz@lygos.com To: makis4ever@hotmail.com CC: galaxy-dev@lists.galaxyproject.org And does find /usr/local/include -name container.hppor find / -name container.hppreturn torque/src/include/container.hpp ? My guess is you installed binaries for Torque, but the scrambling requires the Torque source (or at least the header files form Torque). -Will On Thu, Sep 24, 2015 at 1:16 AM, Makis Ladoukakis <makis4ever@hotmail.com> wrote: Hello Will, Thanks again for your reply. Yes I had already tried these but I got the same container.hpp error. The only thing different in my case is that the names of the directories of the directories are a bit different: export PBS_PYTHON_INCLUDEDIR=/usr/local/include/ export PBSCONFIG=/usr/local/bin/pbs-config export LIBTORQUE_DIR=/usr/local/lib/libtorque.so python scripts/scramble.py -e pbs_python Any ideas? Kind regards, Makis Date: Wed, 23 Sep 2015 07:06:32 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz@lygos.com To: makis4ever@hotmail.com CC: galaxy-dev@lists.galaxyproject.org Hi Makis, The thread I posted does have the exact same container.hpp error and proposes a solution. Did you try setting your PBS_PYTHON_INCLUDEDIR to point to your torque/include directory, as mentioned in that thread? from the thread:export PBS_PYTHON_INCLUDEDIR=/usr/local/torque/include/ export PBSCONFIG=/usr/local/torque/bin/pbs-config export LIBTORQUE_DIR=/usr/local/torque/lib/libtorque.so -Will On Wed, Sep 23, 2015 at 5:49 AM, Makis Ladoukakis <makis4ever@hotmail.com> wrote: Hello Will, Thank you for your reply. I had already checked this thread before as I browsed through the mailing list for "pbs_python" issues. It doesn't have the same errors (can't locate log.h and when i provide it it wants the container.hpp ) and up to one point i've already followed the "rescrambling" of the pbs_python. Anyone else had any experience with that? Thank you, Makis Date: Tue, 22 Sep 2015 09:35:11 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz@lygos.com To: makis4ever@hotmail.com CC: galaxy-dev@lists.galaxyproject.org Here is an old thread that looks rather similar to your problems:https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/msg00078.htm... -Will On Tue, Sep 22, 2015 at 2:43 AM, Makis Ladoukakis <makis4ever@hotmail.com> wrote: Hello everyone, I'm trying to set up a Galaxy instance on a multi-core server in my university so according to the instructions here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#PBS I installed Torque, changed the eggs.ini file by adding the 4.4.0. version of pbs_python and tried to setup pbs_python via scramble: LIBTORQUE_DIR=/usr/local/lib/ python scripts/scramble.py -e pbs_python When I did that I got the error: Failed to find log.h in inlcude dir /usr/include/torque. (Set incude dir via PBS_PYTHON_INCLUDEDIR variable) Traceback (most recent call last): File "scripts/scramble.py", line 50, in <module> egg.scramble() File "/home/galaxy/galaxy/lib/galaxy/eggs/scramble.py", line 57, in scramble self.run_scramble_script() File "/home/galaxy/galaxy/lib/galaxy/eggs/scramble.py", line 210, in run_scramble_script raise ScrambleFailure( self, "%s(): Egg build failed for %s %s" % ( sys._getframe().f_code.co_name, self.name, self.version ) ) galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for pbs_python 4.4.0 so I did some digging around and found that the file I need is in /scripts/scramble/build/py2.7-linux-x86_64-ucs4/pbs_python/src/C++/ (please correct me if I am wrong) So I tried again using: PBS_PYTHON_INCLUDEDIR=/home/galaxy/galaxy/scripts/scramble/build/py2.7-linux-x86_64-ucs4/pbs_python/src/C++/ LIBTORQUE_DIR=/usr/local/lib/ python scripts/scramble.py -e pbs_python but then I got the error: In file included from src/C++/pbs_ifl.h:90:0, from src/C++/pbs_wrap.cxx:2978: /usr/local/include/u_hash_map_structs.h:82:25: fatal error: container.hpp: No such file or directory #include "container.hpp" Can someone help me please? Kind regards, Makis ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email.
Hello again, So I figured out a workaround. I had to downgrade torque to version 4.2.10 (that is the latest version of torque that doesn't have any compatibility issues with pbs_python 4.4.0). Then i used: export PBS_PYTHON_INCLUDEDIR=/home/user/torque-4.2.10/src/include/ to include... well... the include directory So the pbs_python egg for 4.4.0. was created succesfully. But then after running galaxy: GALAXY_RUN_ALL=1 sh run.sh --daemon I got another error: galaxy.jobs.runners DEBUG 2015-09-25 15:08:22,030 Loading DRMAARunner with params: {'invalidjobexception_retries': '0', 'invalidjobexception_state': 'ok', 'internalexception_retries': '0', 'internalexception_state': 'ok'} galaxy.jobs.runners.state_handler_factory DEBUG 2015-09-25 15:08:22,030 Loaded 'failure' state handler from module galaxy.jobs.runners.state_handlers.resubmit Traceback (most recent call last): File "/home/galaxy/galaxy/lib/galaxy/webapps/galaxy/buildapp.py", line 51, in app_factory app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs ) File "/home/galaxy/galaxy/lib/galaxy/app.py", line 142, in __init__ self.job_manager = manager.JobManager( self ) File "/home/galaxy/galaxy/lib/galaxy/jobs/manager.py", line 23, in __init__ self.job_handler = handler.JobHandler( app ) File "/home/galaxy/galaxy/lib/galaxy/jobs/handler.py", line 32, in __init__ self.dispatcher = DefaultJobDispatcher( app ) File "/home/galaxy/galaxy/lib/galaxy/jobs/handler.py", line 722, in __init__ self.job_runners = self.app.job_config.get_job_runner_plugins( self.app.config.server_name ) File "/home/galaxy/galaxy/lib/galaxy/jobs/__init__.py", line 667, in get_job_runner_plugins rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get( 'kwds', {} ) ) File "/home/galaxy/galaxy/lib/galaxy/jobs/runners/drmaa.py", line 62, in __init__ drmaa = __import__( "drmaa" ) File "/home/galaxy/galaxy/eggs/drmaa-0.7.6-py2.7.egg/drmaa/__init__.py", line 48, in <module> ZipImportError: bad local file header in /home/galaxy/galaxy/eggs/drmaa-0.7.6-py2.7.egg Removing PID file handler0.pid Has anyone ever encountered that? Thank you, Makis From: makis4ever@hotmail.com To: wholtz@lygos.com Date: Fri, 25 Sep 2015 12:07:02 +0300 CC: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble Hello Will, The second command returns the container.hpp inside the directory of the torque source which i used to install it: find / -name container.hpp /home/user/torque-5.1.1.2-1_18e4a5f1/src/include/container.hpp I noticed there is also a log.h file there. the first command doesn't return anything (I've search manually too, nothing inside /usr/local/include/). How do I fix this? Shall I change the export command of PBS_PYTHON_INCLUDEDIR? Thank you, Makis Date: Thu, 24 Sep 2015 08:31:07 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz@lygos.com To: makis4ever@hotmail.com CC: galaxy-dev@lists.galaxyproject.org And does find /usr/local/include -name container.hppor find / -name container.hppreturn torque/src/include/container.hpp ? My guess is you installed binaries for Torque, but the scrambling requires the Torque source (or at least the header files form Torque). -Will On Thu, Sep 24, 2015 at 1:16 AM, Makis Ladoukakis <makis4ever@hotmail.com> wrote: Hello Will, Thanks again for your reply. Yes I had already tried these but I got the same container.hpp error. The only thing different in my case is that the names of the directories of the directories are a bit different: export PBS_PYTHON_INCLUDEDIR=/usr/local/include/ export PBSCONFIG=/usr/local/bin/pbs-config export LIBTORQUE_DIR=/usr/local/lib/libtorque.so python scripts/scramble.py -e pbs_python Any ideas? Kind regards, Makis Date: Wed, 23 Sep 2015 07:06:32 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz@lygos.com To: makis4ever@hotmail.com CC: galaxy-dev@lists.galaxyproject.org Hi Makis, The thread I posted does have the exact same container.hpp error and proposes a solution. Did you try setting your PBS_PYTHON_INCLUDEDIR to point to your torque/include directory, as mentioned in that thread? from the thread:export PBS_PYTHON_INCLUDEDIR=/usr/local/torque/include/ export PBSCONFIG=/usr/local/torque/bin/pbs-config export LIBTORQUE_DIR=/usr/local/torque/lib/libtorque.so -Will On Wed, Sep 23, 2015 at 5:49 AM, Makis Ladoukakis <makis4ever@hotmail.com> wrote: Hello Will, Thank you for your reply. I had already checked this thread before as I browsed through the mailing list for "pbs_python" issues. It doesn't have the same errors (can't locate log.h and when i provide it it wants the container.hpp ) and up to one point i've already followed the "rescrambling" of the pbs_python. Anyone else had any experience with that? Thank you, Makis Date: Tue, 22 Sep 2015 09:35:11 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz@lygos.com To: makis4ever@hotmail.com CC: galaxy-dev@lists.galaxyproject.org Here is an old thread that looks rather similar to your problems:https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/msg00078.htm... -Will On Tue, Sep 22, 2015 at 2:43 AM, Makis Ladoukakis <makis4ever@hotmail.com> wrote: Hello everyone, I'm trying to set up a Galaxy instance on a multi-core server in my university so according to the instructions here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#PBS I installed Torque, changed the eggs.ini file by adding the 4.4.0. version of pbs_python and tried to setup pbs_python via scramble: LIBTORQUE_DIR=/usr/local/lib/ python scripts/scramble.py -e pbs_python When I did that I got the error: Failed to find log.h in inlcude dir /usr/include/torque. (Set incude dir via PBS_PYTHON_INCLUDEDIR variable) Traceback (most recent call last): File "scripts/scramble.py", line 50, in <module> egg.scramble() File "/home/galaxy/galaxy/lib/galaxy/eggs/scramble.py", line 57, in scramble self.run_scramble_script() File "/home/galaxy/galaxy/lib/galaxy/eggs/scramble.py", line 210, in run_scramble_script raise ScrambleFailure( self, "%s(): Egg build failed for %s %s" % ( sys._getframe().f_code.co_name, self.name, self.version ) ) galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for pbs_python 4.4.0 so I did some digging around and found that the file I need is in /scripts/scramble/build/py2.7-linux-x86_64-ucs4/pbs_python/src/C++/ (please correct me if I am wrong) So I tried again using: PBS_PYTHON_INCLUDEDIR=/home/galaxy/galaxy/scripts/scramble/build/py2.7-linux-x86_64-ucs4/pbs_python/src/C++/ LIBTORQUE_DIR=/usr/local/lib/ python scripts/scramble.py -e pbs_python but then I got the error: In file included from src/C++/pbs_ifl.h:90:0, from src/C++/pbs_wrap.cxx:2978: /usr/local/include/u_hash_map_structs.h:82:25: fatal error: container.hpp: No such file or directory #include "container.hpp" Can someone help me please? Kind regards, Makis ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello everyone, I am trying to set up a local Galaxy instance publicly available and configured via pbs/torque in order to fully exploit a multi-cpu server. So I changed my job_conf.xml and my galaxy.ini files (described in the end of the e-mail) according to the istructions on the Galaxy site but when I tried to run the run.sh script: GALAXY_RUN_ALL=1 sh run.sh --daemon all the handlers start from 127.0.0.1 instead of 0.0.0.0. Has anyone encountered that issue before. Am I missing something? Thank you, Makis The changes in the files: galaxy.ini [uwsgi] processes = 12 stats = 0.0.0.0:9191 # i had this at 127.0.0.1:9191 but with no difference socket = 0.0.0.0:4001 # I had this at 127.0.0.1:4001 but with no difference pythonpath = lib threads = 4 logto = /home/galaxy/uwsgi.log master = True [server:handler0] use = egg:Paste#http port = 8090 use_threadpool = True threadpool_workers = 10 [server:handler1] use = egg:Paste#http port = 8091 use_threadpool = True threadpool_workers = 10 [server:handler2] use = egg:Paste#http port = 8092 use_threadpool = True threadpool_workers = 10 [server:handler3] use = egg:Paste#http port = 8093 use_threadpool = True threadpool_workers = 10 [server:handler4] use = egg:Paste#http port = 8094 use_threadpool = True threadpool_workers = 10 [server:handler5] use = egg:Paste#http port = 8095 use_threadpool = True threadpool_workers = 10 # The port on which to listen. #port = 8090 # The address on which to listen. By default, only listen to localhost (Galaxy # will not be accessible over the network). Use '0.0.0.0' to listen on all # available network interfaces. host = 0.0.0.0 job_conf.xml <?xml version="1.0"?> <job_conf> <plugins workers="16"> <!-- "workers" is the number of threads for the runner's work queue. The default from <plugins> is used if not defined for a <plugin>. --> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" workers="16"/> </plugins> <handlers default="handlers"> <!-- Additional job handlers - the id should match the name of a [server:<id>] in galaxy.ini. --> <handler id="handler0" tags="handlers"/> <handler id="handler1" tags="handlers"/> <handler id="handler2" tags="handlers"/> <handler id="handler3" tags="handlers"/> <handler id="handler4" tags="handlers"/> <handler id="handler5" tags="handlers"/> </handlers> <destinations default="torque"> <!-- Destinations define details about remote resources and how jobs should be executed on those remote resources. --> <destination id="torque" runner="pbs"/> <destination id="other_cluster" runner="pbs"> <param id="destination">@other.cluster</param> </destination> <destination id="long_jobs" runner="pbs"> <param id="Resource_List">walltime=72:00:00, nodes=1:ppn=8</param> <param id="-p">128</param> </destination> </destinations>
Actually I just fixed that by adding the "host" in each handler but I get the same error as before from the browser when I try to access the webpage: Bad Gateway The proxy server received an invalid response from an upstream server. My apache configuration settings are the following: RewriteEngine on #RewriteRule ^(.*) http://localhost:8080$1 [P] RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /home/galaxy/galaxy/static/june_2007_sty$ RewriteRule ^/galaxy/static/scripts/(.*) /home/galaxy/galaxy/static/scripts/pac$ RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /home/galaxy/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt [L] LoadModule proxy_uwsgi_module /usr/lib64/httpd/modules/mod_proxy_uwsgi.so LoadModule uwsgi_module /usr/lib64/httpd/modules/mod_uwsgi.so <Location "/galaxy"> Sethandler uwsgi-handler uWSGISocket 0.0.0.0:4001 uWSGImaxVars 512 </Location> From: makis4ever@hotmail.com To: galaxy-dev@lists.galaxyproject.org Date: Wed, 14 Oct 2015 12:17:05 +0300 Subject: [galaxy-dev] Galaxy in cluster mode won't start from 0.0.0.0. Hello everyone, I am trying to set up a local Galaxy instance publicly available and configured via pbs/torque in order to fully exploit a multi-cpu server. So I changed my job_conf.xml and my galaxy.ini files (described in the end of the e-mail) according to the istructions on the Galaxy site but when I tried to run the run.sh script: GALAXY_RUN_ALL=1 sh run.sh --daemon all the handlers start from 127.0.0.1 instead of 0.0.0.0. Has anyone encountered that issue before. Am I missing something? Thank you, Makis The changes in the files: galaxy.ini [uwsgi] processes = 12 stats = 0.0.0.0:9191 # i had this at 127.0.0.1:9191 but with no difference socket = 0.0.0.0:4001 # I had this at 127.0.0.1:4001 but with no difference pythonpath = lib threads = 4 logto = /home/galaxy/uwsgi.log master = True [server:handler0] use = egg:Paste#http port = 8090 use_threadpool = True threadpool_workers = 10 [server:handler1] use = egg:Paste#http port = 8091 use_threadpool = True threadpool_workers = 10 [server:handler2] use = egg:Paste#http port = 8092 use_threadpool = True threadpool_workers = 10 [server:handler3] use = egg:Paste#http port = 8093 use_threadpool = True threadpool_workers = 10 [server:handler4] use = egg:Paste#http port = 8094 use_threadpool = True threadpool_workers = 10 [server:handler5] use = egg:Paste#http port = 8095 use_threadpool = True threadpool_workers = 10 # The port on which to listen. #port = 8090 # The address on which to listen. By default, only listen to localhost (Galaxy # will not be accessible over the network). Use '0.0.0.0' to listen on all # available network interfaces. host = 0.0.0.0 job_conf.xml <?xml version="1.0"?> <job_conf> <plugins workers="16"> <!-- "workers" is the number of threads for the runner's work queue. The default from <plugins> is used if not defined for a <plugin>. --> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" workers="16"/> </plugins> <handlers default="handlers"> <!-- Additional job handlers - the id should match the name of a [server:<id>] in galaxy.ini. --> <handler id="handler0" tags="handlers"/> <handler id="handler1" tags="handlers"/> <handler id="handler2" tags="handlers"/> <handler id="handler3" tags="handlers"/> <handler id="handler4" tags="handlers"/> <handler id="handler5" tags="handlers"/> </handlers> <destinations default="torque"> <!-- Destinations define details about remote resources and how jobs should be executed on those remote resources. --> <destination id="torque" runner="pbs"/> <destination id="other_cluster" runner="pbs"> <param id="destination">@other.cluster</param> </destination> <destination id="long_jobs" runner="pbs"> <param id="Resource_List">walltime=72:00:00, nodes=1:ppn=8</param> <param id="-p">128</param> </destination> </destinations> ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (1)
-
Makis Ladoukakis