Hi,

I was just looking at what the galaxy program was doing using top and it is using all the memory! I've rebooted the machine with 90GB RAM and it remained up and running using 57GB until I logged into galaxy and looked to see if there were any running jobs in the admin panel. The memory usages nudged up past 90GB and then died. So it's definitely a memory problem.

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mjv08@aber.ac.uk
t: 01970 62 2807



From: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> on behalf of Martin Vickers [mjv08] <mjv08@aber.ac.uk>
Sent: 16 July 2015 08:48 PM
To: Nate Coraor
Cc: galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] ./run.sh segfault
 

Hi Nate,


Thanks for the reply. In syslog I'm getting;


Jul 16 20:36:41 galaxy kernel: [  117.123921] Out of memory: Kill process 1390 (python) score 986 or sacrifice child
Jul 16 20:36:41 galaxy kernel: [  117.124087] Killed process 1390 (python) total-vm:43496348kB, anon-rss:32611892kB, file-rss:1800kB
(END)

It's a 32GB VM. I could increase it but I wouldn't expect 32GB to be too little. I've attached the full syslog.


Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mjv08@aber.ac.uk
t: 01970 62 2807



From: Nate Coraor <nate@bx.psu.edu>
Sent: 16 July 2015 04:36 PM
To: Martin Vickers [mjv08]
Cc: galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] ./run.sh segfault
 
Hi Martin,

Is there anything in the syslog?

--nate

On Thu, Jul 16, 2015 at 11:26 AM, Martin Vickers <mjv08@aber.ac.uk> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All,

I have a weird issue that's just cropped up. After a new install of
galaxy (checked out on Monday from github) on a ubuntu vm, using
postgres rather than sqlite as well as a few other production
recommendations, I started playing around with the Data Libraries
functionality. I linked a bunch of fastq.gz files into galaxy (around
150 in total) and everything was working fine. I went home and the
next day, it was down.

I tried to start it up as usual (using an init.d script), it worked
for less than a minute and then disappeared again. So I tried running
it as the galaxy user using ./run.sh and I get a seg fault;

Starting server in PID 23173.
serving on http://144.124.110.39:8080
Segmentation fault

Tried again with strace

Starting server in PID 23552.
serving on http://144.124.110.39:8080
[{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 23552
- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=23552,
si_status=SIGKILL, si_utime=1590, si_stime=1930} ---
rt_sigreturn()                          = 23552
write(2, "Killed\n", 7Killed
)                 = 7
read(10, "", 8192)                      = 0
exit_group(137)                         = ?
+++ exited with 137 +++

I can't see anything odd in the log file and I've turned debugging on
in galaxy.ini. I'm at a bit of a loss. Does anyone know what might be
causing it?

Cheers,

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mjv08@aber.ac.uk
t: 01970 62 2807
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJVp80YAAoJEHa0a8GkKQgIGlIH/1VfAPbs/5ApDBdyoOV5qf1y
oCOv93IojARyfI0ksSjF8NRNzw5fNp1R8AzZzomaR3SOUkBuZutre600sy0azTZw
E6gjxtMuvaMyEsOTXtToVarVJT0wTG8+5DJRIYLxtYZm7kvbZK0WuzrN2zDT6663
Rnm7zI/zBpTAyp6uXwgmz0x5gpH6KFwRcEHEbU3JWy6nj1zithJShwYPlBuhT5IB
OaPwOKflcZpZ8NBTEGsh038JrkU+eE50a9aEjQ2m/DpfM/TN9ujgEFm1dyy/iQS7
ewwQUpWJDkA/u0ZX602dsNdV2LvGuKVVMEHiQ25zaUQZ/iGTwKBQsFM2LlDybgA=
=jzYG
-----END PGP SIGNATURE-----
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/