Jul 20 10:07:41 galaxy kernel: [ 1605.276887] python invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jul 20 10:07:41 galaxy kernel: [ 1605.276896] python cpuset=/ mems_allowed=0 Jul 20 10:07:41 galaxy kernel: [ 1605.276906] CPU: 1 PID: 2881 Comm: python Not tainted 3.16.0-30-generic #40~14.04.1-Ubuntu Jul 20 10:07:41 galaxy kernel: [ 1605.276909] Hardware name: Xen HVM domU, BIOS 4.4.1-xs100346 03/18/2015 Jul 20 10:07:41 galaxy kernel: [ 1605.276913] 0000000000000000 ffff88061d8d7a20 ffffffff81762590 ffff88158daf8a30 Jul 20 10:07:41 galaxy kernel: [ 1605.276919] ffff88061d8d7aa8 ffffffff8175c17f 0000000000000000 0000000000000000 Jul 20 10:07:41 galaxy kernel: [ 1605.276923] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jul 20 10:07:41 galaxy kernel: [ 1605.276929] Call Trace: Jul 20 10:07:41 galaxy kernel: [ 1605.276945] [] dump_stack+0x45/0x56 Jul 20 10:07:41 galaxy kernel: [ 1605.276953] [] dump_header+0x7f/0x1f1 Jul 20 10:07:41 galaxy kernel: [ 1605.276966] [] oom_kill_process+0x205/0x360 Jul 20 10:07:41 galaxy kernel: [ 1605.276978] [] ? security_capable_noaudit+0x15/0x20 Jul 20 10:07:41 galaxy kernel: [ 1605.276984] [] out_of_memory+0x492/0x4d0 Jul 20 10:07:41 galaxy kernel: [ 1605.276990] [] __alloc_pages_slowpath+0x63e/0x775 Jul 20 10:07:41 galaxy kernel: [ 1605.276997] [] __alloc_pages_nodemask+0x273/0x2a0 Jul 20 10:07:41 galaxy kernel: [ 1605.277006] [] alloc_pages_vma+0x9a/0x150 Jul 20 10:07:41 galaxy kernel: [ 1605.277014] [] handle_mm_fault+0xcfc/0x10b0 Jul 20 10:07:41 galaxy kernel: [ 1605.277023] [] __do_page_fault+0x19e/0x550 Jul 20 10:07:41 galaxy kernel: [ 1605.277030] [] ? mprotect_fixup+0x151/0x290 Jul 20 10:07:41 galaxy kernel: [ 1605.277035] [] do_page_fault+0x31/0x70 Jul 20 10:07:41 galaxy kernel: [ 1605.277045] [] page_fault+0x28/0x30 Jul 20 10:07:41 galaxy kernel: [ 1605.277048] Mem-Info: Jul 20 10:07:41 galaxy kernel: [ 1605.277051] Node 0 DMA per-cpu: Jul 20 10:07:41 galaxy kernel: [ 1605.277055] CPU 0: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277057] CPU 1: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277060] CPU 2: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277062] CPU 3: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277064] CPU 4: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277067] CPU 5: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277069] CPU 6: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277071] CPU 7: hi: 0, btch: 1 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277073] Node 0 DMA32 per-cpu: Jul 20 10:07:41 galaxy kernel: [ 1605.277077] CPU 0: hi: 186, btch: 31 usd: 56 Jul 20 10:07:41 galaxy kernel: [ 1605.277079] CPU 1: hi: 186, btch: 31 usd: 30 Jul 20 10:07:41 galaxy kernel: [ 1605.277081] CPU 2: hi: 186, btch: 31 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277084] CPU 3: hi: 186, btch: 31 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277086] CPU 4: hi: 186, btch: 31 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277088] CPU 5: hi: 186, btch: 31 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277091] CPU 6: hi: 186, btch: 31 usd: 10 Jul 20 10:07:41 galaxy kernel: [ 1605.277093] CPU 7: hi: 186, btch: 31 usd: 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277095] Node 0 Normal per-cpu: Jul 20 10:07:41 galaxy kernel: [ 1605.277098] CPU 0: hi: 186, btch: 31 usd: 95 Jul 20 10:07:41 galaxy kernel: [ 1605.277100] CPU 1: hi: 186, btch: 31 usd: 56 Jul 20 10:07:41 galaxy kernel: [ 1605.277103] CPU 2: hi: 186, btch: 31 usd: 168 Jul 20 10:07:41 galaxy kernel: [ 1605.277105] CPU 3: hi: 186, btch: 31 usd: 30 Jul 20 10:07:41 galaxy kernel: [ 1605.277107] CPU 4: hi: 186, btch: 31 usd: 88 Jul 20 10:07:41 galaxy kernel: [ 1605.277110] CPU 5: hi: 186, btch: 31 usd: 24 Jul 20 10:07:41 galaxy kernel: [ 1605.277112] CPU 6: hi: 186, btch: 31 usd: 171 Jul 20 10:07:41 galaxy kernel: [ 1605.277114] CPU 7: hi: 186, btch: 31 usd: 55 Jul 20 10:07:41 galaxy kernel: [ 1605.277121] active_anon:19217986 inactive_anon:790560 isolated_anon:0 Jul 20 10:07:41 galaxy kernel: [ 1605.277121] active_file:91 inactive_file:160 isolated_file:74 Jul 20 10:07:41 galaxy kernel: [ 1605.277121] unevictable:0 dirty:0 writeback:0 unstable:0 Jul 20 10:07:41 galaxy kernel: [ 1605.277121] free:108035 slab_reclaimable:4265 slab_unreclaimable:6233 Jul 20 10:07:41 galaxy kernel: [ 1605.277121] mapped:756 shmem:634 pagetables:45602 bounce:0 Jul 20 10:07:41 galaxy kernel: [ 1605.277121] free_cma:0 Jul 20 10:07:41 galaxy kernel: [ 1605.277127] Node 0 DMA free:15884kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 20 10:07:41 galaxy kernel: [ 1605.277135] lowmem_reserve[]: 0 3744 90951 90951 Jul 20 10:07:41 galaxy kernel: [ 1605.277141] Node 0 DMA32 free:351548kB min:2780kB low:3472kB high:4168kB active_anon:2518960kB inactive_anon:405888kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3285420kB mlocked:0kB dirty:0kB writeback:0kB mapped:1412kB shmem:1428kB slab_reclaimable:532kB slab_unreclaimable:660kB kernel_stack:128kB pagetables:5704kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:137472 all_unreclaimable? yes Jul 20 10:07:41 galaxy kernel: [ 1605.277149] lowmem_reserve[]: 0 0 87207 87207 Jul 20 10:07:41 galaxy kernel: [ 1605.277154] Node 0 Normal free:64708kB min:64788kB low:80984kB high:97180kB active_anon:74352984kB inactive_anon:2756352kB active_file:364kB inactive_file:640kB unevictable:0kB isolated(anon):0kB isolated(file):296kB present:90783744kB managed:77993904kB mlocked:0kB dirty:0kB writeback:0kB mapped:1612kB shmem:1108kB slab_reclaimable:16528kB slab_unreclaimable:24256kB kernel_stack:4320kB pagetables:176704kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:24304 all_unreclaimable? yes Jul 20 10:07:41 galaxy kernel: [ 1605.277162] lowmem_reserve[]: 0 0 0 0 Jul 20 10:07:41 galaxy kernel: [ 1605.277167] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15884kB Jul 20 10:07:41 galaxy kernel: [ 1605.277188] Node 0 DMA32: 75*4kB (UEM) 20*8kB (UE) 15*16kB (UE) 18*32kB (UEM) 7*64kB (UE) 3*128kB (UE) 1*256kB (U) 2*512kB (UE) 0*1024kB 2*2048kB (EM) 84*4096kB (MR) = 351548kB Jul 20 10:07:41 galaxy kernel: [ 1605.277209] Node 0 Normal: 197*4kB (UE) 121*8kB (UE) 105*16kB (UEM) 85*32kB (UEM) 38*64kB (E) 15*128kB (E) 9*256kB (UEM) 13*512kB (UEM) 3*1024kB (UM) 3*2048kB (UEM) 9*4096kB (MR) = 65548kB Jul 20 10:07:41 galaxy kernel: [ 1605.277234] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 20 10:07:41 galaxy kernel: [ 1605.277237] 1447 total pagecache pages Jul 20 10:07:41 galaxy kernel: [ 1605.277240] 352 pages in swap cache Jul 20 10:07:41 galaxy kernel: [ 1605.277243] Swap cache stats: add 4346140, delete 4345788, find 29273/41887 Jul 20 10:07:41 galaxy kernel: [ 1605.277245] Free swap = 0kB Jul 20 10:07:41 galaxy kernel: [ 1605.277247] Total swap = 8640508kB Jul 20 10:07:41 galaxy kernel: [ 1605.277249] 23678877 pages RAM Jul 20 10:07:41 galaxy kernel: [ 1605.277251] 0 pages HighMem/MovableOnly Jul 20 10:07:41 galaxy kernel: [ 1605.277253] 3197460 pages reserved Jul 20 10:07:41 galaxy kernel: [ 1605.277254] 0 pages hwpoisoned Jul 20 10:07:41 galaxy kernel: [ 1605.277256] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jul 20 10:07:41 galaxy kernel: [ 1605.277269] [ 418] 0 418 5069 1 13 274 0 upstart-udev-br Jul 20 10:07:41 galaxy kernel: [ 1605.277274] [ 428] 0 428 13030 1 27 289 -1000 systemd-udevd Jul 20 10:07:41 galaxy kernel: [ 1605.277278] [ 607] 102 607 9809 0 23 102 0 dbus-daemon Jul 20 10:07:41 galaxy kernel: [ 1605.277282] [ 613] 0 613 5871 2 17 65 0 rpc.idmapd Jul 20 10:07:41 galaxy kernel: [ 1605.277286] [ 631] 0 631 10864 2 27 91 0 systemd-logind Jul 20 10:07:41 galaxy kernel: [ 1605.277290] [ 635] 101 635 63962 0 27 212 0 rsyslogd Jul 20 10:07:41 galaxy kernel: [ 1605.277293] [ 714] 0 714 3952 1 12 180 0 upstart-file-br Jul 20 10:07:41 galaxy kernel: [ 1605.277297] [ 737] 0 737 3816 0 12 58 0 upstart-socket- Jul 20 10:07:41 galaxy kernel: [ 1605.277301] [ 746] 0 746 5859 1 17 78 0 rpcbind Jul 20 10:07:41 galaxy kernel: [ 1605.277304] [ 793] 0 793 2559 34 9 542 0 dhclient Jul 20 10:07:41 galaxy kernel: [ 1605.277308] [ 802] 109 802 5387 2 16 115 0 rpc.statd Jul 20 10:07:41 galaxy kernel: [ 1605.277311] [ 892] 0 892 90717 0 39 604 0 docker.io Jul 20 10:07:41 galaxy kernel: [ 1605.277315] [ 1040] 0 1040 3956 2 13 40 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277319] [ 1044] 0 1044 3956 2 13 39 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277323] [ 1053] 0 1053 3956 2 13 40 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277326] [ 1054] 0 1054 3956 2 13 38 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277330] [ 1056] 0 1056 3956 2 13 38 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277333] [ 1074] 0 1074 15343 0 32 176 -1000 sshd Jul 20 10:07:41 galaxy kernel: [ 1605.277337] [ 1139] 0 1139 5915 0 17 74 0 cron Jul 20 10:07:41 galaxy kernel: [ 1605.277340] [ 1140] 0 1140 4786 0 13 42 0 atd Jul 20 10:07:41 galaxy kernel: [ 1605.277344] [ 1150] 0 1150 4832 52 15 36 0 irqbalance Jul 20 10:07:41 galaxy kernel: [ 1605.277348] [ 1155] 0 1155 1093 1 8 36 0 acpid Jul 20 10:07:41 galaxy kernel: [ 1605.277351] [ 1210] 0 1210 4494 2 14 68 0 xe-daemon Jul 20 10:07:41 galaxy kernel: [ 1605.277356] [ 1306] 106 1306 61947 23 58 339 -900 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277359] [ 1309] 106 1309 61983 0 55 365 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277363] [ 1310] 106 1310 61947 6 52 350 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277367] [ 1311] 106 1311 61947 10 50 346 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277370] [ 1312] 106 1312 62162 95 53 444 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277374] [ 1313] 106 1313 25937 83 48 332 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277377] [ 1421] 107 1421 5835 0 15 124 0 nrpe Jul 20 10:07:41 galaxy kernel: [ 1605.277381] [ 1465] 0 1465 27293 455 56 7486 0 puppet Jul 20 10:07:41 galaxy kernel: [ 1605.277385] [ 1538] 0 1538 24942 7 53 482 0 apache2 Jul 20 10:07:41 galaxy kernel: [ 1605.277388] [ 1541] 33 1541 97235 10 80 1070 0 apache2 Jul 20 10:07:41 galaxy kernel: [ 1605.277392] [ 1542] 33 1542 228324 0 90 1197 0 apache2 Jul 20 10:07:41 galaxy kernel: [ 1605.277396] [ 1796] 0 1796 3198 2 12 38 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277399] [ 1798] 0 1798 3956 2 12 41 0 getty Jul 20 10:07:41 galaxy kernel: [ 1605.277403] [ 1917] 0 1917 26410 2 56 248 0 sshd Jul 20 10:07:41 galaxy kernel: [ 1605.277406] [ 1965] 1000 1965 26410 30 52 218 0 sshd Jul 20 10:07:41 galaxy kernel: [ 1605.277410] [ 1966] 1000 1966 5820 5 17 666 0 bash Jul 20 10:07:41 galaxy kernel: [ 1605.277414] [ 1990] 1000 1990 6248 95 18 31 0 top Jul 20 10:07:41 galaxy kernel: [ 1605.277417] [ 2267] 0 2267 26410 2 54 247 0 sshd Jul 20 10:07:41 galaxy kernel: [ 1605.277421] [ 2315] 1000 2315 26410 39 53 212 0 sshd Jul 20 10:07:41 galaxy kernel: [ 1605.277424] [ 2316] 1000 2316 5820 2 17 668 0 bash Jul 20 10:07:41 galaxy kernel: [ 1605.277428] [ 2333] 0 2333 16300 2 36 153 0 sudo Jul 20 10:07:41 galaxy kernel: [ 1605.277431] [ 2334] 0 2334 5860 418 17 294 0 bash Jul 20 10:07:41 galaxy kernel: [ 1605.277435] [ 2867] 1001 2867 22875168 20005510 43972 2136795 0 python Jul 20 10:07:41 galaxy kernel: [ 1605.277439] [ 2900] 106 2900 62422 227 57 572 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277442] [ 2911] 106 2911 63110 273 62 1238 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277446] [ 2957] 106 2957 63090 509 62 923 0 postgres Jul 20 10:07:41 galaxy kernel: [ 1605.277449] [ 3037] 0 3037 16134 114 35 0 0 su Jul 20 10:07:41 galaxy kernel: [ 1605.277453] [ 3038] 1001 3038 5276 134 15 0 0 bash Jul 20 10:07:41 galaxy kernel: [ 1605.277456] [ 3103] 1001 3103 6250 145 18 0 0 top Jul 20 10:07:41 galaxy kernel: [ 1605.277460] [ 3261] 0 3261 1087 0 8 22 0 sleep Jul 20 10:07:41 galaxy kernel: [ 1605.277463] Out of memory: Kill process 2867 (python) score 986 or sacrifice child Jul 20 10:07:41 galaxy kernel: [ 1605.277560] Killed process 2867 (python) total-vm:91500672kB, anon-rss:80021168kB, file-rss:872kB