Vertica shuts down? Why?

[Ivy]

Hi , I am new to Vertica. I noticed this morning that one of our 3-node Vertica servers went down. I checked the vertica log and here is what it said. I cleaned it up to make it shorter here.

0/5071: Total memory accounted for by FileCache Allocator: 0
2013-05-10 22:58:30.137 unknown:0x2b583066c1d0 <LOG> @v_intersect_node0001: 0000
0/3916: Memory usage in FileCache:
DETAIL: Size 1073741832: 0 on free list; 0 still in use (0 bytes)
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 <LOG> @v_intersect_node0001: 0000
0/5071: Total memory accounted for by FileCache Allocator: 0
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [SAL] <INFO> Typical LRU usage: 0
free 0 in use
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [SAL] <INFO> Large LRU usage: 0 f
ree 0 in use
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [SAL] <INFO> Large LRU usage: 0 f
ree 0 in use
.....
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [Init] <INFO> Global pool memory
usage: NewPool(0x4712380) 'GlobalPool': totalDtors 0 totalSize 266338304 (121102
224 unused) totalChunks 7
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [Init] <INFO> SAL global pool mem
ory usage: NewPool(0x47023e0) 'SALGlobalPool': totalDtors 0 totalSize 520093696
(313854080 unused) totalChunks 14
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [Init] <INFO> SS::stopPoller()
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [Init] <INFO> DC::shutDown()
2013-05-10 22:58:30.138 unknown:0x2b583066c1d0 [Init] <INFO> Shutdown complete.
Exiting.
2013-05-10 22:58:30.280 unknown:0x2b583066c1d0 [Command] <INFO> Library file has
been unloaded successfully

Can someone please help me interpret this? Thank you!

[Julie]

Hello!

I can't see any issues in the excerpt of the log you provided. Can you you take a look at the spread software log? It is in the /opt/vertica/log directory. See anything odd in it?

[Ivy]

Julie, thanks for your reply. I checked the server and found that all of the spread logs are empty with 0 byte. Does it make sense? Am I looking at the wrong files? Here is an example.

-rw-r--r-- 1 dbadmin dbadmin 0 May 12 04:02 spread_verticatst01.log.1

[Julie]

Do the files (on each of your nodes) all have the same timestamp?

[JimKnicely]

Hi Peng,

Can you please check the value of the kernal parameter "vm.min_free_kbytes"? The installer sets it to the value of 4096.

Increasing the value to a more realistic number normally resolves instability issues on nodes.

If your setting is < 65536, here is how to change it:

1. To update the current value:

• As the root user, execute the sysctl -w vm.min_free_kbytes=65536 command

2. To make the change permanent, edit the /etc/sysctl.conf file and replace...

• vm.min_free_kbytes = 4096

with...

• vm.min_free_kbytes = 65536

After making the changes, the kernel will try and keep 64MB of RAM free at all times.

I hope this helps!

[Ivy]

Hi Jim,

Thanks for your help. I checked the kernel parameters and it has already been set to 65536.

vm.min_free_kbytes = 65536

Thanks,
Peng

[JimKnicely]

Hi,

This issue maybe be a bit complicated. It's hard to diagnose the problem without seeing the entire log file.

Perhaps you can open a case with Vertica. They'll have you run the diagnostics utility (/opt/vertica/bin/diagnostics) and send the results off to them to look over.

Please let us know what you find out!