Monitoring Linux Resource Usage

You should monitor system resource usage on any or all nodes in the cluster. You can use System Activity Reporting (SAR) to monitor resource usage.

Micro Focus recommends that you install pstack and sysstat to help monitor Linux resources. The SYSSTAT package contains utilities for monitoring system performance and usage activity, such as sar, as well as tools you can schedule via cron to collect performance and activity data. See the SYSSTAT Web page for details.

The pstack utility lets you print a stack trace of a running process. See the PSTACK man page for details.
  1. Log in to the database administrator account on any node.
  2. Run the top utility

    $ top

    A high CPU percentage in top indicates that Vertica is CPU-bound. For example:

    top - 11:44:28 up 53 days, 23:47,  9 users,  load average: 0.91, 0.97, 0.81
    Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
    Cpu(s):  26.9%us,  1.3%sy,  0.0%ni, 71.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:   4053136 total,  3882020k used,  171116 free,   407688 buffers
    Swap:  4192956 total,   176k used,  4192780 free,  1526436 cached
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    13703 dbadmin    1   0 1374m 678m  55m S 99.9 17.1   6:21.70 vertica
     2606 root      16   0 32152  11m 2508 S  1.0  0.3   0:16.97 X
        1 root      16   0  4748  552  456 S  0.0  0.0   0:01.51 init
        2 root      RT  -5     0    0    0 S  0.0  0.0   0:04.92 migration/0
        3 root      34  19     0    0    0 S  0.0  0.0   0:11.75 ksoftirqd/0
    ...
    

    Some possible reasons for high CPU usage are:

    • The Tuple Mover runs automatically and thus consumes CPU time even if there are no connections to the database.
    • Log in as root and change the Linux parameter swappiness to 1.

      # echo 1 > /proc/sys/vm/swappiness

      Setting swappiness in this manner does not save the value after a reboot. To permanently set the swappiness value, add or update the following in /etc/sysctl.conf:

      vm.swappiness = 1

      You can also check the swappiness value as follows:

      cat /proc/sys/vm/swappiness
    • Some information sources:

  3. Run the iostat utility. A high idle time in top at the same time as a high rate of blocks read in iostat indicates that Vertica is disk-bound. For example:

    $ /usr/bin/iostat
    Linux 2.6.18-164.el5 (qa01)     02/05/2011
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.77    2.32    0.76    0.68    0.00   95.47
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    hda               0.37         3.40        10.37    2117723    6464640
    sda               0.46         1.94        18.96    1208130   11816472
    sdb               0.26         1.79        15.69    1114792    9781840
    sdc               0.24         1.80        16.06    1119304   10010328
    sdd               0.22         1.79        15.52    1117472    9676200
    md0               8.37         7.31        66.23    4554834   41284840