When flushing dirty data in the cache to disk, by default, Linux allows 120 seconds for the flush to take place. During this time, your Vertica process may appear to have hung.
If the disk is overloaded, 120 seconds is not sufficient and you see this error in the dmesg or var/log/messages output:
The root cause of this problem is an imbalance in processing power. In Vertica, the memory and processing power (CPU) may be too high and the disk throughput may be comparatively too low to keep up. All Vertica nodes must have an appropriate balance of CPU, RAM, disk throughput and network bandwidth.
This error usually occurs on a system that
Has a large amount of memory and CPU processing power relative to the amount of disk throughput
Has low disk throughput (for example, virtual machines or machines using storage area networks (SANs) where disk throughput is limited)
Solution
To prevent this problem, consider the following tunings. These tunings are valid for all versions of Vertica:
Reduce vm.dirty_ratio to reduce the maximum amount of memory allocated to dirty pages.
If you have 400 MB/sec write throughput, you can write 6 GB of data in 15 seconds. To calculate the vm.dirty_ratio, divide the amount of data you can transfer in 15 seconds by the amount of RAM. For example:
- For a 256 GB system:6/256 = 2.5%, set vm.dirty_ratio = 3 (approx.)
- For a 128 GB system: 6/128 = 4.6%, set vm.dirty_ratio to 5 (approx.)
If you double the disk throughput, say from 400MB/sec to 800MB/sec, you should also double vm.dirty_ratio.
Reduce the vm.dirty_background_ratio value to flush the cache to disk more frequently. In particular, set vm.dirty_background_ratio to half the value of vm.dirty_ratio.That way, when the cache is half full, the task of flushing to disk starts. For example, if vm.dirty_ratio=6, set vm.dirty_background_ratio to 3.
The vm.dirty_background_ratio parameter specifies a percentage of system memory occupied by dirty data. When this percentage is reached, Linux begins to flush the buffer. The vm.dirty_background_ratio depends on the available disk throughput and is inversely related to the total memory on the system.
To prevent a kernel panic while Vertica is paused during the flush, set hung_task_panic to 0. This allows the process unlimited time to flush the data instead of shutting down the node.