K-Safety Best Practices
This document was co-authored by Shrirang Kamat and Soniya Shah.
This document provides an overview of K-safety, data safety, and node dependencies. These concepts are important when considering the high availability and recovery features in Vertica.
K-safety is a measure of fault tolerance in your database cluster. The value K represents the number of replicas of the data that exists in the database. To achieve high availability in Vertica, a segmented projection should have at least one buddy projection, and an unsegmented projection should be replicated on all the nodes. In Vertica, K-safety enforces these projection requirements.
In a database with a K-safety of 1, auto projections that Vertica creates will have a buddy projection. If a custom projection was creating without the ksafe1 keyword, the projection does not have a buddy projection, and the projection is marked unsafe until the buddy projection is explicitly created. All unsafe projections are restricted as follows:
- Not up to date
- Cannot be refreshed
- Cannot receive data loaded into a table
- Are not used for answering queries
You can run the following statement to check your database K-safety:
=> SELECT GET_DESIGN_KSAFE();
Unsegmented projections have a copy of the data on every node in the cluster. For a database with a K-safety of 1, every segmented projection has a buddy projection that contains a copy of the data.
In segmented projections, the data is split into segments based on the segmentation expression specified in the projection design. Data segments that belong to a segmented projection and the buddy projections are laid out on nodes with offset.
Nodes that hold a data segment and a copy are considered buddy nodes. Buddy nodes store either a projection segment or its copy. These nodes share a dependency.
The following figure shows node dependency. Node 1 and Node 2 are buddy nodes and share dependency for data segment S1. Node 2 and Node 3 are buddy nodes and share dependency for data segment S2, and so forth.
You can use the following query to get a list of node dependencies in your database:
=> SELECT dependency_id, min(node_name ) AS node_x, max(node_name) AS node_y FROM vs_node_dependencies join nodes on node_oid = node_id GROUP BY 1 having count(*) = 2 ORDER BY 1; dependency_id | node_x | node_y --------------+----------------------+---------------------- 0 | node1 | node3 1 | node2 | node4 2 | node3 | node4 3 | node2 | node5 4 | node1 | node5 (5 rows)
You can also get node dependencies by running the following command:
=> SELECT GET_NODE_DEPENDENCIES (); get_node_dependencies ----------------------------------------------------------------------------- Deps: 00110 - cnt: 2 01001 - cnt: 2 01100 - cnt: 2 10001 - cnt: 2 10010 - cnt: 2 11111 - cnt: 9
00001 – name: Node 2 00010 – name: Node 1 00100 – name: Node 3 01000 – name: Node 4 10000 – name: Node 5 (1 row)
In Vertica versions prior to 8.0.1, the get_node_dependencies API did not provide dependency bits to node mapping. If you are running an earlier version of Vertica and reading the node dependency bits, the rightmost bit represents the permanent node with the lowest node_id and the leftmost bit represents the permanent bit with the highest node_id.
In Vertica 8.0.1, you can run the verbose version of get_node_dependencies to see the node names next to the bit pattern:
=> SELECT get_node_dependencies_verbose ();
You can run the following statement to get the node names in ascending order of node_id:
=> SELECT node_name FROM nodes ORDER BY node_id; node_name ------------------- node2 node1 node3 node4 node5 (5 rows)
A bit value of 1 means that the node representing the bit is the buddy of another node with a bit value of 1. For example, in dependency bits 00110, a node with the second lowest node_id is the buddy node for a node with the third lowest node_id.
The cnt value represents the number of projection buddy pairs in the database. A dependency with all bits set to 1 represents an unsegmented projection and the cnt value gives the number of unsegmented projections in the database.
If you drew a graph with the nodes as vertices and the dependencies between the nodes as the edges of the graph, the graph would be a ring where each vertice had two edges to maintain a K-safety of 1. The following figure shows the node dependency ring:
If all the nodes in the ring had three edges, that would represent a database with a K-safety of 2. If only some nodes had more than two edges, that ring would indicate that rebalance activity is incomplete.
You can run the following statement to recompute node dependencies:
=> SELECT RECOMPUTE_NODE_DEPENDENCIES();
High Availability and Data Safety
A Vertica database with a K-safety of 1 will remain UP and fully functional if more than half the nodes in the database are UP and no two adjacent nodes in the dependency ring are down. In Vertica, this feature is known as data safety.
For example, the database remains UP if node 1 and node 4 go DOWN, because they are not adjacent nodes in the dependency ring. The database will perform an unsafe SHUTDOWN if node 1 and node 5 go DOWN at the same time. This happens because the segment of data on those nodes is no longer accessible.
When you have a node DOWN in your cluster, you can run the following query to get a list of critical nodes. The results table is useful when database administrators are planning scheduled maintenance on more than one node:
=> SELECT * FROM critical_nodes;
For best results, Vertica engineers recommend the following:
- A K-safety of 1 and do not recommend a K-safety of 2. Although you can have two adjacent nodes in the dependency ring go DOWN while the system remains UP, the chances of that occurrence are slim. Additionally, a K-safety of 2 uses 50% more system resources. This usage could slow system performance.
- Making your database rack aware by defining fault groups for large clusters that span more than two racks. For more information, see High Availability with Fault Groups in the Vertica documentation.
Database Recovery and Data Safety
When you start Vertica, you need more than half the nodes to form a quorum. No two adjacent nodes in the dependency ring can be left out.
For more information, see K-Safety in the Vertica documentation.