Terrace Routing

By default, nodes in a Vertica cluster form a fully connected network (complete graph). Therefore, large Vertica clusters contain an extremely large number of connections between nodes. Each connection requires buffering. Thus, your current system can have many connections, causing queries to have extremely large data requirements on each node.

Terrace routing is a feature that can reduce the buffer requirements of large queries. Use terrace routing in situations where you have large queries and clusters with a large number of nodes. Without terrace routing, these situations would otherwise require excessive buffer space.

How Terrace Routing Changes Routing Behavior

In the case when data needs to be sent from each node across an entire network. In this case, Vertica uses the n^th node in each rack to send to all the nodes in all the racks. Terrace routing changes this behavior. Instead of sending to all nodes in all racks, the n^th node in each rack sends to the n^th nodes in all the other racks. It additional sends to all the nodes in its own rack.

Before Enabling Terrace Routing

Before you apply terrace routing to your database make sure you have read the following topics and completed the tasks described:

• Fault Groups
• Creating a Fault Group Input File
• Creating Fault Groups Using the Fault Group Script
• High Availability with Fault Groups

How to Apply Terrace Routing

Your Vertica Analytics Platform is not aware of the topology of the cluster on which your Vertica database is running. By applying fault groups to your database, you can make your Vertica Analytics Platform aware of your network's topology.

Implementing terrace routing requires use of fault groups. Terrace routing uses the fault-group definitions to represent the topology of the network of the cluster, because rack membership is a natural division for fault groups. In a terrace-routing approach, Vertica first distributes data within the rack (fault group) before forwarding it between racks. For information about creating fault groups, see Creating Fault Groups Using the Fault Group Script.

For this reason, terrace routing uses the network within the rack to reduce the buffering requirements for queries. Networks between racks can have a higher capacity so there is a performance tradeoff when a connection goes off-rack.

Calculate the TerraceRoutingFactor

TerraceRoutingFactor is a general parameter that sets the ratio of connections without terrace routing to the number of connections with terrace routing.

You can calculate the TerraceRoutingFactor value using this formula:

((RackCount * RackPopulation) - 1)/(RackCount + RackPopulation - 2) = TerraceRoutingFactor

RackCount is the number of racks or fault groups in your cluster, and RackPopulation is the number of nodes in each rack or fault group.

Enable Terrace Routing

The TerraceRoutingFactor is set to a default value of 1000.0, which is large enough to disable the parameter for even the largest clusters. Vertica recommends enabling terrace routing when your cluster contains 64 or more nodes or if your queries require excessive buffer space.

The following examples show how you can enable terrace routing for varying node and rack configurations and then disable it.

Set the TerraceRoutingFactor on a 64 node cluster with 8 racks and 8 nodes in each rack:

=> ALTER DATABASE MyDB SET TerraceRoutingFactor = 4.5;

Set the TerraceRoutingFactor on a 80 node cluster with 10 racks and 8 nodes in each rack in the current session:

=> ALTER SESSION SET TerraceRoutingFactor = 4.94;

Disable terrace routing:

=> ALTER DATABASE MyDB SET TerraceRoutingFactor = 1000;