Spread Configuration Best Practices

Important: Spread is a critical component of the Vertica database management system. You must take maximum care when manually changing spread configuration. Any misspellings or an incorrect configuration can disable your database and prevent the database to start/become functional.

When a statement is executed, Vertica communicates with other nodes in the cluster via a data channel using the TCP protocol to transmit data. For example, when a query executes a COPY statement, the rows that are loaded and sent between nodes are controlled by the data channel.

The spread process uses the control channel to coordinate all messages and communication between nodes in the cluster. For example, the node state, query plan execution, and COMMIT statements are all controlled by the control channel. The control channel does not coordinate the flow of data.

The following graphic illustrates this architecture:

Spread provides control messaging between nodes about the state of nodes. Since the Vertica database management system is based on an active active redundancy high availability paradigm, Spread provides the database management system with targeted high availability (based on K-Safety level). Spread provides fault tolerance by controlling active processes and reliably delivering messages to all members/nodes in a database group.

Change the Communication Mode (Broadcast or Point to Point)

Spread configuration is stored in the Vertica catalog and a human readable version of the configuration is stored in the spread.conf text file. You can make changes to the spread configuration using Vertica SQL functions that update the spread.conf text file. Manual changes in spread.conf are not persistent and must be avoided at all times.

You can configure spread to run in one of two ways – either using broadcast or using point to point.

UDP Broadcast

IPv4 Broadcast protocol is the default configuration for the control channel. In this configuration, nodes broadcast one message on the targeted network switch and distribute it to all the nodes.

To set the spread control mode to broadcast, use the following statement:

=> SELECT set_control_mode ('broadcast');
set_control_mode
---------------------------
Control mode set to broadcast
(1 row)

UDP Point to Point

In point-to-point configuration, the topology is the same as Broadcast. However, a node can send a package with a header to each node (IPv4 unicast protocol), and the switch can then reroute that package to the corresponding node. If the switch only has Vertica nodes, this option could present more traffic. However, if the switch has other applications or nodes, using this option reduces the chance that the package is lost. The point-to-point mode is preferred when there are many network hops between nodes, or when targeted network is shared and not dedicated to this task such as running clusters in the cloud.

To set the spread control mode to point to point, use the following statement:

=> SELECT set_control_mode ('pt2pt');
set_control_mode
---------------------------
Control mode set to pt2pt
(1 row)

Changing Spread Log Configuration

Spread logging is disabled by default. For best results, keep spread logging disabled so that spread focuses on communication messages. In some situations, it might be necessary to enable spread logging to debug a particular issue.

Set spread incidentals using the following statement:

=> SELECT set_spread_options ('logfile', ‘debugflags’, ‘extras’);

Disable spread logging using the following statement:

=> SELECT set_spread_options('/dev/null/,'EXIT','ExitOnIdle = yes');

Enable spread logging using the following statement:

=> SELECT set_spread_options('/opt/vertica/log/spread.log', 'MEMBERSHIP PRINT CONFIGURATION GROUPS SESSION EXIT', 
'ExitOnIdle = yes');

Viewing and Reloading Spread

Spread uses the spread.conf file, located in the catalog folder. If you make changes to the spread configuration, you must generate a new spread.conf file, and reload spread for the changes to take effect.

To see the catalog view of the spread configuration, use the following statement:

=> SELECT generate_spread_config ();

To write out the spread.conf on the initiator node, use the following statement:

=> SELECT write_spread_config('');

To reload the spread daemons, use the following statement:

=> SELECT reload_spread (true);
reload_spread
---------------
Reloaded
(1 row)

After you have reloaded the spread daemons, you must stop the database and restart it for the new spread configuration to take effect.

spread.conf

The following show example entries in the spread .conf file for broadcast and point-to-point modes.

Broadcast

Note that this example has one Spread_Segment sending messages to several nodes:

# 7
# Auto-generated by vertica - do not edit
ActiveIPVersion = IPv4
Spread_Segment 198.51.100.255:4803 {
 N0119851100202    198.51.100.202 {
  198.51.100.202
 }
N0119851100203    198.51.100.203 {
  198.51.100.203
 }
N0119851100204    198.51.100.204 {
  198.51.100.204 
 }
N0119851100205    198.51.100.205 {
  198.51.100.205
 }
}
# begin end matter
EventLogFile = /dev/null
EventTimeStamp = "[%a %d %b %Y %H:%M:%S]"
DebugFlags = { PRINT EXIT }
ExitOnIdle = yes

Point-to-Point

Note that this example has several Spread_Segments sending messages to individual nodes:

# 7
# Auto-generated by vertica - do not edit
ActiveIPVersion = IPv4
Spread_Segment 198.51.100.42:4803 {
N011985110042 198.51.100.42 {
198.51.100.42
}
}
Spread_Segment 198.51.100.59:4803 {
N011985110059 198.51.100.59 {
198.51.100.59
}
}
Spread_Segment 198.51.100.156:4803 {
N0119851100156 198.51.100.156 {
198.51.100.156
}
}
Spread_Segment 198.51.100.189:4803 {
N0119851100189 198.51.100.189 {
198.51.100.189
}
}
# begin end matter
EventLogFile = /dev/null
EventTimeStamp = "[%a %d %b %Y %H:%M:%S]"
DebugFlags = { PRINT EXIT }
ExitOnIdle = yes