Scaling Vertica Pods Horizontally

Beta Feature — For Test Environments Only

Horizontal scaling is when you add or remove nodes or pods from your Kubernetes cluster. Change the replica count in your Vertica StatefulSet to meet your resource requirements and workload needs.

Consider the following when adding or removing Vertica pods to your cluster:

  • Scaling a cluster up and down is a manual process in 10.1.1.
  • AutoScaling is not supported.
  • As a best practice, Vertica recommends that you assign 1 Vertica pod per node. If you add pods to your cluster, you must add host nodes to maintain the 1 Vertica pod per node relationship.

Adding Vertica Pods to a Cluster

If the number of replicas exceeds 120, you must manually configure Vertica for large cluster support.

  1. Access a shell within a pod. The following command uses the kubectl CLI tool to access a shell within the pod-0 pod:

    $ kubectl exec -it pod-0 -- /bin/bash

  2. Verify the current host IP addresses in your Vertica cluster with the admintools and the list_allnodes option:

    [dbadmin@pod-0 /]$ admintools -t list_allnodes
 Node                 | Host        | State | Version        | DB
----------------------+-------------+-------+----------------+-----------
 v_verticadb_node0001 | 10.20.30.40 | UP    | vertica-10.1.1 | verticadb
 v_verticadb_node0002 | 10.20.1.130 | UP    | vertica-10.1.1 | verticadb
 v_verticadb_node0003 | 10.20.2.210 | UP    | vertica-10.1.1 | verticadb

  3. In a different terminal window outside the pod, use the helm upgrade command to increase the number of pods in your cluster.

    The following command uses the --set option to increase the number of pods from 3 to 6 for a release named my-release:

    $ helm upgrade my-release vertica-charts/vertica --set subclusters.defaultsubcluster.replicaCount=6

  4. After a few seconds, there are 3 new pods running. Confirm the STATUS for the new pods with kubectl get pods:

    $ kubectl get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
...
cluster-vertica-defaultsubcluster-3                     1/1     Running     0          61s
cluster-vertica-defaultsubcluster-4                     1/1     Running     0          57s
cluster-vertica-defaultsubcluster-5                     1/1     Running     0          53s

  5. Pass deployment information to the update_vertica command to initialize the configuration in the new pods.

    1. Define separate shell variables to store the cluster namespace, and Helm chart release name:

      $ NAMESPACE=my-namespace
$ HELM_RELEASE_NAME=my-release

    2. Use labels and the new HELM_RELEASE_NAME shell variable to define the SELECTOR shell variable:

      $ SELECTOR=app.kubernetes.io/name=vertica,vertica.com/usage=server,app.kubernetes.io/instance=$HELM_RELEASE_NAME

    3. Define a shell variable to store the names of the new hosts.

      When you define the jsonpath option, change the range to reflect the number of nodes that you are adding to the cluster. The following command uses .items[-3:] because we are adding 3 nodes to the cluster:

      $ NEW_HOSTS=$(kubectl get pods -n $NAMESPACE --selector=$SELECTOR -o jsonpath="{range .items[-3:]}{.metadata.name}.{.spec.subdomain},{end}" | sed 's/.$//')

    4. Define a shell variable to store the name of the pod accessed to execute update_vertica:

      $ POD_NAME=$(kubectl get pods --namespace $NAMESPACE --selector=$SELECTOR -o jsonpath="{.items[0].metadata.name}")

    5. Use the new shell variables to execute the update_vertica command. The following command accesses a shell within a pod and adds the new hosts to the cluster:

      $ kubectl --namespace $NAMESPACE exec $POD_NAME -it -- sudo /opt/vertica/sbin/update_vertica --add-hosts $NEW_HOSTS --failure-threshold NONE --no-system-configuration --point-to-point --data-dir /home/dbadmin/docker/data

  6. Add the new hosts to your Vertica database:

    $ kubectl --namespace $NAMESPACE exec $POD_NAME -i -t -- /opt/vertica/bin/admintools -t db_add_node -d database-name -p password -c subcluster-name --hosts $NEW_HOSTS

  7. Verify the nodes were added.

    1. Access a shell within a pod:

      $ kubectl exec --it pod-0 -- /bin/bash

    2. Use admintools and the list_allnodes option to verify the nodes were added:

      [dbadmin@pod-0 /]$ admintools -t list_allnodes
Node                 | Host        | State | Version        | DB
---------------------+-------------+-------+----------------+-----------
v_verticadb_node0001 | 10.20.30.40 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0002 | 10.20.1.130 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0003 | 10.20.2.210 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0004 | 10.20.30.41 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0005 | 10.20.1.132 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0006 | 10.20.2.212 | UP    | vertica-10.1.1 | verticadb

  8. Rebalance the shards in your cluster using vsql:

    $ vsql
=> SELECT REBALANCE_SHARDS('subcluster-name');
 REBALANCE_SHARDS  
-------------------
 REBALANCED SHARDS
(1 row)

Removing Vertica Pods from a Cluster

You might determine that you do not need as many Vertica pods as you currently have deployed, and you need to manually scale the cluster down:

  1. Access a shell within a pod. The following command uses the kubectl CLI tool to access a shell within the pod-0 pod:

    $ kubectl exec -it pod-0 -- /bin/bash

  2. Verify the current host IP addresses in your Vertica cluster with admintools and the list_allnodes option:

    [dbadmin@pod-0 /]$ admintools -t list_allnodes
Node                 | Host        | State | Version        | DB
---------------------+-------------+-------+----------------+-----------
v_verticadb_node0001 | 10.20.30.40 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0002 | 10.20.1.130 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0003 | 10.20.2.210 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0004 | 10.20.30.41 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0005 | 10.20.1.132 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0006 | 10.20.2.212 | UP    | vertica-10.1.1 | verticadb

  3. In a different terminal window outside the pod, initialize the new configuration in the pod.

    1. Define separate shell variables to store the cluster namespace, and Helm chart release name:

      $ NAMESPACE=my-namespace
$ HELM_RELEASE_NAME=mycluster

    2. Use labels and the new HELM_RELEASE_NAME shell variable to define the SELECTOR shell variable:

      $ SELECTOR=app.kubernetes.io/name=vertica,vertica.com/usage=server,app.kubernetes.io/instance=$HELM_RELEASE_NAME

    3. Define a shell variable to store the names of the pods containing the hosts that you want to remove.

      When you define the jsonpath option, change the range to reflect the number of nodes that you are adding to the cluster. The following command uses .items[-3:] because we are removing 3 nodes from the cluster:

      $ REMOVED_HOSTS=$(kubectl get pods -n $NAMESPACE --selector=$SELECTOR -o jsonpath="{range .items[-3:]}{.metadata.name}.{.spec.subdomain},{end}" | sed 's/.$//')

      Kubernetes removes pods with the highest StatefulSet index. You cannot remove a pod from the beginning of a StatefulSet.

    4. Define a shell variable to store the name of a pod that you access to remove hosts from your cluster:

      $ POD_NAME=$(kubectl get pods --namespace $NAMESPACE --selector=$SELECTOR -o jsonpath="{.items[0].metadata.name}")

  4. Use the new shell variables to remove nodes from your Vertica database using admintools and the remove_node option:

    $ kubectl --namespace $NAMESPACE exec $POD_NAME -i -t -- /opt/vertica/bin/admintools -t db_remove_node --database database-name -p password --hosts ${REMOVED_HOSTS}

  5. Run update_vertica with the --remove-hosts option to remove the hosts from the Vertica cluster:

    $ kubectl --namespace $NAMESPACE exec $POD_NAME -i -t -- sudo /opt/vertica/sbin/update_vertica --remove-hosts ${REMOVED_HOSTS}

  6. Scale down the cluster with the helm upgrade command.

    Use the --set option and the subclusters.defaultsubcluster.replicaCount parameter to decrease the number of pods from 6 to 3 for a release named my-release:

    $ helm upgrade my-release helm-charts/vertica --set subclusters.defaultsubcluster.replicaCount=3

  7. Use kubectl to access a shell within a pod:

    $ kubectl exec --it pod-0 -- /bin/bash

  8. Verify the current host IP addresses in your Vertica cluster with the admintools and the list_allnodes option:

    [dbadmin@pod-0 /]$ admintools -t list_allnodes
Node                 | Host        | State | Version        | DB
---------------------+-------------+-------+----------------+-----------
v_verticadb_node0001 | 10.20.30.40 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0002 | 10.20.1.130 | UP    | vertica-10.1.1 | verticadb
v_verticadb_node0003 | 10.20.2.210 | UP    | vertica-10.1.1 | verticadb