Hybrid Kubernetes Clusters

An Eon Mode database can run hosts separate from the database and within Kubernetes. This architecture is useful in scenarios where you want to:

• Leverage Kubernetes tooling to quickly create a secondary subcluster for a database.
• Create an isolated sandbox environment to run ad hoc queries on a communal dataset.
• Experiment with the Vertica on Kubernetes performance overhead without migrating your primary subcluster into Kubernetes.

Define the Kubernetes portion of a hybrid architecture with a custom resource. The custom resource has no knowledge of Vertica hosts that exist separately from the custom resource. This limits the operator's functionality and requires that you manually complete some tasks that the operator automates for a standard Vertica on Kubernetes custom resource.

Requirements and Restrictions

The hybrid Kubernetes architecture has the following requirements and restrictions:

• Hybrid Kubernetes clusters require a tool that enables Border Gateway Protocol (BGP) so that pods are accessible to your on-premises subcluster for external communication. For example, you can use the Calico CNI plugin to enable BGP.
• You cannot use network address translation (NAT) between the Kubernetes pods and the on-premises cluster.

Operator Limitations

In a hybrid architecture, the operator has no visibility outside of the custom resource. This limited visibility means that the operator cannot interact with the Eon Mode database or the primary subcluster. Within the scope of the custom resource, the operator automates only the following:

• Schedules pods based on the manifest.
• Creates service objects for the subcluster.
• Creates a PersistentVolumeClaim (PVC) that persists data for each pod.
• Executes the restart_node administration tool command if the Vertica server process is not running. To override this default behavior, set autoRestartVertica to false.

Manual Tasks

Because of the limited operator functionality, the administrator must manually perform the following tasks:

• Restart the cluster if quorum is lost. For details about maintaining quorum, see Data Integrity and High Availability in an Eon Mode Database.

• Execute the update_vertica script to set up the configuration directory. Vertica on Kubernetes requires the following configuration options for update_vertica:

  $ /opt/vertica/sbin/update_vertica \
      --accept-eula \
      --add-hosts host-list \
      --dba-user su-passwd-Secret \
      --failure-threshold NONE \
      --no-system-configuration \
      --point-to-point \
      --data-dir /data-dir \
      --dba-user dbadmin \
      --no-package-checks

  In the previous example, the --dba-user option accepts a Secret that is provided in the superuserPasswordSecret configuration parameter.

  After you call update_vertica, use admintools with the db_add_node option to add the nodes and complete the setup:

  $ /opt/vertica/bin/admintools \
      -t db_add_node \
      --hosts host-list \
      --database db-name\
      --subcluster sc-name \
      --noprompt
  

  For details, see Adding and Removing Nodes From Subclusters.

Defining a Hybrid Cluster

Before you define a hybrid cluster, you must create a Secret to store SSH credentials. In an Eon Mode database, nodes communicate through SSH. The Vertica container uses SSH, but the SSH key is regenerated each time a container is built.

The following command creates a Secret named ssh-key that stores SSH credentials that persists between life cycles to allow secure connections between the on-premises nodes and the CR:

$ kubectl create secret generic ssh-keys --from-file=$HOME/.ssh

Create a custom resource to define a subcluster that runs outside your standard Eon Mode database:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: hybrid-secondary-sc
spec:
  image: vertica/vertica-k8s:11.0.2-0-minimal
  initPolicy: ScheduleOnly
  sshSecret: ssh-keys
  subclusters:
    - name: sc1
      size: 3
    - name: sc2
      size: 3

In the previous example:

• initPolicy: Hybrid clusters require that you set this to ScheduleOnly.
• sshSecret: The Secret that contains SSH keys that authenticate connections to Vertica hosts outside the CR.
• dbName: This value must match the name of the standard Eon Mode database that is associated with this subcluster.
• local: Required. The values persist data to the PersistentVolume (PV). These values must match the directory locations in the Eon Mode database that is associated with the Kubernetes pods.
• subclusters: Definition for each subcluster.

For complete implementation details, see Creating a Custom Resource. For details about each setting, see Custom Resource Definition Parameters.