Troubleshooting Kerberos Authentication

These tips can help you avoid issues related to Kerberos authentication with Vertica Analytic Database and to troubleshoot any problems that occur.

JDBC Client Authentication Fails

If Kerberos authentication fails on a JDBC client, check the JAAS login configuration file for syntax issues. If syntax is incorrect, authentication fails.

Working Domain Name Service (DNS) Not Configured

Verify that the DNS entries and the system host file (/etc/hosts or /etc/hostnames) on the network are all properly configured for your environment. If you are using a fully qualified domain name, ensure that is properly configured as well. Refer to the Kerberos documentation for your platform for details.

System Clocks Out of Sync

All Systems Except Red Hat 7/CentOS 7

System clocks in your network must remain in sync for Kerberos authentication to work properly. To do so:

Install NTP on the Kerberos server (KDC).
Install NTP on each server in your network.
Synchronize system clocks on all machines that participate in the Kerberos realm within a few minutes of the KDC and each other

Clock skew can be a problem on Linux virtual machines that need to sync with the Windows Time Service. Use the following stepsto keep time in sync:

Using any text editor, open /etc/ntp.conf.
Under the Undisciplined Local Clock section, add the IP address for the Vertica Analytic Database server. Then, remove existing server entries.
Log in to the server as root, and set up a cron job to sync time with the added IP address every half hour, or as often as needed. For example:
```
# 0 */2 * * * /etc/init.d/ntpd restart
```
Alternatively, run the following command to force clock sync immediately:
```
$ sudo /etc/init.d/ntpd restart
```

For more information, see Enabling Network Time Protocol (NTP) in Installing Vertica and the Network Time Protocol website.

Red Hat 7/CentOS 7 Systems

In Red Hat 7/CentOS 7, ntpd is deprecated in favor of chrony. To keep system clocks in your network in sync for Kerberos authentication to work properly, do the following:

Install chrony on the Kerberos server (KDC).
Install chrony on each server in your network.
Synchronize system clocks on all machines that participate in the Kerberos realm within a few minutes of the KDC and each other.

Clock Skew on Linux Virtual Machines

Clock skew can be problematic on Linux virtual machines that need to sync with the Windows Time Service. Try the following to keep time in sync:

Using any text editor, open /etc/chrony.conf.
Under the Undisciplined Local Clock section, add the IP address for the Vertica Analytic Database server. Then, remove existing server entries.
Log in to the server as root, and set up a cron job to sync time with the added IP address every half hour, or as often as needed. For example:
```
# 0 */2 * * * systemctl start chronyd
```
Alternatively, run the following command to force clock sync immediately:
```
$ sudo systemctl start chronyd
```

For more information, see the Red Hat chrony guide.

Kerberos Ticket Is Valid, But Hadoop Access Fails

Vertica uses Kerberos tickets to obtain Hadoop tokens. It then uses the Hadoop tokens to access the Hadoop data. Hadoop tokens expire after a period of time, so Vertica periodically refreshes them. However, if your Hadoop cluster is set to expire tokens frequently, it is possible that tokens might not be refreshed in time. If the token expires, you cannot access data.

Setting the HadoopFSTokenRefreshFrequency configuration parameter allows you to specify how often Vertica should refresh the token. Specify this value, in seconds, to be smaller than the expiration period set for Hadoop. For example:

=> ALTER DATABASE exampledb SET HadoopFSTokenRefreshFrequency = '86400';

Encryption Algorithm Choices

Kerberos is based on symmetric encryption. Be sure that all Kerberos parties used in the Kerberos realm agree on the encryption algorithm to use. If they do not agree, authentication fails. You can review the exceptions in the vertica.log.

On a Windows client, be sure the encryption types match the types set on Active Directory. See Configure Vertica for Kerberos Authentication.

Be aware that Kerberos is used only for securing the login process. After the login process completes, by default, information travels between client and server without encryption. If you want to encrypt traffic, use SSL. For details, see Implementing SSL.

Kerberos Passwords Not Recognized

If you change your Kerberos password, you must re-create all of your keytab files.

Using the ODBC Data Source Configuration Utility

On Windows vsql clients, you may choose to use the ODBC Data Source Configuration utility and supply a client Data Source. If so, be sure you enter a Kerberos host name in the Client Settings tab to avoid client connection failures with the Vertica Analytic Database server.

Authentication Failure in Backup, Restore, or Admin Tools

This problem can arise in configurations where each Vertica node uses its own Kerberos principal. (This configuration is recommended.) When using vbr or admintools you might see an error such as the following:

$ vsql: GSSAPI continuation error: Miscellaenous failure
GSSAPI continuation error: Server not found in Kerberos database

Backup/restore and the admin tools use the value of KerberosHostname, if it is set, in the Kerberos principal used to authenticate. The same value is used on all nodes. If you have defined one Kerberos principal per node, as recommended, this value does not match. To correct this, unset the KerberosHostname parameter:

=> ALTER DATABASE mydb CLEAR KerberosHostname;

Server's Principal Name Does Not Match Host Name

This problem can arise in configurations where a single Kerberos principal is used for all nodes. OpenText recommends against using a single Kerberos principal for all nodes. Instead, use one principal per node and do not set the KerberosHostname parameter.

In some cases during client connection, the Vertica server's principal name might not match the host name in the connection string. (See also Using the ODBC Data Source Configuration Utility in this topic.)

On ODBC, JDBC, and ADO.NET clients, set the host name portion of the server's principal using the KerberosHostName connection string.

On vsql clients, you set the host name portion of the server's principal name using the -K KRB HOST command-line option. The default value is specified by the -h switch, which is the host name of the machine on which the Vertica server is running. -K is equivalent to the drivers' KerberosHostName connection string value.

For details, see Command Line Options in Connecting to Vertica.

Principal/Host Mismatch Issues and Resolutions

This section lists potential issues that may occur if the principal and host are mismatched.

The KerberosHostName configuration parameter has been overridden.

For example, consider the following connection string:
```
jdbc:vertica://v_vmart_node0001.example.com/vmart?user=kuser
```
Because the this connection string includes no explicit KerberosHostName parameter, the driver defaults to the host in the URL (v_vmart_node0001.example.com). If you overwrite the server-side KerberosHostName parameter as “abc”, the client generates an incorrect principal.

To resolve this issue, explicitly set the client’s KerberosHostName to the connection string, as in this example:
```
jdbc:vertica://v_vmart_node0001.example.com/vmart?user=kuser&kerberoshostname=abc
```
Connection load balancing is enabled, but the node against which the client authenticates might not be the node in the connection string.

In this situation, consider changing all nodes to use the same KerberosHostName setting. When you use the default to the host that was originally specified in the connection string, load balancing cannot interfere with Kerberos authentication.
You have a DNS name that does not match the Kerberos host name.

For example, imagine a cluster of six servers, where you want hr-servers and finance-servers to connect to different nodes on the Vertica Analytic Database cluster. Kerberos authentication, however, occurs on a single (the same) KDC. In the following example, the Kerberos service host name of the servers is server.example.com.

Suppose you have the following list of example servers:
```
server1.example.com 192.16.10.11
server2.example.com 192.16.10.12
server3.example.com 192.16.10.13
server4.example.com 192.16.10.14
server5.example.com 192.16.10.15
server6.example.com 192.16.10.16
```
Now, assume you have the following DNS entries:
```
finance-servers.example.com 192.168.10.11, 192.168.10.12, 192.168.10.13 
hr-servers.example.com 192.168.10.14, 192.168.10.15, 192.168.10.16
```
When you connect to finance-servers.example.com, specify:
- Kerberos -h host name option as server.example.com
- -K host option for hr-servers.example.com
For example:
```
$ vsql -h finance-servers.example.com -K server.example.com
```
You do not have DNS set up on the client machine, so you must connect by IP only.

To resolve this issue, specify:
- Kerberos -h host name option for the IP address
- -K host option for server.example.com
For example:
```
$ vsql -h 192.168.1.12 -K server.example.com
```
There is a load balancer involved (Virtual IP), but there is no DNS name for the VIP.

Specify:
- Kerberos -h host name option for the Virtual IP address
- -K host option for server.example.com
For example:
```
$ vsql -h <virtual IP> -K server.example.com
```
You connect to Vertica using an IP address, but there is no host name to construct the Kerberos principal name.

Provide the instance or host name for the Vertica as described in Inform Vertica About the Kerberos Principal

You set the server-side KerberosHostName configuration parameter to a name other than the Vertica node's host name, but the client cannot determine the host name based on the host name in the connection string alone.

Rename the KerberosHostName

Rename the KerberosHostName to match the name of the Vertica node's host name. For more information, see the following topics in Connecting to Vertica: