HDFS File System (Libhdfs++)

HDFS is the Hadoop Distributed File System. This page describes the hdfs scheme, which uses the Libhdfs++ library. You can also access data in HDFS through WebHDFS. Regardless of which scheme you use, the WebHDFS service must be available. For more information about WebHDFS, see the Hadoop documentation.

URI Format

One of the following:

  • hdfs://[nameservice]/path
  • hdfs://namenode-host:port/path

Characters may be URL-encoded (%NN where NN is a two-digit hexadecimal number) but are not required to be, except that the '%' character must be encoded.

To use the default name service specified in the HDFS configuration files, omit nameservice and use hdfs:///. Use this shorthand only for reading external data, not for creating a storage location.

Always specify a name service or host explicitly when using Vertica with more than one HDFS cluster. The name service or host name must be globally unique. Using hdfs:/// could produce unexpected results because Vertica uses the first value of fs.defaultFS that it finds.


Vertica can use Kerberos authentication with Cloudera or Hortonworks HDFS clusters. See Accessing Kerberized HDFS Data.

Configuration Parameters

The following database configuration parameters apply to the HDFS file system. You can set parameters globally and for the current session with ALTER DATABASE…SET PARAMETER and ALTER SESSION…SET PARAMETER, respectively. For more information about these parameters, see Apache Hadoop Parameters.

Parameter Description

Boolean, specifies whether to distribute block location metadata collected during planning on the initiator to all database nodes for execution, reducing NameNode contention. Disabled by default.


Directory path containing the XML configuration files copied from Hadoop. The same path must be valid on every Vertica node. The files are accessed by the Linux user under which the Vertica server process runs.

HadoopImpersonationConfig Session parameter specifying the delegation token or Hadoop user for HDFS access. See HadoopImpersonationConfig Format for information about the value of this parameter and Proxy Users and Delegation Tokens for more general context.

Boolean, specifies whether to use the webhdfs scheme instead of hdfs, regardless of the URI. If you do not specifically need a feature not supported in the hdfs scheme, you should not change the value of this parameter.

Configuration Files

The path specified in HadoopConfDir must include a directory containing the files listed in the following table. Vertica reads these files at database start time. If you do not set a value, Vertica looks for the files in /etc/hadoop/conf.

If a property is not defined, Vertica uses the defaults shown in the table. If no default is specified for a property, the configuration files must specify a value.

File Properties Default
core-site.xml fs.defaultFS none

(for doAs users:) hadoop.proxyuser.username.users

(for doAs users:) hadoop.proxyuser.username.hosts none
hdfs-site.xml dfs.client.failover.max.attempts 15
dfs.client.failover.sleep.base.millis 500
dfs.client.failover.sleep.max.millis 15000
(For HA NN:) dfs.nameservices none
(WebHDFS:) dfs.namenode.http-address or dfs.namenode.https-address none
(WebHDFS:) dfs.datanode.http.address or dfs.datanode.https.address none

If using High Availability (HA) NameNodes, the individual NameNodes must also be defined in hdfs‑site.xml

If you are using Eon Mode with communal storage on HDFS, then if you set dfs.encrypt.data.transfer you must use the swebhdfs scheme for communal storage.

To verify that Vertica can find configuration files in HadoopConfDir, use the VERIFY_HADOOP_CONF_DIR function.

To test access through the hdfs scheme, use the HDFS_CLUSTER_CONFIG_CHECK function.

For more information about testing your configuration, see Verifying HDFS Configuration.

To reread the configuration files, use the CLEAR_HDFS_CACHES function.


The following example creates an external table using data in HDFS, specifying a name service.

	AS COPY FROM 'hdfs://hadoopNS/data/users.csv';