Getting the Spark Connector

The Vertica Connector for Apache Spark is packaged as a JAR file. In addition to the connector JAR file, you also need the Vertica JDBC client library. The connector uses this library to connect to the Vertica database.

You can download the Spark Connector from the myVertica portal in the Downloads section. The Vertica JDBC client library is also available from the myVertica portal in the client driver downloads section.

Choosing the Correct Connector Version

The connector JAR file is compatible with specific versions of Vertica, Apache Spark, and Scala. The connector is compatible with versions of Vertica later than 8.0. See Vertica Integration for Apache Spark in the Supported Platforms document.

You must download the connector file that matches the versions in your installation. For example, suppose you want to connect your Vertica 8.1.0 database to your Spark 2.0 cluster which has Scala version 2.11 installed on it. Then you need the connector JAR file named vertica-8.1.0_spark2.0_scala2.11.jar.

Note: The Vertica version number in the connector's file name refers to the Vertica server version in which the connector was released. Connector versions may be backwards compatible with earlier versions of the Vertica server. Vertica Integration for Apache Spark in Supported Platforms tells you which versions of Vertica the latest connector supports.

You can determine your Spark and Scala version by starting a Spark shell:

$ spark-shell
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://node01:4040
Spark context available as 'sc' (master = local[*], app id = local-1488824765565).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.0.0.2.5.3.0-37
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77)
Type in expressions to have them evaluated.
Type :help for more information.

The startup messages contain the version numbers of both Spark and Scala (shown in bold in the previous example for clarity).