Vertica Integration for Apache Spark
You can use the Vertica Connector for Apache Spark to transfer data between Vertica and Apache Spark. The following table shows the versions Apache Spark and Scala the Connector supports as well as the name of the Spark Connector JAR file to use for each combination:
|
Apache Spark Version |
Scala Version | Spark Connector JAR file |
|---|---|---|
| 2.0* | 2.11 | vertica-spark2.0_scala2.11.jar
|
| 2.1* | 2.11 | vertica-spark2.1_scala2.11.jar
|
| 2.2 | 2.11 | vertica-spark2.1_scala2.11.jar
|
| 2.3 | 2.11 | vertica-spark2.1_scala2.11.jar
|
* Vertica is phasing out support for this Apache Spark version. See End-of-Support Notices for more information.
Notes
- A Spark Connector JAR file can support multiple versions of Spark. For example,
vertica-spark2.1_scala2.11.jarsupports Spark 2.1, 2.2, and 2.3. - Vertica recommends you always use the version of the Spark Connector shipped with your version of the Vertica server. When you upgrade your Vertica server, you should also upgrade your version of the Spark Connector.
For more information on Apache Spark integration, refer to Integrating with Apache Spark .