Exporting to Object Stores
Object-store file systems (S3, Google Cloud Storage, and Azure Blob Storage) have some differences from other file systems that affect data export. You must set some additional configuration parameters for authentication and region, and there are some restrictions on the output.
URI formats and configuration parameters are described on the following reference pages:
Configuration parameters affect all access to the corresponding object stores, both reads and writes. Instead of setting them globally, you can limit the effects of your settings by setting them at the session level before exporting data.
Output Restrictions
Object-store file systems do not support renaming files in place; they implement a rename as a copy followed by a delete. On other file systems, EXPORT TO PARQUET supports atomicity by writing its output into a temporary directory and renaming it when complete. Such an approach is impractical for object stores, so EXPORT TO PARQUET writes directly to the destination path. It is therefore possible to begin reading the exported data before the export has finished, which could lead to errors. Be careful to wait for the export to finish before using the data.
On other file systems, Vertica retries a failed query if the database is K-safe. When exporting to an object store, Vertica does not retry the export and instead reports an error. For example, if a node goes down during export, Vertica returns Error 4142 (Node failure during execution).
If you cancel an export or an export fails, Vertica does not clean up the partial output. On cloud object stores that partial output costs you money, so after a cancellation or failure, delete the output files. Further, because the output path must not exist, you would need to delete the partial output before retrying the export with the same destination.
Vertica does not support simultaneous exports to the same path in object stores. The results are undefined.
S3 limits buckets to 5TB. You might need to divide very large exports.