If you want to troubleshoot why the database process is not starting, follow this checklist.
| Step | Task | Results |
|---|---|---|
| 1 | Ensure Vertica is not already running on any node.
$ ps –ef | grep vertica
The Vertica process displays as follows:
/opt/vertica/bin/vertica -D <catalog directory> -C <dbname> -n <node name> -h <host IP> -p <port>
Prior to proceeding to any further checks, ensure the dbadmin is the owner of and has access to the catalog directory and the data directory. |
If Vertica is not running on any nodes, go to Step 2. |
| 2 | Check whether the previous database shutdown was clean.
$ cat <catalog directory of each node>/epoch.log |
The presence of Epoch.log indicates the shutdown was clean. If the shutdown is clean, go to Step 3. If the shutdown is not clean, start Vertica database. At times, you may be prompted to start Vertica at a Last Good Epoch (LGE). |
| 3 | Check whether all the nodes are connected to each other using passwordless SSH.
$ ssh <IP address of the node> |
A successful connection displays last login details with date, day, and time stamp. An unsuccessful connection prompts you for a password and displays “No route to host”. If the connection is using passwordless SSH, go to Step 4. If the SSH needs password, see Enable Secure Shell (SSH) Logins. |
| 4 | Check whether spread is communicating with all the nodes.
$ /opt/vertica/spread/bin/spuser -s spreadport@<local machine's IP address> |
If the spread is communication with other nodes, go to Step 5. If the spread is not communicating with other nodes, see the Spread Debugging Checklist. |
| 5 | Check whether nothing else is using/bound to the Vertica ports.
$ netstat –nap | grep 5433
$ netstat –nap | grep 5434
$ netstat –nap | grep 4803 |
If Vertica is bound to Vertica ports, kill all the processes holding onto Vertica ports. |
| 6 | Check for errors in the following locations.
a. Admintools log.
$ /opt/vertica/logs/admintools.log
b. Vertica log and dbLog.
$ /home/dbadmin/<databasename>/<nodename>_catalog/
$ /home/dbadmin/<databasename>
c. Spread log.
By default, spread logging is disabled. To enable spread logging, edit the spread configuration file.
$ /home/dbadmin/<databasename>/<nodename>/spread.log
d. PANICs errors in ErrorReport.txt
$ tail /home/dbadmin/<databasename>/<nodename>_catalog/ErrorReport.txt
e. Operating System logs
$ /var/log/messages
f. Kernel log
$ dmesg -T |
Resolve the errors in the above logs. If you cannot resolve the error contact Vertica Technical Support with scrutinize. |
| 7 | Check whether database process is starting after addressing the above errors, if any.
$ /opt/vertica/bin/admintools -t view_cluster -d ; |
If the database process is restarting, the checklist is complete. If the database process is not restarting, go to Step 8. |
| 8 | a.If node cannot find the LGE, database status changes to Lost Contact and admintools suggests recovery after unclean shutdown. See Failure Recovery in the Vertica Documentation.
b.If database is still unable to start, there may be corruption or loss of data. In some cases, Vertica can still start with lost/partial data. If that is acceptable then use the following command:
$ /opt/vertica/bin/admintools -t start_db –d <databasename> --force |
Please contact Vertica Technical Support. |
Learn More
Learn more about Restarting the Database in the Vertica Documentation.About the Author
Soniya Shah
Information Developer
Currently, a first year law student with a background in science and technology. Experienced technical writer, with specializations in software documentation, big data, blog development, and website development. I build user-centered content to communicate complex and technical information more easily.
I used to work for Vertica full time for about 3 years. I still work at Vertica part time while going to law school.
Update: Soniya is now doing her law internship, and no longer working at Vertica. Good luck, Soniya!
