What Should I do When the Database Process is not Starting?

Posted June 14, 2018 by Soniya Shah, Information Developer

If you want to troubleshoot why the database process is not starting, follow this checklist.
Step Task Results
1 Ensure Vertica is not already running on any node. $ ps –ef | grep vertica The Vertica process displays as follows: /opt/vertica/bin/vertica -D <catalog directory> -C <dbname> -n <node name> -h <host IP> -p <port> Prior to proceeding to any further checks, ensure the dbadmin is the owner of and has access to the catalog directory and the data directory. If Vertica is not running on any nodes, go to Step 2.
2 Check whether the previous database shutdown was clean. $ cat <catalog directory of each node>/epoch.log The presence of Epoch.log indicates the shutdown was clean. If the shutdown is clean, go to Step 3. If the shutdown is not clean, start Vertica database. At times, you may be prompted to start Vertica at a Last Good Epoch (LGE).
3 Check whether all the nodes are connected to each other using passwordless SSH. $ ssh <IP address of the node> A successful connection displays last login details with date, day, and time stamp. An unsuccessful connection prompts you for a password and displays “No route to host”. If the connection is using passwordless SSH, go to Step 4. If the SSH needs password, see Enable Secure Shell (SSH) Logins.
4 Check whether spread is communicating with all the nodes. $ /opt/vertica/spread/bin/spuser -s spreadport@<local machine's IP address> If the spread is communication with other nodes, go to Step 5. If the spread is not communicating with other nodes, see the Spread Debugging Checklist.
5 Check whether nothing else is using/bound to the Vertica ports. $ netstat –nap | grep 5433 $ netstat –nap | grep 5434 $ netstat –nap | grep 4803 If Vertica is bound to Vertica ports, kill all the processes holding onto Vertica ports.
6 Check for errors in the following locations. a. Admintools log. $ /opt/vertica/logs/admintools.log b. Vertica log and dbLog. $ /home/dbadmin/<databasename>/<nodename>_catalog/ $ /home/dbadmin/<databasename> c. Spread log. By default, spread logging is disabled. To enable spread logging, edit the spread configuration file. $ /home/dbadmin/<databasename>/<nodename>/spread.log d. PANICs errors in ErrorReport.txt $ tail /home/dbadmin/<databasename>/<nodename>_catalog/ErrorReport.txt e. Operating System logs $ /var/log/messages f. Kernel log $ dmesg -T Resolve the errors in the above logs. If you cannot resolve the error contact Vertica Technical Support with scrutinize.
7 Check whether database process is starting after addressing the above errors, if any. $ /opt/vertica/bin/admintools -t view_cluster -d ; If the database process is restarting, the checklist is complete. If the database process is not restarting, go to Step 8.
8 a.If node cannot find the LGE, database status changes to Lost Contact and admintools suggests recovery after unclean shutdown. See Failure Recovery in the Vertica Documentation. b.If database is still unable to start, there may be corruption or loss of data. In some cases, Vertica can still start with lost/partial data. If that is acceptable then use the following command: $ /opt/vertica/bin/admintools -t start_db –d <databasename> --force Please contact Vertica Technical Support.

Learn More

Learn more about Restarting the Database in the Vertica Documentation.