What Should I do When the Database Node is DOWN?

Posted June 14, 2018 by Soniya Shah, Information Developer

When database node is DOWN, troubleshoot using the following checklist.
Step Task Results
1 Check whether your database is UP. $ admintools -t db_status -s UP If the database is UP, go to Step 2. If the database if not UP, restart your database. $ admintools -t start_db -d <Database_name> -p <Database_password> If the database starts, the checklist is complete. If the database does not start, see the Database Process Not Starting checklist.
2 Identify all the DOWN nodes. => SELECT node_name, node_address, node_state FROM nodes WHERE node_state = 'DOWN'; Upon identification of all the DOWN node/s, proceed to Step 3.
3 Check whether you can establish a connection with the DOWN nodes using SSH. $ ssh dbadmin @<nodedown_ip> If you can SSH into the node, restart Vertica process on the DOWN node. $ admintools -t restart_node -d <database_name> -s <node_host_name or IP>
  • If the restart was successful, the checklist is complete.
  • If the restart failed, go to Step 4.
If you cannot SSH into the node, contact your system administrator to find if it is a port issue or a network issue.
4 To find reasons for restart failure, on the DOWN node tail startup.log. A snippet of the tail startup.log is as follows: $ tail -f catalog-path/database-name/v_database-name_node_catalog/startup.log { "node" : "v_cdmt0_node0001", "stage" : "Database Halted", "text" : Data consistency problems found; Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data. "timestamp" : "2016-07-31 18:17:04.122" } The log results shows the latest state of the DOWN node. Proceed to Step 5 to see these stages.
5 If the startup.log… a.
    • Remains in the
Waiting for cluster invitestage.
See the Spread Debugging Checklist.
b. Remains in the Recovery stage. See the Node Recovery Checklist.
c. Shows an error message, Data inconsistency problems found, restart the node with the force option. $ admintools -t restart_node -d Database_name -s node_name --force Upon restart the checklist is complete.
d. Shows no new data in the startup.log after last restart, check dbLog file for errors. If you cannot resolve errors, contact Vertica Support.
e. Shows Shutdown Complete but the node is still DOWN. tail vertica.log and look for <ERROR> and <PANIC>. Contact Vertica Support with PANIC report, ErrorReport.txt, and scrutinize.

Learn More

Learn more about Node Down in the Vertica Documentation.