What Should I do When the Database Node is DOWN?
When database node is DOWN, troubleshoot using the following checklist.
| Step | Task | Results |
|---|---|---|
| 1 | Check whether your database is UP.
$ admintools -t db_status -s UP
|
If the database is UP, go to Step 2.
If the database if not UP, restart your database. $ admintools -t start_db -d <Database_name> -p <Database_password>
If the database starts, the checklist is complete.
If the database does not start, see the Database Process Not Starting checklist. |
| 2 | Identify all the DOWN nodes.
=> SELECT node_name, node_address, node_state FROM nodes WHERE node_state = 'DOWN'; |
Upon identification of all the DOWN node/s, proceed to Step 3. |
| 3 | Check whether you can establish a connection with the DOWN nodes using SSH.
$ ssh dbadmin @<nodedown_ip>
|
If you can SSH into the node, restart Vertica process on the DOWN node.
$ admintools -t restart_node -d <database_name> -s <node_host_name or IP>
If you cannot SSH into the node, contact your system administrator to find if it is a port issue or a network issue. |
| 4 | To find reasons for restart failure, on the DOWN node tail startup.log. A snippet of the tail startup.log is as follows:
$ tail -f catalog-path/database-name/v_database-name_node_catalog/startup.log
{
"node" : "v_cdmt0_node0001",
"stage" : "Database Halted",
"text" : Data consistency problems found; Check that all file systems are properly mounted.
Also, the --force option can be used to delete corrupted data.
"timestamp" : "2016-07-31 18:17:04.122"
}
| The log results shows the latest state of the DOWN node. Proceed to Step 5 to see these stages. |
| 5 | If the startup.log…
|
See the Spread Debugging Checklist. |
| b. Remains in the Recovery stage. | See the Node Recovery Checklist. | |
c. Shows an error message, Data inconsistency problems found, restart the node with the force option.
$ admintools -t restart_node -d Database_name -s node_name --force
|
Upon restart the checklist is complete. | |
| d. Shows no new data in the startup.log after last restart, check dbLog file for errors. | If you cannot resolve errors, contact Vertica Support. | |
| e. Shows Shutdown Complete but the node is still DOWN. tail vertica.log and look for <ERROR> and <PANIC>. | Contact Vertica Support with PANIC report, ErrorReport.txt, and scrutinize. |
Learn More
Learn more about Node Down in the Vertica Documentation.