If I’m correct with SDK 3.0 it’s not possible to enable forging on multiple nodes for a single delegate, you must disable forging on all node and then enable it on a single one.
My question is in case a node goes down (power failure, network partition, app crash, hardware issue), how can we disable the forging to enable it on another node ?
Enabling forging on multiple nodes is possible, but it would violate the protocol by producing two identical blocks in the same slot (double forging). If your forging node goes down, it can’t forge (forging is disabled on restart by default). So you only have to enable forging on a different node to continue forging. You would have to have a script that does it for you.
@przemer are you sure you don’t need to inform the network your node is no longer forging ? I’ve read somewhere you must disable on all node before enabling on a new one. Therefore my question.
Obviously failover scripts are still necessary. In my solution I first always disabled forging on delayed node before enabling it on another. Is there now any period that should be considered as safe value between enabling forging on different server? Is status that forging has been disables enough to prevent protocol violation?
It is not enough that the other node does not forge any more. Assume you want to change from node1 to node2. Then you need to basically do the following:
Deactivate forging on node1.
Get last block A forged by node1.
Use last block A to initialize the forging on node2 correctly.
The last step ensures that node2 is on the correct chain and does not cast contradicting consensus votes that could lead to a protocol violation. See the links posted by @michielmulders for details.
Note that Lisk SDK 3.0 uses a new improved consensus algorithm which avoids that nodes do not forge because of low Broadhash consensus (actually there is no Broadhash any more). Therefore, it should not be necessary to have multiple forging servers to guarantee high availability.
If you are worried about DoS attacks, then a sentry architecture good be a good solution:
1 server which forges block, but which keeps it IP address private. This can be achieved by only connecting to trusted fixed peers and not advertise your IP, see LIP 0004 for details.
Multiple servers that do not forge and run default p2p configuration with whitelisting the forging server. This means that the IP address of these servers is broadcast as part of the normal P2P discovery in lisk-p2p. The private forging server only connects these trusted public nodes.
This setup ensures that the IP of the forging server is never known to anyone in the Lisk P2P network except for the trusted public nodes.