Hyperledger fabric peers panic due to previous block hash mismatch

Question

I have a hyperledger fabric network with 2 organizations and 2 peers per organization. I restored the network from backup and the peers started throwing the following error:

2021-01-28 09:49:55.374 UTC [gossip.state] commitBlock -> ERRO 26b Got error while committing(unexpected Previous block hash. Expected PreviousHash = [91e8aafc47e2f521afc8a52d44b80c60fe781084dec9e1b92ab40d8b6138e7d0], PreviousHash referred in the latest block= [042c3059d75a1622fea88f7f2a5f268363206004ebca635118039e0496b83197]
github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage.(*blockfileMgr).addBlock
        /opt/gopath/src/github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage/blockfile_mgr.go:254
github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage.(*fsBlockStore).AddBlock
        /opt/gopath/src/github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage/fs_blockstore.go:42
github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).CommitWithPvtData
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:132
github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).CommitWithPvtData
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger.go:312
github.com/hyperledger/fabric/core/ledger/ledgermgmt.(*closableLedger).CommitWithPvtData
        <autogenerated>:1
github.com/hyperledger/fabric/core/committer.(*LedgerCommitter).CommitWithPvtData
        /opt/gopath/src/github.com/hyperledger/fabric/core/committer/committer_impl.go:93
github.com/hyperledger/fabric/gossip/privdata.(*coordinator).StoreBlock
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/privdata/coordinator.go:243
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).commitBlock
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:810
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).deliverPayloads
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:598
runtime.goexit
        /opt/go/src/runtime/asm_amd64.s:1333
commit failed
github.com/hyperledger/fabric/gossip/privdata.(*coordinator).StoreBlock
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/privdata/coordinator.go:246
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).commitBlock
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:810
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).deliverPayloads
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:598
runtime.goexit
        /opt/go/src/runtime/asm_amd64.s:1333
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).commitBlock
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:811
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).deliverPayloads
        /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:598
runtime.goexit
        /opt/go/src/runtime/asm_amd64.s:1333)

I am not sure what went wrong but all 4 peers started throwing the error. Any suggestion on what should be done next? Thanks

question from:https://stackoverflow.com/questions/65934836/hyperledger-fabric-peers-panic-due-to-previous-block-hash-mismatch