Client update error while forking from sovereign to consumer chain

[jstr1121]

Summary of Bug

I am trying to setup relayer using hermes v0.15.0 between two Cosmos chains for interchain-security provider chain and consumer chain after soft fork from sovereign to consumer chain.
To setup minimal version, locally, installed one node provider chain, 2 nodes consumer chain. And when setting up connection this issue persists.

An action done on the first block after soft fork is validator set update to provider chain's one. And this hermes operation started after waiting 10s executing the validator set update block.

ERROR log on hermes

2022-12-28T16:59:35.808947Z ERROR ThreadId(89) send_tx_commit{id=ConnectionOpenTry}:send_tx_with_account_sequence_retry{id=provider}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: cannot update client with ID 07-tendermint-0: trusted validators validators:<address:\"\\355D(\\260\\363S\\246\\014b`\\375\\252\\347H6\\006\\204\\002\\234Z\" pub_key:<ed25519:\"`V\\256\\241\\365D\\376\\306\\277\\227\\271`\\246\\201!\\347\\227v\\311\\022\\350c^\\270\\\\@\\254F\\220\\273\\374F\" > voting_power:10000 > proposer:<address:\"\\355D(\\260\\363S\\246\\014b`\\375\\252\\347H6\\006\\204\\002\\234Z\" pub_key:<ed25519:\"`V\\256\\241\\365D\\376\\306\\277\\227\\271`\\246\\201!\\347\\227v\\311\\022\\350c^\\270\\\\@\\254F\\220\\273\\374F\" > voting_power:10000 > total_voting_power:10000 , does not hash to latest trusted validators. Expected: B6CFE4517DBE86988EE1F71C11F48BA275B5170FF08514DD0AA078FFEBC44DAC, got: B02F61DF3987DDB895F5608E6519A18887E831F4BDB83C1C2D1FDA88AFAB15CB: invalid validator set [informalsystems/ibc-go/v3@v3.4.1-0.20221202165607-3dc5ba251371/modules/light-clients/07-tendermint/types/update.go:156] With gas wanted: '0' and gas used: '100724' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "31"} }
2022-12-28T16:59:35.809033Z ERROR ThreadId(01) failed ConnTry ConnectionSide { chain: BaseChainHandle { chain_id: ChainId { id: "provider", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: None }: failed during a transaction submission step to chain 'provider': gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: cannot update client with ID 07-tendermint-0: trusted validators validators:<address:\"\\355D(\\260\\363S\\246\\014b`\\375\\252\\347H6\\006\\204\\002\\234Z\" pub_key:<ed25519:\"`V\\256\\241\\365D\\376\\306\\277\\227\\271`\\246\\201!\\347\\227v\\311\\022\\350c^\\270\\\\@\\254F\\220\\273\\374F\" > voting_power:10000 > proposer:<address:\"\\355D(\\260\\363S\\246\\014b`\\375\\252\\347H6\\006\\204\\002\\234Z\" pub_key:<ed25519:\"`V\\256\\241\\365D\\376\\306\\277\\227\\271`\\246\\201!\\347\\227v\\311\\022\\350c^\\270\\\\@\\254F\\220\\273\\374F\" > voting_power:10000 > total_voting_power:10000 , does not hash to latest trusted validators. Expected: B6CFE4517DBE86988EE1F71C11F48BA275B5170FF08514DD0AA078FFEBC44DAC, got: B02F61DF3987DDB895F5608E6519A18887E831F4BDB83C1C2D1FDA88AFAB15CB: invalid validator set [informalsystems/ibc-go/v3@v3.4.1-0.20221202165607-3dc5ba251371/modules/light-clients/07-tendermint/types/update.go:156] With gas wanted: '0' and gas used: '100724' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "31"} }
Error: failed due to missing counterparty connection id

Version

$ hermes version
hermes 0.15.0

Steps to Reproduce

The error can be reproduced here by running follow commands
Stride-Labs/interchain-security#1

rm -rf $HOME/.provider1
rm -rf $HOME/.provider
rm -rf $HOME/.consumer1
rm -rf $HOME/.consumer
rm -rf $HOME/.sovereign
sh run.sh

Acceptance Criteria

1. Find the solution to bypass this issue with current version
2. Figure out the reason of the issue

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[jstr1121]

Where is hermes fetching validators list?

[romac]

@jtremback @sainoe You guys should probably take a look at this

[ancazamfir]

I managed to repro the issue, made some local changes to Stride-Labs/interchain-security#1 to be able to run with hermes v1.2.0.
The connection handshake fails because the update of the consumer client on the provider chain fails. It is not clear to me how or where in the code the consumer client is created on the provider (how does it know the valset at height 1 for a sovereign chain?). If I query the consensus state on the provider chain:

$ hermes query client consensus --chain provider --client 07-tendermint-0
SUCCESS [
    AnyConsensusStateWithHeight {
        height: Height {
            revision: 0,
            height: 1,
        },
        consensus_state: Tendermint(
            ConsensusState {
                timestamp: Time(
                    2023-01-04 3:04:06.419233,
                ),
                root: CommitmentRoot(
                    "73656E74696E656C5F726F6F74",
                ),
                next_validators_hash: Hash::Sha256(1F133303DC5AAA1A8A06ECA5E0D902073DFF7DE7C8F400AD98AED121E115DDDD),
            },
        ),
    },
]

This says that the header at height 1 on consumer chain had next_validator_hash: 1F1333..
I looked a bit at the chain headers and the validator set doesn't change until height 9, then something happens at 9 (idk, maybe the sovereign chain became consumer) and the valset changes 100%.
After this, if we try to update the client, we see the hash errors (see Expected: 1F1333.., got: 50854A..):

$ hermes update client --client 07-tendermint-0 --host-chain provider
...
2023-01-04T04:18:25.969440Z ERROR ThreadId(19) foreign_client.build_update_client_and_send{client=consumer->provider:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=provider tracking_id=update client}:send_tx_with_account_sequence_retry{chain=provider account.sequence=4}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: cannot update client with ID 07-tendermint-0: trusted validators validators:<address:\"\\010\\251\\345\\341\\210\\256T$A\\325\\303\\231\\345tWiLL\\013!\" pub_key:<ed25519:\"|\\376\\324:\\236\\344>\\370\\352\\305\\000|\\241\\335/\\353\\244\\314\\377\\351\\207\\331\\214\\256~\\250ZL\\311-\\000\\324\" > voting_power:10000 > proposer:<address:\"\\010\\251\\345\\341\\210\\256T$A\\325\\303\\231\\345tWiLL\\013!\" pub_key:<ed25519:\"|\\376\\324:\\236\\344>\\370\\352\\305\\000|\\241\\335/\\353\\244\\314\\377\\351\\207\\331\\214\\256~\\250ZL\\311-\\000\\324\" > voting_power:10000 > total_voting_power:10000 , does not hash to latest trusted validators. Expected: 1F133303DC5AAA1A8A06ECA5E0D902073DFF7DE7C8F400AD98AED121E115DDDD, got: 50854A8583996C322A26AD8982724A84D46BB0F33A46B2682D37D96512FE753F: invalid validator set [informalsystems/ibc-go/v3@v3.4.1-0.20221202165607-3dc5ba251371/modules/light-clients/07-tendermint/types/update.go:156] With gas wanted: '0' and gas used: '65710' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "866"} }
...

I enabled trace level in hermes and also added some debugs. I think the initial client creation on the provider chain uses the wrong validator set hash for height 1 as that hash (1F1333..) is the one after the validator set changes at height 9.
I also tried to create a new client for the consumer chain on the provider chain and the initial consensus state, taken at height ~500, also uses the same hash (1F1333..). Updates work fine on this new client.

Side note: @romac, since the client needs to be updated after major valset changes, from height 1 to 858 in my run, there will be intermediate supporting headers (height 8 where next valset changes > 1/3rd) and I believe this is the first time we see this live. So we should keep an eye on this once we sort the client creation.

[ancazamfir]

Where is hermes fetching validators list?

From the validators at height H+1 (uses the validators RPC)

[sainoe]

Thanks for the thorough issue analysis @ancazamfir! It also seems to me that the consumer chain starts with a different validator set than the one used to create the client on the provider after the consumer proposal passes.

[ancazamfir]

Thanks for the thorough issue analysis @ancazamfir! It also seems to me that the consumer chain starts with a different validator set than the one used to create the client on the provider after the consumer proposal passes.

Actually this is not exactly what happens. The validator sets are the same. Let me try to explain better what I think happens. The client is initially created with a "trusted" consensus state that has a height before the consumer proposal passes but with the validator set after the proposal passed. This initial state is not correct and because of the way the IBC (ics-02) protocol is defined, this inconsistency cannot be detected by the on-chain IBC (ics-02 and ics-07) client handlers.

One way to fix this is

• create the client with a height after the proposal passes and validator set has changed, OR
• create the client with height 1 (or any before proposal) but with the sovereign chain validator set. I don't think this make sense though as how would the provider know the old sovereign chain valset. But I know little about interchain security in general and this migration in particular.

[ancazamfir]

@sainoe do you know where in the code the consumer IBC client is created on the provider chain? We use the code and scripts in Stride-Labs/interchain-security#1.
There is a client created for the new consumer chain (old sovereign) with wrong initial consensus state. This client is later used by the script to setup connections and the connection handshake fails.

[shaspitz]

@ancazamfir, I don't have all the context of this issue, but this may be the method you're looking for. Consensus state is instantiated here from MakeConsumerGenesis above

[ancazamfir]

@ancazamfir, I don't have all the context of this issue, but this may be the method you're looking for. Consensus state is instantiated here from MakeConsumerGenesis above

thanks @smarshall-spitzbart, that was very helpful! after browsing the code it became clear that the consumer client on provider chain is set with the initial height from the proposal.
@jstr1121 please use the diffs here. The important change is the one for initial_height in the consumer-proposal.json.
With these changes I was able to run the script without errors.

Note: the other changes are adapting to hermes v1.2.0 CLIs. I also changed the hermes config to disable some features not required by your setup.

[jstr1121]

Thanks all for your inputs - I was able to build connection successfully by adding a change on provider chain side where client is created.

{
    "title": "Create a chain",
    "description": "Gonna be a great chain",
    "chain_id": "consumer",
    "initial_height": {
        "revision_number": 0,
        "revision_height": 8
    },
    "genesis_hash": "519df96a862c30f53e67b1277e6834ab4bd59dfdd08c781d1b7cf3813080fb28",
    "binary_hash": "09184916f3e85aa6fa24d3c12f1e5465af2214f13db265a52fa9f4617146dea5",
    "spawn_time": "2022-06-01T09:10:00.000000000-00:00", 
    "deposit": "10000001stake",
    "consumer_redistribution_fraction": "0.75",
    "blocks_per_distribution_transmission": 1000,
    "ccv_timeout_period": 2419200000000000,
    "transfer_timeout_period": 3600000000000,
    "historical_entries": 10000,
    "unbonding_period": 1200000000000
}

Updated revision height from 1 to 8 - where the block 8 is where validator set is updated to consumer chain one.

[adizere]

Thank you Anca, Jun, and Shawn. Great work debugging this!