HIP-993: Improve record stream legibility and extensibility

Abstract

As the Hedera protocol evolves, it is critical to refine the legibility and extensibility of the record stream.

In this HIP we identify six opportunities for such improvements.

1. Itemized auto-creation fees
   • Current behavior - When the network creates an account as a side effect of a CryptoTransfer sending assets to an unused account alias, it “squashes” the fees for the implicit CryptoCreate into the record of the CryptoTransfer instead of itemizing them in the preceding auto-creation record. For example, take this CryptoTransfer from HashScan. Its transactionFee is given as $0.0474, even though $0.0473 of that amount was actually charged for the work done in the preceding child here.
   • Proposed change - Itemize the auto-creation fees in the preceding child record.
2. Unified child consensus times
   • Current behavior - When a contract operation dispatches a child transaction, the network gives the transaction handler the tentative synthetic consensus time “planned” for the child record stream item.
   • Proposed change - Give all child transaction handlers the consensus time of their parent transaction. (This will make the creation_time of an NFT minted via child transaction the consensus time of its parent transaction instead of a synthetic consensus time.)
3. Fail fast on throttled child transactions
   • Current behavior - When a contract operation dispatches more work via child transactions than is allowed by the network throttles, it only fails at the very end of the parent transaction.
   • Proposed change - Fail the first child transaction that exceeds the throttle.
4. Clean hollow account completion records
   • Current behavior - When the recipient address of an EVM message frame “collides” with a hollow account, the synthetic ContractUpdate externalized in the stream has unused fields that suggest the hollow account inherits properties from the sender address even though it does not.
   • Proposed change - Remove the unused fields.
5. Synthetic file creations at genesis
   • Current behavior - When a pre-production network starts from a genesis state, any mirror node in its environment has no way to know the initial contents of the bootstrapped system files.
   • Proposed change - Externalize these contents via synthetic FileCreate transactions as preceding children of the first user transaction in the record stream.
6. Use natural order for preceding dispatch records
   • Current behavior - The records of any work done via a “preceding” internal dispatch are externalized before the record of the user transaction being handled; even if a child of the user transaction initiated the dispatch.
   • Proposed change - Externalize preceding dispatch records directly before their logical parent transaction, even if it is not the user transaction being handled.

Motivation

We have the following motivations for each of the proposed changes:

1. It is more intuitive to externalize fees payments in the record of the work they were charged for. And as the Services codebase adopts a uniform “unit of work” that can be dispatched from multiple sources (user transaction business logic, the scheduling service, auto-renewal and archival processes, etc.), it would take material effort to avoid itemizing these fee payments.

2. Business logic gets no value from receiving a tentative synthetic consensus time instead the parent consensus time, but using this time makes it harder to evolve toward more flexible strategies for assigning synthetic consensus times and hence removing artificial limits on the number of child transactions.

3. Failing the first child transaction that exceeds the throttle is more intuitive and allows the network to fail fast when a contract operation dispatches more work than is allowed.

4. Clearly there is no reason to set unused fields in any record stream item.

5. Externalizing the initial contents of the bootstrapped system files allows a mirror node to reconstruct the state of the network from the record stream alone.

6. It is easier to see the relationships among records in the stream if records of preceding dispatches appear directly before their parent, even if it is not the user transaction.

Rationale

As noted in the abstract, each proposed change makes the record stream either more legible or extensible, or both.

User stories

• As a HashScan user, I want to see exactly what fees were charged for my parent CryptoTransfer versus what fees were charged for its CryptoCreate child.
• As a Services developer, I want to align the fees charged for a dispatched “unit of work” with its own record instead of its parent record.
• As a Services developer, I want to simplify the logic for assigning synthetic consensus times to child transactions.
• As a contract service user, I want to have more flexibility in the number of child transactions I can dispatch.
• As a contract service user, I want to see the first child transaction that exceeded the network throttles.
• As a contract service user, I want the ContractUpdate externalized for completion of a hollow account as a contract to have only the fields that are actually given to the new contract.
• As a mirror node operator, I want to be able to reconstruct the state of the network from the record stream alone.
• As a HashScan user, I want to easily see which record was the logical origin of a preceding dispatch.

Specification

Itemized auto-creation fees

Given the example transactions from the abstract, instead of externalizing the two record excerpts below,

*** <<<Current preceding child>>> ***
receipt {
  status: SUCCESS
  accountID {
    accountNum: 6134246
  }
}
...
transactionID {
...
  nonce: 1
}
memo: "auto-created account"
transactionFee: 54047391
transferList {
}

*** <<<Current following parent>>> ***
receipt {
  status: SUCCESS
...
}
...
transactionFee: 54162549
transferList {
  accountAmounts {
    accountID {
      accountNum: 8
    }
    amount: 4664
  }
  accountAmounts {
    accountID {
      accountNum: 98
    }
    amount: 48742097
  }
  accountAmounts {
    accountID {
      accountNum: 800
    }
    amount: 5415788
  }
  accountAmounts {
    accountID {
      accountNum: 3929606
    }
    amount: -55162549
  }
  accountAmounts {
    accountID {
      accountNum: 6134246
    }
    amount: 1000000
  }
}

The network should externalize,

*** <<<Proposed preceding child>>> ***
receipt {
  status: SUCCESS
  accountID {
    accountNum: 6134246
  }
}
...
transactionID {
...
  nonce: 1
}
memo: "auto-created account"
transactionFee: 54047391
transferList {
  accountAmounts {
    accountID {
      accountNum: 98
    }
    amount: 48642652
  }
  accountAmounts {
    accountID {
      accountNum: 800
    }
    amount: 5404739
  }
  accountAmounts {
    accountID {
      accountNum: 3929606
    }
    amount: -54047391
  }
}

*** <<<Proposed following parent>>> ***
receipt {
  status: SUCCESS
...
}
...
transactionFee: 115158
transferList {
  accountAmounts {
    accountID {
      accountNum: 8
    }
    amount: 4664
  }
  accountAmounts {
    accountID {
      accountNum: 98
    }
    amount: 99445
  }
  accountAmounts {
    accountID {
      accountNum: 800
    }
    amount: 11049
  }
  accountAmounts {
    accountID {
      accountNum: 3929606
    }
    amount: -1115158
  }
  accountAmounts {
    accountID {
      accountNum: 6134246
    }
    amount: 1000000
  }
}

Unified child consensus time

In the current record stream, we give each child transaction a unique synthetic consensus time to simplify reuse of existing mirror node importers. However, these synthetic consensus times are more of an implementation detail than a system invariant, since the state changes from child dispatches must really be committed as a transactional unit with their parent.

So it makes sense to hide synthetic consensus time from child transaction handlers at the point of dispatch, and allow the handle workflow to defer assignment of synthetic consensus times until the end of the parent transaction. This will ultimately let us remove artificial limits on the number of child transactions that can be dispatched from a parent.

Until those limits are removed, this change will be almost invisible in state or the record stream. The only observable impact will be that the creation_time of an NFT minted via child transaction will now be the consensus time of its parent transaction, instead of a synthetic consensus time.

A mirror node that is storing a creation_time for child NFT mints and is concerned with nanosecond fidelity of this field, will need to update its importer logic to switch to using parent consensus time as the creation_time of NFTs minted via child transactions after the first release that implements this HIP.

There is no known material impact of this change, however, even on mirror nodes that choose to ignore it completely.

Fail fast on throttled child transactions

A contract operation can dispatch more work via child transactions than is allowed by the network throttles. In the current implementation, the network only detects this at the very end of the parent transaction, and reverts the parent EVM transaction with final status CONSENSUS_GAS_EXHAUSTED.

This does unnecessary work past the point of the first child transaction that is throttled, and makes it unclear to the user exactly which child transaction caused the failure. We propose to instead fail the first child transaction that exceeds the throttle, and to set the CONSENSUS_GAS_EXHAUSTED status in the record of that child transaction.

It is, of course, possible to write a contract that ignores the result of a child transaction, and will thus behave differently if the throttling of that child does not unconditionally revert the parent transaction. Such a contract’s behavior will already vary with transient and reversible aspects of state other than network congestion, however; and it could not reasonably be considered broken by this change. (If anything, we will now allow it to proceed as “normal” even in the presence of network congestion.)

Clean hollow account completion records

When the recipient address of an EVM message frame “collides” with a hollow account, the hollow account is completed as an immutable contract with no additional properties inherited. However, the record being externalized in the stream currently implies the new contract inherits the memo, auto_renew_account_id, and other fields from the sender address. Clearly we should remove these unused and misleading fields.

Synthetic file creations at genesis

It is currently tricky to initialize a mirror node for use at genesis in a pre-production environment, because the network does not automatically externalize the initial contents of the bootstrapped system files. (Most notably, the fee schedules in file 0.0.111 and the exchange rates in file 0.0.112). We propose to externalize these contents via synthetic FileCreate transactions added as preceding children of the first user transaction in the record stream. This is in exact analogy to how the network externalizes the initial system accounts as preceding children of the first transaction.

Natural ordering of preceding records

Presently, the only case in which a non-user transaction triggers preceding dispatches is the auto-creation scenario already discussed; that is, when a CryptoTransfer sends assets to an unused alias and triggers a CryptoCreate. This can happen when, for example,

1. A user ContractCall dispatches a CryptoTransfer that triggers an auto-creation.
2. A user ScheduleSign triggers immediate execution of a CryptoTransfer that triggers an auto-creation.

With the current behavior, the order of records in these scenarios are:

1. CryptoCreate, ContractCall, CryptoTransfer
2. CryptoCreate, ScheduleSign, CryptoTransfer

After adoption of this HIP, the order will become,

1. ContractCall, CryptoCreate, CryptoTransfer
2. ScheduleSign, CryptoCreate, CryptoTransfer

Backwards Compatibility

This HIP is almost entirely backward compatible. Any mirror node capable of ingesting the current record stream will be able to ingest the new record stream, and derive a functionally identical network state.

All contracts whose operations only proceed when their child transactions are successful will revert earlier with a more legible failure reason during network congestion. If there are any contracts that ignore the results of their child transactions, they will now be able to continue even in the presence of network congestion that before this HIP would have forcibly reverted them.

The only breaking change is more or less cosmetic, adjusting the creation_time of some NFTs by a few nanoseconds as described in the section on unified child consensus time.

Security Implications

This HIP does not have any obvious security implications. Besides the noted infinitesimal change in child NFT mint creation times, it does not alter the effects of any transaction on state. Instead it only clarifies or simplifies how such effects are externalized in the record stream.

How to Teach This

Apply principles of least surprise and maximum legibility when constructing record streams for auto-account creations, throttled child transactions, hollow account completions, and bootstrapped system files.

Reference Implementation

1. This PR implemented the desired behavior by adopting a standard unit of work for both user and child transactions.
2. This PR implemented the desired behavior by reusing the platform-assigned consensus time for all dispatches in the scope of a single user transaction.
3. TBD
4. This PR implemented the desired behavior by removing the unused fields from the synthetic ContractUpdate externalized in the record stream.
5. This PR implemented the desired behavior by dispatching synthetic FileCreate transactions when handling the genesis transaction.
6. This PR implemented the desired behavior by managing pending records in the same “savepoint” lifecycle used for pending state changes.

Rejected Ideas

In each case we considered leaving the current behavior; but deemed the proposed changes to be more intuitive, legible, or extensible, and offering a net return on investment for the effort required to implement them.

Open Issues

There are no known issues. All the reference implementations have been tested and expected behavior validated.

References

1. Example auto-creation records on HashScan
2. The parent record in the auto-creation example
3. The child record in the auto-creation example
4. Reference implementation for itemized auto-creation fees
5. Reference implementation for removal of unused ContractUpdate fields
6. Reference implementation for unified consensus times
7. Reference implementation for normalized preceding records

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: