HIP-415: Introduction Of Blocks

Abstract

Specifies how to introduce and formalize the concepts of Blocks in Hedera Hashgraph so that it can be used as a foundation on which further interoperability with existing DLT networks and infrastructure can be built.

Motivation

The concept of blocks is a vital part of the existing Ethereum infrastructure and, as such, the introduction of a standard mapping between existing transaction ordering in Hedera and the notion of blocks can be considered a foundational step towards greater interoperability with EVM based tooling, explorers, exchanges and wallet providers.

Rationale

Hedera must have a single consistent answer to what transactions belong to a block and the identifying hash and number for that block. This is required for two reasons:

1. The Smart Contracts Service to have the context of a number and the hash of the current block while running the EVM bytecode.
2. Mirror Nodes to have the information in order to implement standard JSON RPC endpoints and be able to answer queries such as:
   1. Getting current block number
   2. Getting block by hash/number
   3. Getting a list of transactions for a block by number/hash
   4. Getting logs based on block number filtering

Design Goal #1 Minimize Changes

The Block concept should fit naturally into the existing processes, mechanisms and state updates. It must keep the same responsibilities between consensus, services and mirror nodes.

Design Goal #2 Lightweight

The Block concept must not add a lot of complexity and performance overhead to the processing of transactions. It must have minimal impact on the TPS of the network.

Based on the described design goals above, the outlined specification defines that block properties are to be computed and populated at different points in time and by different components of the network, based on their responsibility.

Specification

Definitions

• block → Record file containing all Record Stream Objects for a given time frame. Block times are to be at least hedera.recordStream.logPeriod seconds. The genesis block and therefore number (blockNumber=0) is considered the stream start date with the first RCD file exported from services nodes.
• block number → consecutive number of the Record file that is being incremented by 1 for every new Record file. For already existing networks, this value will be initially bootstrapped through Mirror Nodes and after that maintained by services nodes.
• block hash → 32 byte prefix (out of 48 bytes) of the running hash of the last Record Stream Object from the previous Record File
• block timestamp → Instant of consensus timestamp of the first transaction/Record Stream Object in the Record file.

Platform

Adapt TimestampStreamFileWriter to include blockNumber in the Record File. Introduce a new field firstConsensusTimeInCurrentFile to be used as a marker when to start a new Record file. Use the field lastConsensusTimestamp to keep track of the last-seen consensus timestamp that was processed. In this way, we can ensure that we have at least 1000ns difference between the last processed transaction before a new file is created. The unit of time to be used for those 2 properties is nanos. In this way if we have a parent transaction with at least one child precompile transaction they all will be included in the same block/Record file. Otherwise, we might have a corner case where a parent transaction is included in one block, and its child precompile transaction falls into the next block since it will increase the consensus timestamp with 1ns. Therefore, the algorithm for checking whether to start a new file would be the following:

A new Record Stream Object enters addObject(T object) in TimestampStreamFileWriter. It has a consensus timestamp T. We create a new file only if both (1) and (2) conditions are met:

1. T - lastConsensusTime > 1000ns
2. T - firstConsensusTimeInCurrentFile > 2s

or

if lastConsensusTime or firstConsensusTimeInCurrentFile is null

Services

Services are to update the processing logic of transactions so that it supports logic for determining new record file periods, incrementing block number and keeping block relevant data. The proposed solution specifies a long field to be used for the block number counter, incremented every hedera.recordStream.logPeriod seconds. Using a signed 32-bit int would result in the block number rolling over in 140 years (if current 2-second length is kept). Sub-second block lengths would exhaust that number well within the operational lifetime of typical networks. A signed 64-bit integer provides a much longer timeframe.

Pseudo-code of the record streaming algorithm

Properties in State:
- blockHashes `map(number -> keccak256(RunningHash))` - stores the hashes of the last 256 blocks
- blockNumber `long` - stores the current block number
- blockTimestamp - `Instant` - stores the timestamp of the block

handleTransaction() {
	...
	bool `newBlock` = shouldCreateNewBlock() {
			if (`currentTS` - `lastConsensusTime` > `1000ns` && `currentTS` - `blockTimestamp` > `hedera.recordStream.logPeriod`) return `true`; else `false`
	}
	if (`newBlock`) {
			`blockHashes[blockNumber] = keccak256(runningHash)` // `runningHash` is stored in `RecordStreaming`. It is a running hash of the last processed RSO
			delete `blockHashes[blockNumber - 256]`
			`blockNumber++`
			`blockTimestamp = currentTS`
	}
	processTransaction(tx)
	...
}

Migration:
`blockNumber = bootstrapLastBlockNumber`, where `bootstrapLastBlockNumber` is a system property added on startup.
`blockTimestamp = bootstrapLastBlockTimestamp`, where `bootstrapLastBlockTimestamp` is a system property added on startup.

number, timestamp and hash (map of the 256 most recent blocks) must be available during transaction execution since the following opcodes are to be supported as per the EVM specification:

• BLOCKHASH → Accepts the block NUMBER for which to return the hash. Valid range is the last 256 blocks (not including the current one)
• NUMBER → Returns the current block number
• TIMESTAMP → Returns the unix timestamp of the current block

Record File

Once HIP-435 is implemented, record files are to be migrated to protobufs. New long blockNumber property is to be added in the record stream.

It is required for services to propagate this property to mirror nodes since there are partial mirror nodes, that don’t keep the full history from the first record file. Due to that, they are unable to calculate the block number, thus all other block properties. Block hash and timestamp will not be included in the record files, since block hash is the 32 byte prefix (out of 48 bytes) of the running hash of the last Record Stream Object from the previous Record File and timestamp is 1st Record Stream Object’s consensus timestamp.

With the introduction of the new version of Record Stream Objects, the respective libraries for state proofs must be updated:

• https://github.com/hashgraph/hedera-mirror-node/tree/main/hedera-mirror-rest/check-state-proof
• https://github.com/hashgraph/hedera-state-proof-verifier-go

Mirror Nodes

Based on the updates specified above, the record stream objects will pass all the necessary information for mirror nodes to build Record files/blocks and store enough data about them to be able to answer all block related queries outlined above. To do this they will need to read and store the number specified in the Record file. For old record files, they will be derived through a migration process as specified below.

Migration

Record files prior to the introduction of the block properties will not expose block information. Mirror nodes that have a full testnet and mainnet history will maintain the most recent consensus timestamp to block number mapping. The values will be retrieved from mirror nodes that have a full history from stream start. Additionally, a hedera.mirror.importer.startBlockNumber property will be added to be able to customize the block number for new mirror nodes just starting up.

A repeatable database migration would be added to correct historical record files for partial mirror nodes. It would work as follows:

• Skip if not testnet or mainnet
• Look up the block in the database associated with the latest hardcoded timestamp for the environment.
• If the row’s block number matches then there’s nothing to do.
• If the row’s block number does not match, calculate the offset and update every row with that offset.

Later, after the block number is included in the record file, the mirror node will dynamically detect the presence of the first record file with block information and re-run the same migration to adjust historical values.

List Blocks REST API

A new list block REST API will be added to support the ability to search for block information. Note it only contains aggregate information about the blocks. It’s expected that users use the returned information like timestamps to query the transaction or contract results REST APIs for details about the transactions within the block.

{
  "blocks": [{
    "count": 4,
    "gas_limit": 150000000,
    "gas_used": 50000000,
    "hapi_version": "0.24.0",
    "hash": "0xa4ef824cd63a325586bfe1a66396424cd33499f895db2ce2292996e2fc5667a69d83a48f3883f2acab0edfb6bfeb23c4",
    "logs_bloom": "0x549358c4c2e573e02410ef7b5a5ffa5f36dd7398",
    "name": "2022-04-07T16_59_23.159846673Z.rcd",
    "number": 19533336,
    "previous_hash": "0x4fbcefec4d07c60364ac42286d5dd989bc09c57acc7370b46fa8860de4b8721e63a5ed46addf1564e4f8cd7b956a5afa",
    "size": 8489,
    "timestamp": {
      "from": "1649350763.159846673",
      "to": "1649350763.382130000"
    }
  }],
  "links": {
    "next": null
  }
}

Note: gas_limit won’t be available in initial implementations.

Query parameters:

• block.number - The block number. Used as the primary key in pagination. Supports eq, gt, gte, lt, lte operators.
• limit - The maximum number of results to return. Defaults to 25 and max 100 allowed.
• order - Sort by the block number. Value of asc or desc with a default of desc.
• timestamp - The consensus timestamp of the last transaction in the block. Supports eq, gt, gte, lt, lte operators.

Get Block REST API

The new mirror node /api/v1/blocks/{hashOrNumber} REST API will return the block information associated with a specific block. The path parameter will accept either a block hash or block number to uniquely identify a specific block. The block hash will accept a 64 or 96 hex-encoded string with a 0x prefix. The JSON response will be the same as /api/v1/blocks but without the pagination structure:

{
  "count": 4,
  "gas_limit": 150000000,
  "gas_used": 50000000,
  "hapi_version": "0.24.0",
  "hash": "0xa4ef824cd63a325586bfe1a66396424cd33499f895db2ce2292996e2fc5667a69d83a48f3883f2acab0edfb6bfeb23c4",
  "logs_bloom": "0x549358c4c2e573e02410ef7b5a5ffa5f36dd7398",
  "name": "2022-04-07T16_59_23.159846673Z.rcd",
  "number": 19533336,
  "previous_hash": "0x4fbcefec4d07c60364ac42286d5dd989bc09c57acc7370b46fa8860de4b8721e63a5ed46addf1564e4f8cd7b956a5afa",
  "size": 8489,
  "timestamp": {
    "from": "1649350763.159846673".
    "to": "1649350763.382130000"
  }
}

Query parameters: None

Note: gas_limit won’t be available in initial implementations.

List Contract results REST API

A new /api/v1/contracts/results mirror node REST API will be added to be able to search for contract results across addresses and blocks. By default, this API will only show transactions that originate from an EOA (externally owned accounts). Its JSON response will be identical to /api/v1/contracts/{id}/results. The contract ID/address will not be supported as a filter option as it is assumed users will use /api/v1/contracts/{id}/results if that functionality is desired. It will support the following query parameters:

• block.number - The block number in base 10 or hex format w/ 0x prefix. Only eq operator is supported.
• block.hash - The 32 or 48 byte block hash. With or without optional 0x prefix. Only eq operator is supported.
• internal - Whether internal/child transactions should be returned. Defaults to false.
• limit - The maximum number of results to return. Defaults to 25 and max 100 allowed.
• order - Sort by timestamp. Value of asc or desc with a default of desc.
• timestamp - The consensus timestamp of the transaction. Supports eq, gt, gte, lt, lte operators. Primary key for pagination.
• transaction.index - The position of the transaction within a block. Requires either block.number or block.hash to be present. Only eq operator is supported.

So that it has parity with the new API, the existing /api/v1/contracts/{id}/results will also be updated to add the block.number, block.hash, internal, and transaction.index query parameters.

Block Properties

The following table specifies all block properties and at which point they will be computed. Mirror nodes must hex encode all properties prior to exposing them through their APIs. This table defines the properties that must be returned through APIs from Mirror Nodes.

Backwards Compatibility

The following breaking changes will be introduced with the implementation of the HIP:

• BLOCKHASH will no longer return an empty hash but the actual hash of the block as per the EVM specification.
• NUMBER will no longer return the timestamp of the block, but rather the proper block number as specified in the HIP.

Security Implications

• The specification does not have any security implications.

How to Teach this

• Respective documentation will be added.

Reference Implementation

Initial POC in services:

• https://github.com/hashgraph/hedera-services/commit/04501de76f3e649b3f741dd2453c05e2631e7393#diff-1ca5738f0fe42a25f91d641aea2104dd51bec20398ce6c2aa18afde3d5a590d9R417

Rejected Ideas

Two iterations have been conducted prior to settling on this approach.

1. Rounds as blocks - the first iteration was proposing that each block is the consensus round. It became clear that this approach is not suitable as it implied that multiple blocks are to be issued per second. That would add additional load to infrastructure providers when clients are iterating and going through blocks to query for data.
   • Events - the second iteration suggested that consensus events are to be defined as blocks. The proposal had a much higher cognitive load (1 block = 1 record file is easier to grasp) and it required more changes to the platform in order to be implemented. To add on top of that, the same drawback as the high frequency blocks was present as well.

Open Issues

1. Feeding the current values of contracts.consensusThrottleMaxGasLimit and hedera.recordStream.logPeriod Global Dynamic Properties to Mirror Nodes so that they can compute the gasLimit value. The values / gas throttles are available in the network file 0.0.121 but only if they are overriden. In practice Mirror Nodes can update the value for gas/sec (consensusThrottleMaxGasLimit) and block times (log period) however, they do not have a way to feed the current/initial values. Calculating the gasLimit:

   gasLimit = contracts.consensusThrottleMaxGasLimit * hedera.recordStream.logPeriod
   

References

• Record stream objects
• Events
• Running hash
• EVM opcodes
• Ethereum Yellow Paper

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: