HIP-980: Enhance mirror node entity rest APIs with timestamp query parameter

Abstract

This HIP enhances part of the mirror node REST APIs to support querying by timestamp. This will allow users to query for the historical state of any entity info on the network that currently misses this functionality.

Motivation

To enhance user experience and allow users or developers to query the state of an entity at a specific point in time we will add a timestamp query parameter to all of the entity-based REST APIs that currently don’t have one. This change will be especially valuable for building the foundation of the so-called shadow fork functionality, where a user or developer can use a local mirror node instance but run it against a production environment state. Having this support, users or developers will have the ability to use public state data to simulate and troubleshoot existing issues or replay historical transactions using local mirror node instance against production state.

Rationale

Users may want to run historical transactions on a local mirror node instance using public state data. To make this possible, some of the existing REST APIs should be enhanced with a timestamp query parameter, so that historical public data can be fetched and used for the simulations.

User stories

• As a user of mirror node, I want to be able to query for the historical state of nfts.
• As a user of mirror node, I want to be able to query for the historical state of token relationships for an account.
• As a user of mirror node, I want to be able to query for the historical state of allowances for an account.
• As a user of mirror node, I want to be able to run locally an eth_call request that fails on a production environment against the corresponding public state.
• As a user of mirror node, I want to be able to run locally an eth_estimateGas request that fails on a production environment against the corresponding public state.
• As a user of mirror node, I want to be able to run locally an eth_debugTraceTransaction request that fails on a production environment against the corresponding public state.
• As a mirror node developer, I want to be able to test new features locally against production state, so that I have better understanding of the feature behavior.

Specification

The proposed enhancement involves adding a query parameter timestamp to the existing routes:

• /api/v1/accounts/{idOrAliasOrEvmAddress}/tokens to return correct historical information about an account and their associated fungible tokens for a specific timestamp
• /api/v1/accounts/{idOrAliasOrEvmAddress}/nfts to return correct historical information about an account and their associated nfts for a specific timestamp
• /api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/crypto to return correct historical information about an account’s crypto allowances for a specific timestamp
• /api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/nfts to return correct historical information about an account’s NFT allowances for a specific timestamp
• /api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/tokens to return correct historical information about an account’s token allowances for a specific timestamp
• /api/v1/tokens/{tokenId}/nfts/{serialNumber} to return correct historical information about an NFT for a specific timestamp

Timestamp Query Parameter

A new timestamp query parameter will be added to search for all entities and entity related information that are missing its support. It will be with the format of array[string]. This query parameter will represent a single or multiple consensus timestamps in a Unix format in seconds.nanoseconds format and will have an optional comparison operator. That is, it will search for results that are equal, greater or less than the timestamp itself or its seconds component.

The exact names of the operators will be eq, gt, gte, lt and lte. The user doesn’t need to know the exact timestamp at which an entity was created or updated for it to function. The API will return the results that match the timestamp or the closest one to it. For example, if we have an asc ordering of the results, then ?timestamp=X, timestamp=eq:X, or timestamp=lte:X will have the same behaviour and vice versa, where X is the desired timestamp.

The timestamp query parameter will support having a range of timestamps. For example, timestamp=gt:X&timestamp=lt:Y will return a single result with the entity or entity related info in the most recent state within the range that is greater than X and less than Y, where X and Y are different timestamps.

Here are the affected APIs with the query parameter added:

/api/v1/accounts/{idOrAliasOrEvmAddress}/tokens?timestamp={value}
/api/v1/accounts/{idOrAliasOrEvmAddress}/nfts?timestamp={value}
/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/crypto?timestamp={value}
/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/nfts?timestamp={value}
/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/tokens?timestamp={value}
/api/v1/tokens/{tokenId}/nfts/{serialNumber}?timestamp={value}

Note: There is an existing API with a timestamp query parameter, which is /api/v1/accounts/{idOrAliasOrEvmAddress}?timestamp={value}. While parameter changes won’t be applied for it, it’s behaviour will reflect to actually return modification changes of all account fields. Currently, the timestamp filter is applied only to the transaction list. The balance info section will continue to have granularity of 15 minutes.

Request

/api/v1/accounts/{idOrAliasOrEvmAddress}/tokens?timestamp={value}

Example Request

GET api/v1/accounts/4408244/tokens?order=asc&timestamp=lte:1725334849.603493001

Note that the following two requests will return the same response as the query above:

GET api/v1/accounts/4408244/tokens?order=asc&timestamp=1705334449.803393003
GET api/v1/accounts/4408244/tokens?order=asc&timestamp=eq:1705334449.803393003

Response

The response format of all the mentioned APIs will be enhanced with additional timestamp range with from and to fields, reflecting the period in which the returned data state was valid. The to field will be null if the result represent the latest updated state.

{
  "tokens": [
    {
      "automatic_association": true,
      "balance": 1,
      "created_timestamp": "1705334449.803393003",
      "decimals": 0,
      "token_id": "0.0.4328145",
      "freeze_status": "NOT_APPLICABLE",
      "kyc_status": "NOT_APPLICABLE",
      "timestamp": {
        "from": "1705334449.803393003",
        "to": null
      }
    }
  ],
  "links": {
    "next": null
  }
}

Backwards Compatibility

The change is backwards compatible as it is simply adding a query parameter to an existing route. The data required to filter by timestamp is already stored in the mirror node.

If a timestamp query parameter is not provided, the API will use the latest state of the entity as it does currently.

Security Implications

Historical timestamp filtering can slow down some of the queries, especially if the timestamp is not indexed or the data set is very large.

Rejected Ideas

None

References

https://mainnet-public.mirrornode.hedera.com/api/v1/docs/

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: