HIP-435: Record Stream V6

Abstract

Updates the record file definition to be done via a protobuf.

Introduces sidecar records - a generic way for Hedera nodes to externalize additional information about transactions that is not part of TransactionRecords.

Includes BlockNumber changes for the record files.

Motivation

Using a protobuf definition for the record file schema is far preferable to having a custom parser or random specification someplace (currently a word doc). The goal is to be able to externalize verbose information about transaction executions without bloating the main TransactionRecord object with potentially unnecessary information and requiring mirror nodes to always download and process that information.

Rationale

There are use cases requiring a detailed and verbose output for a HAPI transaction execution. Users of Hedera should have the option to query a mirror node for this information. Meanwhile, bloating the TransactionRecords with information that would be unnecessary in most cases is not an ideal option. It is a much better design decision to create sidecar record files that can be downloaded and processed only by interested parties.

Design Goal #1 Mirror nodes should not be required to download and process verbose transaction information. The sidecar transaction records will be externalized in separate record files.

Design Goal #2 One transaction can have 0, 1, or more than 1 sidecar record related to it located in 0, 1, or more record files.

Design Goal #3 It should be possible and relatively easy to add new types of sidecar transaction records.

User stories

As a Hedera user, I want to have the ability to access verbose information about transactions.

As an operator of a mirror node, I want the ability to download and process only the most important record-related information that is externalized as part of the main RecordFile.

As an operator of a mirror node, I want the ability to download sidecar record files containing all the information about records execution that a Hedera node can provide.

Specification

Define new protobuf messages to support creating record files, sidecar record files, and signature files.

Record Stream File Changes

A new version (V6) of the record stream file format will be created using protobufs:

message RecordFileObject {
  Transaction transaction = 1;
  TransactionRecord record = 2;
}

Transaction and TransactionRecord are already defined protobuf messages and are used by the Record Stream algorithm. No changes will be required for these messages.

RecordFileObjects are serialized in record stream files.

message RecordStreamFile {

  // Record stream file format version
  fixed32 version = 1;

  // Version of HAPI that was used to serialize the file
  SemanticVersion hapiProtoVersion = 2;

  // Running Hash of all RecordFileObjects before writing this file.
  HashObject startObjectRunningHash = 3; 

  // List of all the record file objects from that period.
  repeated RecordFileObject fileObjects = 4; 

  // Running Hash of all RecordFileObjects before closing this file.
  HashObject endObjectRunningHash = 5; 

  // The block number associated with this period.
  int64 blockNumber = 6; 

  // List of the hashes of all the sidecar record files created for the same period.
  // Allows multiple sidecar files to be linked to this RecordStreamFile
  repeated SidecarMetadata sidecars = 7; 
}

RecordStreamFile will be used to serialize all RecordFileObjects that are part of the same period into record stream files. Periods are intervals of hedera.recordStream.logPeriod seconds used to group RecordFileObjects into record stream files. The block structure is similar to the current structure of record files, block number and sidecars are new fields.

enum SidecarType {
  ContractStateChange = 0;
  ContractAction = 1;
  ContractBytecode = 2;
  // Other future categories
}

Holds the possible sidecar record types. New types of sidecar records can be added in the future.

message SidecarMetadata {
  HashObject hash = 1;
  long id = 2;
  repeated SidecarType types = 3;
}

Information about a single sidecar file: the hash of the entire file, the id that will be apended to the file name, the types of sidecar records that will be included in the file. The record signature file that is created for each record stream file signs the hash of the entire corresponding stream file. The list of sidecar file hashes is included in the record stream file. This way mirror nodes or any interested party can download the record stream file and all sidecar files and verify that:

1. repeated bytes SidecarFileHash is correct;
2. the signature file signed the correct hash of the entire record stream file;

message HashObject {
  enum HashAlgorithm {
    SHA_384 = 0;
  }
  HashAlgorithm algorithm = 1;
  int32 length = 2;
  bytes hash = 3;
}

message TransactionSidecarRecord {
  // Consensus timestamp will be the same as the consensus timestamp of the 
  // transaction the side car is related to. This offers a convenient 
  // way to match record to sidecar. 
  Timestamp consensusTimestamp = 1;
  oneof {
    repeated ContractStateChange = 2; // Example sidecar type
    repeated ContractAction = 3; // Example sidecar type
    // Other future categories
  }
}

TransactionSidecarRecord is a protobuf message which will be used to create sidecar records complementing TransactionRecord and storing additional information about a transaction execution.

message SidecarFile {
  repeated TransactionSidecarRecord sidecarRecords = 2;
}

The TransactionSidecarRecords created by Hedera nodes will be externalized in SidecarFile files. Files will be created using the SidecarFile protobuf holding all the TransactionSidecarRecords related to RecordFileObjects created for the same period.

The SidecarFiles will be uploaded into a separate folder in the cloud bucket. Doing so will reduce the number of rows Mirror nodes have to search in S3, reduce S3 list response size, and avoid having to filter them out locally. Probably recordstreams/record0.0.3/sidecar.

message SignatureFile {
  int32 version = 1;
  SignatureObject fileSignature = 2;
  SignatureObject metadataSignature = 3;
}

message SignatureObject {
  enum SignatureType {
    SHA_384_WITH_RSA = 0;
  }

  SignatureType type = 1;
  int32 length = 2;
  int32 checksum = 3;
  bytes signature = 4;
  HashObject hashObject = 5;
}

Record File Names

A record file name is created using a string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility.

The nano-of-second always outputs nine digits with padding when necessary, to ensure same length filenames and proper sorting.

Examples of a record stream file name with corresponding signature file and sidecar record files:

Example: Record File: 2022-10-19T21_35_39.000000000Z.gz Sidecar Record File 1: 2022-10-19T21_35_39.000000000Z_01.gz Sidecar Record File 2: 2022-10-19T21_35_39.000000000Z_02.gz Record Signature File: 2022-10-19T21_35_39.000000000Z.rcd_sig

Record files and sidecar files will be compressed.

Record Stream Flow

Transactions are executed sequentially on the Hedera nodes. TransactionRecords are created as always. Sidecar records are now also created when appropriate. Record file objects are created grouping Transactions to their corresponding TransactionRecords. A map of type <> list of sidecar records is populated as well. Platform receives the list of record file objects and the map. Record stream files are created by including all the record file objects, running hashes, and the list of sidecar records hashes. Sidecar record files are created by grouping all sidecar records of the same type in separate files. The type is the key of the map. For example 1 will represent ContractStateChange, 2 will represent ContractAction, etc. This will provide a straightforward and generic way to group sidecar records into files.

Backwards Compatibility

The new v6 record file is not backward compatible with v5 and clients will need to be updated to support both. Adding support for both concurrently will allow them to ingest historical information and it will allow clients to roll out v6 support ahead of the cutover date.

The added files can be ignored by clients not wishing to trace or replay smart contract transactions.

Sidecar records for prior transactions will not be created and the verbose information related to them will not be available. This reflects current practice.

Security Implications

The new Record Stream format doesn’t provide any risk in itself. Care should be taken about what kind of information is externalized by each added type of sidecar record file.

How to Teach This

Reference Implementation

Rejected Ideas

Enhance V5 records with extra fields

This idea was rejected in order to move to the protobufs format which is more standardized and has native support and GRPC compilers for different programming languages.

Open Issues

References

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: