HIP-830: Event Serialization With Round Number, EventDescriptors, and Multiple Other Parents.

Author	Edward Wertz
Discussions-To	https://github.com/hashgraph/hedera-improvement-proposal/discussions/827
Status	Accepted ⓘ
Needs Council Approval	Yes ⓘ
Review period ends ⓘ	Mon, 04 Dec 2023 07:00:00 +0000
Type	Standards Track ⓘ
Category	Core ⓘ
Created	2023-10-25
Updated	2024-01-10

Abstract

This HIP proposes several changes to the event serialization format.

Add the birthRound to the metadata of new events. The birthRound value is 1 + the latest round to have come to consensus at the time the event was created. This value is used to determine if the event is ancient or not.
It is also used to look up the correct consensus roster for validating the event’s signature.
Represent references to parent events as EventDescriptors. EventDescriptors combine the existing data about each parent and the birthRound into a single data structure: EventDescriptor(hash, creator, generation, birthRound).
Allow for multiple other parents.
Reduce EventUnhashedData to just the event’s signature.

Motivation

Add `birthRound` to Event Metadata

The birthRound of an event is 1+ the latest round to have come to consensus at the time the event was created.
The birthRound is used to determine if an event is ancient or not and to look up the correct roster for validating the event’s signature.

EventDescriptors as Parent Event References

An EventDescriptor is a 4-tuple:

eventHash - The hash of the event.
creatorId - The NodeId of the node that created the event.
generation - The generation of the event. (1 + the max of the event’s parents’ generations)
birthRound - 1 + the latest round to reach consensus on the node when the event was created.

The EventDescriptor is the information that an event must store about each parent event, for use when the parent event is not in memory.

The parent EventDescriptors are used in the following ways:

To determine the generation of this event
To determine which events are its parents
To determine whether a given parent is needed before this one can be added to the hashgraph
To determine whether to create an edge in the hashgraph connecting it to each parent

Multiple Other Parents

Allowing events to have multiple other parents will open up new algorithms for choosing parents from other nodes.
This is expected to help lower the average time it takes an event to reach consensus.

Signature As Unhashed Metadata (Gossip Only)

Prior to the changes proposed in this HIP, the serialized unhashed data of Gossip Events included the following:

A sequence number from the self-parent node (unused field)
A sequence number form the other parent node (unused field)
The node id of the other parent’s node.
The signature of the event.

The changes proposed in this HIP remove the unused sequence numbers and move the node id of the other parent’s node to the hashed portion of the data in the appropriate EventDescriptor. The only data that remains unhashed in gossip events is the signature of the event, which cannot be hashed. Going forward, the implementation will no longer encapsulate unhashed metadata as an object in Gossip Events.

Rationale

This proposal comes from the Swirlds Labs Platform Hashgraph Team which maintains the algorithms and implementation used for gossiping events and determining consensus. We have bundled these serialization changes together to minimize the number of times we change the serialization format.

User stories

Jane is a node operator and pays a per byte data rate for her internet usage. Improved gossip algorithms will reduce data duplication during node communication and save Jane money.

John is a retail vendor with customer payments over the Hedera network. The time an event takes to come to consensus is part of the payment processing experience of his customers. A shorter consensus time translates to improved customer experience.

Jill is retiring her node from the network. Jack is starting a new node in the network. Both want their nodes to transition into (in)activity outside of network maintenance windows. The ability to dynamically change the active nodes participating in consensus while keeping the network secure and performant requires the ability to identify which address book an event is related to when it is created.

Specification

The following specification describes the bytes on the wire for an event and its contained objects. The data structures are described in order of their composite nesting, leaves of the tree first. Serialization formats which have not changed may be omitted.

Unchanged Serialization Formats

+------------------------------------------------------------------------+
| NodeId Object                                                          |
+------------+---------+---------------------------------+---------------+
| # of bytes | Type    | Description                     | Typical Value |
+------------+---------+---------------------------------+---------------+
| 4 bytes    | Integer | Serialization version of object | 0x00000001    |
+------------+---------+---------------------------------+---------------+
| 8 bytes    | Long    | Node id value                   |               |
+------------+---------+---------------------------------+---------------+

+------------------------------------------------------------------------+
| Hash Object                                                            |
+------------+---------+---------------------------------+---------------+
| # of bytes | Type    | Description                     | Typical Value |
+------------+---------+---------------------------------+---------------+
| 4 bytes    | Integer | Serializaiton version of object | 0x00000001    |
+------------+---------+---------------------------------+---------------+
| 4 bytes    | Integer | Hash type                       | 0x58ff811b    |
+------------+---------+---------------------------------+---------------+
| 4 bytes    | Integer | Length of hash                  | 0x00000030    |
+------------+---------+---------------------------------+---------------+
| N bytes    | byte[]  | Hash value (N = length of hash) |               |
+------------+---------+---------------------------------+---------------+

+--------------------------------------------------------------------------------------------------+
| ConsensusData Object                                                                             |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000001    |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | The event's generation                      |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | The event's roundCreated                    |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Boolean               | true if the event is stale, false otherwise |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Boolean               | true if last in roundReceived, else false   |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 16 bytes   | Instant               | the event's consensus timestamp             |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | the round the event was received            |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | the consensus order of the event            |               |
+------------+-----------------------+---------------------------------------------+---------------+

Event Serialization Formats Before HIP-830

+--------------------------------------------------------------------------------------------------+
| EventHashedData Object                                                                           |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000004    |
+------------+-----------------------+---------------------------------------------+---------------+
| V bytes    | SoftwareVersion       | The software version that created the event |  (String)     |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | The id of the node that created the event   |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | The generation of the self-parent event     | (may be -1)   |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | The generation of the other parent event    | (may be -1)   |
+------------+-----------------------+---------------------------------------------+---------------+
| H bytes    | Hash                  | The hash of the self-parent event           | (may be null) |
+------------+-----------------------+---------------------------------------------+---------------+
| H bytes    | Hash                  | The hash of the other parent event          | (may be null) |
+------------+-----------------------+---------------------------------------------+---------------+
| 16 bytes   | Instant               | The time the event was created              |               |
+------------+-----------------------+---------------------------------------------+---------------+
| T bytes    | Transaction[]         | The transactions within the event           |               |
+------------+-----------------------+---------------------------------------------+---------------+

+--------------------------------------------------------------------------------------------------+
| EventUnhashedData Object                                                                         |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | Event Creator Node's Sequence Number        | UNUSED (-1)   |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | Other Parent's Node Id                      |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | Other Parent Node's Sequence Number         | UNUSED (-1)   |
+------------+-----------------------+---------------------------------------------+---------------+
| S bytes    | Byte[]                | The signature of the node signing the event |               |
+------------+-----------------------+---------------------------------------------+---------------+

+--------------------------------------------------------------------------------------------------+
| GossipEvent Object                                                                               |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000003    |
+------------+-----------------------+---------------------------------------------+---------------+
| H bytes    | EventHashedData       | The hashed data of the event                |               |
+------------+-----------------------+---------------------------------------------+---------------+
| U bytes    | EventUnhashedData     | the unhashed data of the event              |               |
+------------+-----------------------+---------------------------------------------+---------------+

+--------------------------------------------------------------------------------------------------+
| DetailedConsensusEvent Object                                                                    |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000001    |
+------------+-----------------------+---------------------------------------------+---------------+
| H bytes    | EventHashedData       | The hashed data of the event                |               |
+------------+-----------------------+---------------------------------------------+---------------+
| U bytes    | EventUnhashedData     | the unhashed data of the event              |               |
+------------+-----------------------+---------------------------------------------+---------------+
| C bytes    | ConsensusData         | The consensus data for the event            |               |
+------------+-----------------------+---------------------------------------------+---------------+

Event Serialization Formats After HIP-830

The EventDescriptor is a new data structure to encapsulate parent event metadata. The definition of the serialized data structure is provided here before it is used in the EventHashedData object.

+------------------------------------------------------------------------+
| EventDescriptor Object                                                 |
+------------+---------+---------------------------------+---------------+
| # of bytes | Type    | Description                     | Typical Value |
+------------+---------+---------------------------------+---------------+
| 4 bytes    | Integer | Serialization version of object | 0x00000003    |
+------------+---------+---------------------------------+---------------+
| H bytes    | Hash    | The hash of the event           |               |
+------------+---------+---------------------------------+---------------+
| 12 bytes   | NodeId  | The node that created the event |               |
+------------+---------+---------------------------------+---------------+
| 8 bytes    | Long    | The generation of the event     |               |
+------------+---------+---------------------------------+---------------+
| 8 bytes    | Long    | The birthRound of the event     |               |
+------------+---------+---------------------------------+---------------+

The EventHashedData contains all the data that is hashed to create the signature in the GossipEvent.

+--------------------------------------------------------------------------------------------------+
| EventHashedData Object                                                                           |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000004    |
+------------+-----------------------+---------------------------------------------+---------------+
| V bytes    | SoftwareVersion       | The software version that created the event |  (String)     |
+------------+-----------------------+---------------------------------------------+---------------+
| 12 bytes   | NodeId                | The node that created the event             |               |
+------------+-----------------------+---------------------------------------------+---------------+
| D bytes    | EventDescriptor       | The descriptor of the event's self parent   |               |
+------------+-----------------------+---------------------------------------------+---------------+
| L bytes    | EventDescriptor[]     | A list of descriptors for other parents     |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 8 bytes    | Long                  | The birthRound of the event                 |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 16 bytes   | Instant               | The time the event was created              |               |
+------------+-----------------------+---------------------------------------------+---------------+
| T bytes    | Transaction[]         | The transactions within the event           |               |
+------------+-----------------------+---------------------------------------------+---------------+

The GossipEvent object contains the EventHashedData and the signature of the event. After the event is Gossiped, the signature is checked against the consensus roster that was effective for the round when the event was created.

+--------------------------------------------------------------------------------------------------+
| GossipEvent Object                                                                               |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000004    |
+------------+-----------------------+---------------------------------------------+---------------+
| H bytes    | EventHashedData       | The hashed data of the event                |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | The length of the signature = S             |               |
+------------+-----------------------+---------------------------------------------+---------------+
| S bytes    | Byte[]                | The signature of the node signing the event |               |
+------------+-----------------------+---------------------------------------------+---------------+

The DetailedConsensusEvent contains the EventHashedData, the signature of the event, and the ConsensusData that was generated when the event came to consensus.

+--------------------------------------------------------------------------------------------------+
| DetailedConsensusEvent Object                                                                    |
+------------+-----------------------+---------------------------------------------+---------------+
| # of bytes | Type                  | Description                                 | Typical Value |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | Serialization version of object             | 0x00000004    |
+------------+-----------------------+---------------------------------------------+---------------+
| H bytes    | EventHashedData       | The hashed data of the event                |               |
+------------+-----------------------+---------------------------------------------+---------------+
| 4 bytes    | Integer               | The length of the signature = S             |               |
+------------+-----------------------+---------------------------------------------+---------------+
| S bytes    | Byte[]                | The signature of the node signing the event |               |
+------------+-----------------------+---------------------------------------------+---------------+
| C bytes    | ConsensusData         | The consensus data for the event            |               |
+------------+-----------------------+---------------------------------------------+---------------+

Backwards Compatibility

This event serialization change is included in v0.46 of the implementation of the platform. The implementation of v0.46 will support deserialization of the event format used in v0.45 and will re-serialize the events in the older serialization scheme for the purpose of checking the signature on the events. All new events created by v0.46 will be in the new serialization scheme. After v0.46 is delivered, the older serialization scheme will not be supported in future releases.

Security Implications

N/A

How to Teach This

The documentation of the serialization scheme change is here in this HIP and as code.

Reference Implementation

The reference implementation is located in this PR: https://github.com/hashgraph/hedera-services/pull/9344

Rejected Ideas

No rejected ideas, yet.

Open Issues

No open issues, currently.

References

A collections of URLs used as references through the HIP.

https://github.com/hashgraph/hedera-services/pull/9344

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: