HIP-423: Long Term Scheduled Transactions

Author	Patrick Staton
Discussions-To	https://github.com/hashgraph/hedera-improvement-proposal/discussions/425
Status	Accepted ⓘ
Needs Council Approval	Yes ⓘ
Review period ends ⓘ	Mon, 25 Apr 2022 07:00:00 +0000
Type	Standards Track ⓘ
Category	Service ⓘ
Created	2022-04-11
Updated	2023-01-20

Abstract

The current Scheduled Transaction implementation’s main purpose is to provide an on-network mechanism for multiple parties to sign a transaction. All Scheduled Transactions must be signed within 30 minutes or they will expire. This was seen as a sufficient amount of time for collaborating parties to all submit their signatures as well as limit the network resources needed to support the feature.

The current Scheduled Transaction implementation also limits the types of transactions that can be scheduled. This was done to reduce the attack surface, testing requirements, and throttling requirements for the feature.

In this HIP we propose extending the existing Scheduled Transaction implementation to support executing transactions at an arbitrary time in the future and allowing more types of transactions to be scheduled.

Motivation

• The current Scheduled Transaction implementation is useful for a specific use case, but does not provide all the functionality that one would expect from Scheduled Transactions. IE transactions that execute at some scheduled date in the future.

• For institutions and corporate users it could take weeks for a proposed transaction to be reviewed and approved. 30 minutes is not sufficient time for that.

• The Hedera council currently creates/approves transactions offline in private and would like to move that process to the network to both reduce reliance on other communication mechanisms and provide more transparency. This requires both a larger expiration interval and allowing all the transactions types that the council currently uses.

• Providing a way to schedule smart contract related operations would be a significant differentiator for Hedera.

Rationale

Consensus Time

• Handling transactions must be deterministic. All nodes must do exactly the same operations in every “round”.
• The common input to a “round” on all nodes is “consensus time”.
• “Consensus time” does not correspond to “real wall clock time”. Although it is very close for the most part.
• Thus, we can only schedule things to happen in “consensus time” if we want all nodes to do the same thing for each round.

expiration_time must be in “consensus time”

• We cannot guarantee Scheduled Transactions will execute exactly at their scheduled “consensus time”.
  • There is no way to prevent another transaction from taking a given nanosecond slot without massive changes to the entire system.
  • Things like throttling and other system factors also interfere with when we can actually execute a transaction.

Scheduled Transactions will execute at the earliest available consensus time after their expiration_time on a best-effort basis.

Throttling

• There are throttles on how many transactions can execute per round.
• Each transaction type can have different throttles.

Scheduled Transactions must be throttled based on the transaction they contain

• Throttles are checked at the HAPI level and cannot know what the consensus time is going to be when the transaction is included.
  • In other words, we cannot know which Scheduled Transactions will be executed in the same round as a given “real time” transaction.
• If the system is 100% loaded with “real time” transactions, Scheduled Transactions could theoretically be blocked from executing.
• Similarly, if left un-checked, Scheduled Transactions could block “real time” transactions from executing.

In a given round, Scheduled Transactions should execute after “real time” transactions run and should be allocated a small amount of space above 100% of the existing throttles (ie we cannot prevent 100% of the throttles from being used up, but we can allocate more than 100% of the throttles)

• If we have a simple limit of x number of Scheduled Transactions per second and use a “best effort” execution algorithm that simply executes transactions as throttles permit, then an attacker could cause transactions with low TPS limits to be pushed back exponentially in time by filling up x with only that transaction type.

When ScheduleCreate is executed, apply throttles to the future second when it’s transaction will execute. In other words, make sure throttles will never be exceeded in any given second in the future.

Backwards Compatible

• The changes to the existing Scheduled Transactions implementation should be additive.
• Our changes should not break any existing functionality.

By default, everything should continue to function as it does today.

Fees

• It will be relatively cheap to “fill up” Scheduled Transactions such that you block others from using them.
• Some transactions, like ContractCall, would be particularly easy and possibly profitable to block.
  • For example, an attacker could schedule a large number of very high gasLimit ContractCall transactions and cheaply use up all the available gas for a future block of time.
    • The attacker doesn’t actually need to have enough HBAR in their account to cover the gas.
    • When the future date comes around all the transactions will fail - but that’s not a problem for the attacker, they prevented others from using the feature.

Charge a larger fee for ScheduleCreate transactions containing certain types of transactions.

Types of transactions

• The Hedera Council does not use all types of transactions, only a subset.
• It would be a significant win to support scheduling smart contract operations.

To limit the testing/attack surface, only add the types of transactions the Hedera Council needs and smart contracts require

Transaction Receipts and Transaction IDs

• Now that Scheduled Transactions can execute autonomously in the future, there needs to be a way to fetch the transaction receipt for them outside the normal mechanism.
• A user should not need to plan to fetch the transaction receipts within 3 minutes of their Scheduled Transaction executing

Data from the transaction receipt is available in the entity_id and other fields in the mirror node transactions API.

• We need a transaction ID to fetch from the transaction api to get receipts.
• The transaction ID of the transaction executed by a Scheduled Transaction is currently just the ScheduleCreate transaction ID with the scheduled flag set to true.

The transaction ID of the child can be found deterministically at ScheduleCreate time

Users can also use the executed_timestamp in the mirror node schedule API to find child transactions via the mirror node transactions API.

Transactions Change Over Time

A Scheduled Transaction being ready to execute, or even not ready to execute, at the time a ScheduleCreate or ScheduleSign comes in does not guarantee it will stay that way. Any number of things can happen over time that impact the transaction. Examples -

• An account changes keys after it has signed a Scheduled Transaction.
  • Scheduled Transactions will need to be re-signed with the new keys for them to succeed in this case.
• Account deletion.
  • If an account that is part of a Scheduled Transaction is deleted, the transaction will fail.
• Account balance changes.
  • If account balances change such that there is insufficient balances to allow the transaction to go through, it will fail.
• Signature requirements for Scheduled Transactions can change (e.g. via CryptoUpdate) such that existing signatures become sufficient to allow the transaction to go through.
  • In this case the transaction will execute at expiration_time unless a ScheduleSign comes in to push it through.

In all cases, transactions must be evaluated for execution at expiration time. We should document particularly thorny corner cases thoroughly.

User stories

1. As an institutional user, I want to schedule transactions that expire a month in the future so that all the parties involved in the transaction have sufficient time to review and approve them.

I ScheduleCreate a CryptoTransfer transaction with an expiration_time set to 30 days in the future and wait_for_expiry set to true. My company and other involved parties use ScheduleSign to add their signatures to the transaction in the next 30 days. A few seconds after the expiration_time, the transaction succeeds if enough signatures were received or fails otherwise. In both cases a record of all the parties that signed it is available.

2. As a business owner with invoices to pay, I want to schedule transactions to pay the invoices on the date they are due.

I ScheduleCreate a CryptoTransfer transaction with an expiration_time set to the morning a payment is due and wait_for_expiry set to true with all the required signatures already included in the ScheduleCreate. The transaction goes through a few seconds after the expiration_time. Because I scheduled for the morning, the small delay doesn’t matter.

3. As a secretary to the Hedera Council, I want to create and have members approve council transactions on the network.

• I ScheduleCreate a Freeze FREEZE_UPGRADE transaction with an expiration_time set to the day before a scheduled upgrade.
• Council membership does not change while the transaction is scheduled.
• Council members send in their signatures via ScheduleSign.
  • Immediately when there are enough council signatures, the freeze transaction is executed and the update will go through at the specified date.
  • If expiration_time passes without enough signatures, the freeze fails.
• In all cases a record of all the council members that approved the freeze is available.

4. As a smart contract user, I want to update a smart contract on a specific date/time automatically.

I ScheduleCreate a ContractUpdate transaction with an expiration_time set to the day I want to update my contract and wait_for_expiry set to true. Owners of the contract then submit their approval of the update via ScheduleSign transactions. A few seconds after the expiration_time, the contract is updated if enough signatures were received or fails otherwise. In both cases a record of all the owners that approved the contract update is available.

5. As a lawyer, I want to manage a trust with scheduled distributions.

• I set up a crypto account for the trust with the balance of the trust and a ThresholdKey set to require one of my law firms signature OR all of the executor’s signatures
• I ScheduleCreate a CryptoTransfer transactions for each distribution with the expiration_time set to the dates of the distributions and wait_for_expiry set to true.

• Executors of the trust submit their approvals for the transactions via ScheduleSign transactions.

• The executors of the trust change.
  • If all the old executors are available -
    • I ScheduleCreate a CryptoUpdate transaction to update the signing requirements on the trust account with the expiration_time set to 30 days from now.
    • I ask the old executors to submit ScheduleSign transactions to approve the changes to the trust account.
    • As soon as enough signatures are received the account is updated. Or it fails to update at expiration_time.
  • If all the old executors are not available -
    • I use my law firm’s key to submit a signed CryptoUpdate transaction to remove the dropped out party from the trust account.
    • The trust account is immediately updated due to the ThresholdKey in the trust account.
  • I ask the executors to re-submit their approval for each distribution via ScheduleSign transactions.
• A few seconds after the expiration_time, the distributions go out if enough signatures were received or fail otherwise.

6. As a lawyer, I want to manage a contract with a payment that goes out when signatures of all parties are received.

• I set up a crypto account for the contract with the balance of the payment and a ThresholdKey set to require one of my law firms signature OR all of the contract parties signatures.
• I ScheduleCreate a CryptoTransfer transaction for the payment with the expiration_time set to the date when the contract signature gathering period expires.

• Parties to the contract submit their approvals for the transaction via ScheduleSign transactions.

• The parties to the payment are reduced after one drops out and all the other parties have already signed the transaction.
  • I use my law firms key to submit a signed CryptoUpdate transaction to remove the dropped out party from the contract account.
  • The account is immediately updated with the new requirements due to the ThresholdKey in the contract account.
  • The contract payment does not immediately go through.
  • I can choose to
    • Wait and the payment will go through at expiration_time.
    • Or ask one of the parties to re-submit their signature and cause the transaction to go through immediately.

7. As a hacker, I want to prevent Scheduled Transactions from executing during a specific time range.

• I first try to flood the network with enough transactions to fill all the throttles during the time range.
  • I am unable to block the Scheduled Transactions because their throttles are in addition to the real time throttles.
  • This also costs me a lot of money.
• Next, I try to fill up all the scheduled transaction throttles during the time range.
  • This costs me more than flooding the network because ScheduleCreate is a couple of orders of magnitude more expensive than most other transaction types.
  • Users can still submit their transactions in real time if they really need to.
• Finally, I try to create smart contract transactions with high gas usage to possibly fill up the time.
  • This costs more than usual because ScheduleCreate for smart contracts is an order of magnitude more expensive than ScheduleCreate for other transactions.
  • I cannot fill up more than just the throttles for smart contracts. Other transaction types will still run fine.

8. As a hacker, I want to prevent real time transactions from executing during a specific time range using Scheduled Transactions.

I attempt to full up the real time throttles with Scheduled Transactions, but I quickly find that scheduled transaction throttles are separate from real time transaction throttles and no matter how many Scheduled Transactions I submit I cannot block real time transactions.

9. As an existing Scheduled Transactions user, I want my transactions that I create before the upgrade to Long Term Scheduled Transactions to work after the upgrade.

I ScheduleCreate a CryptoTransfer transaction with one required signature just before the upgrade to Long Term Scheduled Transactions and do not provide the required signature before the upgrade. After the upgrade I notice that the wait_for_expiry field on my transaction is set to false and the expiration_time is unchanged. As expected, if I submit the required signature before expiration then the transaction goes through, otherwise it fails.

10. As an existing Scheduled Transactions user, I want the code that I created before the upgrade to Long Term Scheduled Transactions to continue to work after.

• I ask my users to test on previewnet before the upgrade. They test, and it does work.
• I run e2e tests to confirm that my code works before and after the upgrade. The e2e tests succeed in both cases.
• I ask my users to test on testnet and prod after the upgrade. They test, and it does work in both cases.

11. As a smart contract user, I want to schedule the creation of a smart contract.

• I ScheduleCreate a ContractCreate transaction with an expiration_time set to the day I want to create my contract and wait_for_expiry set to true.
• I note the transaction ID of the ScheduleCreate and the schedule ID from the transaction receipt.
• Owners of the payer account then submit their approval of the creation via ScheduleSign transactions.
• A few seconds after the expiration_time, the contract is created if enough signatures were received or fails otherwise.
• I then use the ScheduleCreate transaction ID to generate the transaction ID of the ContractCreate by appending ‘?schedule’ to it.
• I use the ContractCreate transaction ID to call the mirror node transaction api and get the entity_id of the contract that was created.
• I can also use the mirror node schedule api and the schedule ID to fetch the executed_timestamp of the Scheduled Transaction which can then be used on the mirror node transaction api to get the entity_id of the contract that was created.

Specification

Protobuf changes

• Add an optional expiration_time Timestamp field to the ScheduleCreate protobuf. This field shall specify the “consensus time” when the transaction should attempt to execute and then expire.
  • If not specified, this shall default to 30 minutes after the consensus time that the ScheduleCreate is processed.
  • Initially the max value of this should be 2 months in the future. This may expand as stability is confirmed.
    • The max value shall be specified in the new scheduling.maxExpirationFutureSeconds setting.
    • A new SCHEDULE_EXPIRATION_TIME_TOO_FAR_IN_FUTURE error shall be created for this validation.
  • If the expiration_time is less than or equal to the current consensus time when the ScheduleCreate is processed then the new SCHEDULE_EXPIRATION_TIME_MUST_BE_HIGHER_THAN_CONSENSUS_TIME error shall be returned.
• Add an optional wait_for_expiry bool field to the ScheduleCreate and ScheduleInfo protobuf. This field shall specify that the transaction will wait till expiration_time to attempt to execute.
  • if not specified, this shall default to false.
  • Setting this to false does not necessarily mean that the transaction will never execute at expiration_time.
    • If the signature requirements for a Scheduled Transaction change via external means (e.g. CryptoUpdate) such that the Scheduled Transaction would be allowed to execute, it will do so autonomously at expiration_time, unless a ScheduleSign comes in to “poke” it and force it to go through immediately.
    • Documentation shall be updated to explicitly call out this case.

wait_for_expiry == false

• wait_for_expiry == false transactions shall be executed immediately after the ScheduleCreate or ScheduleSign transaction that provides sufficient signatures to execute the transaction, inside the same handleTransaction call.
  • Calculating the throttles that are going to be used for a ScheduleSign or ScheduleCreate call shall be “recursive” and take into account transactions that they could trigger.
    • IE the throttling for ScheduleSign and ScheduleCreate for wait_for_expiry == false transactions will always count against throttles as if they are actually going to cause the transaction to execute.
    • This means that ScheduleSign operations will need to fetch the associated Scheduled Transaction from the db to calculate how much throttle they will use.
  • These transactions can still execute at expiration if the signature requirements on the transaction change such that there is sufficient signatures to allow it to execute and there are no ScheduleSign transactions before the expiration date.
  • This is how Scheduled Transactions behave before this HIP, except for the throttle calculation and execution case above.

Evaluation and Expiry

• Scheduled Transactions in all cases shall be evaluated for execution and expiration at the first available time after their expiration_time
  • The order they are evaluated shall be defined first by temporal order, then the order they are received.
    • Transactions that have a lower expiration_time shall always evaluate before transactions with a higher expiration_time.
    • For multiple transactions with the same expiration_time, the transactions shall be evaluated in the consensus order of their ScheduleCreate transactions.
  • A “best effort” should be made to evaluate Scheduled Transactions as close to their expiration_time as possible, given all system factors.
  • To be clear - Scheduled Transactions are NOT guaranteed to execute at exactly their expiration_time. They will execute at the first available time slot after that consensus time.

Throttling

• In all cases, Scheduled Transactions shall be throttled at creation with respected to their expiration_time, a scheduling.maxTxnPerSecond setting, and a contracts.scheduleThrottleMaxGasLimit setting.
  • The scheduling.maxTxnPerSecond setting shall be the limit on the number of Scheduled Transactions that can be created with an expiration_time within any given second.
  • A scheduleThrottles shall be calculated by scaling existing throttles such that the total per second allowed is the same as scheduling.maxTxnPerSecond.
    • The algorithm for this shall be defined by the following pseudo java code:

        int maxTps = getSetting("scheduling.maxTxnPerSecond");
        var scheduleThrottles = existingThrottles.copy();
            
        // if it's imposible to scale the throttles, throw exception
        if (maxTps < scheduleThrottles.allThrottles().size()) {
          throw new Exception();
        }
            
        int left = 1, right = Integer.MAX_VALUE;
        while (left < right) {
            int m = (left + right) / 2, sum = 0;
            for (var i : scheduleThrottles.allThrottles()) {
                // we don't allow any one throttle to go below 1
                int a = (i.tps() + m - 1) / m;
                sum += a < 1 ? 1 : a;
            }
            if (sum > maxTps)
                left = m + 1;
            else
                right = m;
        }
            
        scheduleThrottles.divideAllThrottlesBy(left);
            
        //set all throttles that are 0 to 1
        scheduleThrottles.setAllZerosToOne();
            
      

  • The contracts.scheduleThrottleMaxGasLimit setting shall define the max gas that Scheduled Transactions can use in any given second in the future.
  • Evaluating if a ScheduleCreate transaction exceeds throttling shall be defined as follows -
    • Get all the transactions already scheduled for the second that the expiration_time falls in.
    • Send all the transactions plus the new transaction through scheduleThrottles
      • if the throttles are exceeded, reject the transaction
      • Add a SCHEDULE_FUTURE_THROTTLE_EXCEEDED error code for this case.
    • Add up the gasLimit for all the transactions plus the new transaction.
      • if the total gas limit is greater than contracts.scheduleThrottleMaxGasLimit, reject the transaction
      • Add a SCHEDULE_FUTURE_GAS_LIMIT_EXCEEDED error code for this case.
      • NOTE: like throttling at the GRPC level, gasLimit is the max amount of gas the user specified in the transaction, not the actual gas it uses.
  • Throttling cannot happen in pre-check, it shall happen in handleTransaction.
    • ie ScheduleCreate transactions can fail after they are submitted due to throttles.
  • Because of the throttling described above, Scheduled Transactions executing at their expiration_time shall be -
    • exempt from all throttles.
    • exempt from congestion pricing.
  • Scheduled Transaction’s executing immediately after a ScheduleSign or ScheduleCreate due to wait_for_expiry == false shall continue to be subject to congestion pricing and throttling.
    • Since the worst-case throttle and gas usage are checked at the HAPI level, the throttles should never be exceeded in this case.
  • When planning capacity, Hedera should consider up to scheduling.maxTxnPerSecond extra transactions being executed per second.
  • When planning capacity, Hedera should consider up to contracts.scheduleThrottleMaxGasLimit extra gas being used per second.
  • When starting the system after downtime, Hedera should expect all delayed scheduled transactions to process rapidly, faster than throttles would normally allow.

handleTransaction logic

• During handleTransaction, after the “current” transaction executes, Scheduled Transactions with an expiration_time less than the current second shall be executed or expired.
  • The number of Scheduled Transactions that execute shall be up to 999 per handleTransaction call.
    • This is based on there being up to 999 extra consensus timestamps available per call to handleTransaction.
    • A smaller number of transactions shall execute if the “current” transaction causes other synthetic transactions to process before we execute Scheduled Transactions.

Required Signatures

• If sufficient signatures are not received before the expiration_time, Scheduled Transactions shall expire when they are evaluated, without executing, in all cases.
  • There is a corner case where a Scheduled Transaction is still in the database after it’s expiration_time, pending evaluation, and a ScheduleSign or ScheduleDelete comes in for it.
    • Add a SCHEDULE_PENDING_EXPIRATION error code for this case.

Fees

• ScheduleCreate containing ContractCall transactions shall have a fee of $0.10.
  • ScheduleCreate shall remain at a $0.01 fee for all other transaction types.
  • Fees will still be dynamic based on resource usage.

Transaction Types

• Hedera should consider changing the scheduling.whitelist setting to include the following transaction types - ConsensusSubmitMessage,CryptoTransfer,TokenMint,TokenBurn,CryptoCreate,CryptoUpdate,FileUpdate,SystemDelete,SystemUndelete,Freeze,ContractCall,ContractCreate,ContractUpdate,ContractDelete.

Mirror Node Change

• Add wait_for_expiry and expiration_time to the mirror node schedule api.

Documentation Updates

• Update documentation to reflect all new fields and logic.
• Add documentation indicating that the transaction ID of the transaction executed by a Scheduled Transaction is the ScheduleCreate transaction ID with the scheduled flag set to true.

Backwards Compatibility

This HIP is backwards compatible. Transactions from earlier versions of the system simply keep their expiration_time and have wait_for_expiry = false.

There will be a migration process that runs on the first startup of the system after this HIP is incorporated.

Security Implications

This HIP will allow a much larger number of Scheduled Transactions to be created.

• This could impact performance and memory usage and lead to a denial of service attack.
• We will change from storing all Scheduled Transactions in memory to storing them on disk with fast indicies.

Scheduled Transactions could theoretically block other transactions from executing if they consume all the system throttles

• The throttling in the Specification section should mitigate this

Adding more types of transactions that can be scheduled adds to the attack surface of Scheduled Transactions.

• Extensive testing with all transaction types in scheduling.whitelist will be needed.

How to Teach This

• Any documentation for Scheduled Transactions will need to be updated to reflect the new fields and semantics.
• The fee calculator needs to be updated to support the new fees for ScheduleCreate based on it’s content.

Reference Implementation

An incomplete prototype implementation is at https://github.com/hashgraph/hedera-services/tree/long-term-scheduled-transactions. A production version will follow.

Rejected Ideas

• Add separate signing list to the Scheduled Transaction from the requirements to sign the inner transaction.

Open Issues

N/A

References

• https://github.com/hashgraph/hedera-services/blob/master/docs/scheduled-transactions/revised-spec.md
• https://docs.hedera.com/guides/docs/hedera-api/schedule-service
• https://docs.hedera.com/guides/docs/sdks/schedule-transaction
• https://docs.hedera.com/guides/docs/mirror-node-api/rest-api#schedule-transactions

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: