Contents
Note: hyperchains have been renamed subnets, due to the trademark for "hyperchains" already being registered. Learn more here.
Hyperchains Progress Update
Earlier this week, we announced that this scaling solution for Stacks needs a new name due to "hyperchains" already being trademarked. You can learn more about that announcement here and why we favour the new name "subnets." We are letting the community weigh in on the issue, and for this post, we will continue to refer to this upcoming solution as "hyperchains."
As discussed in our last hyperchains update, hyperchains have been live on testnet since June 2022. Since then, we launched an NFT use case to highlight the deposit-withdraw features on hyperchains. Watch a new video on Hiro's YouTube channel to see how to interact with an NFT on a hyperchain.
This use case is live only on devnet—a local Stacks network spun up inside Clarinet. For now, the NFT use case is in a controlled environment to help us set performance benchmarks for both throughput and latency.
Our initial goal to speed up the hyperchain has been to reduce confirmation times. One of the significant performance benefits of hyperchains is that because they functionally have a single miner (whether they truly have a single miner or a federated set), microblocks are guaranteed to be confirmed by the subsequent anchor block. This means that transactions are confirmed much faster than in the stacks-node: hyperchain microblocks count as confirmed blocks in hyperchains.
Learn more about subnets and the challenges with scaling a blockchain in a conversation I had with Max, Hiro's Developer Advocate, on our YouTube channel:
Our Approach to Performance Benchmarking and Improvements
In the current implementation, we are diving deep into how microblock production works and if a transaction is broadcast to a hyperchain mempool, how quickly it is included in a microblock. We are also evaluating the dependency on Stacks block confirmation times and Bitcoin confirmation times. This benchmarking will also assess how often the hyperchain miner's block commit possibly misses the intended Stacks block. With this data, the logical next step is to improve the performance based on the benchmarks.
To paraphrase Arthur C. Clarke - We are relentless in our commitment to test the possible and to go beyond into the impossible.
To MARF or Not to MARF
Hyperchain performance is ultimately dependent on hyperchain block production and confirmation speed. Performance might also be influenced by MARF improvements- precisely the pace of operations in the SortitionDB and ClarityDB, both of which use the MARF.
Hyperchains will essentially be free of forks, so there is less need to use the MARF in its current form. The purpose of the MARF is to allow nodes to quickly validate the value of a particular key in ANY fork. Replacing the MARF with a flat key-value store will lead to considerable speedups.
We are replacing the MARF in hyperchains to use a flat key-value store, implemented with an SQLite database. The key/value updates with each new block, and those values will be stored in the database. As a result, when a new block is created:
- If the block builds off the current chain tip, then the key value updates will be applied.
- If the block builds on a fork, and the fork is shorter than the current tip, the key value updates are stored as a blob but not applied to the flat key-value store.
- If the block builds on a fork, and the fork is longer than the current tip, the blockchain will "undo" the key value updates up until the point of the fork and then apply the updates in the blocks in the fork.
Multi-Party BFT
Our previous version of the hyperchains contract validated the anchor block submission by allowing any miner to commit an anchor block. However, to support BFT consensus, the block commit public function has been updated to only allow a commit to succeed if it was signed by the threshold number of miners.
We added a new RPC interface for receiving block proposals to the hyperchain nodes to allow the participating miners to accept the proposed block data and provide their signatures over that data. The signature is then verified as coming from one of the other miners.
Block (and microblock) assembly in a hyperchain is a process that will involve one leader. This leader assembles the block and then proposes it to the other members of the mining coalition. This leader (or, failing that, any other members) can issue the block commit when enough members have signed. This leader is at the heart of the BFT algorithm, selecting and rotating when necessary. This Leader Selection doesn't modify much of the behaviour of the neon_node; the leader is going to act the same as the current miner, except for the changes in the other multi-party mining issues.
Other notable changes associated with this:
Sender validation to the block proposal RPC: this ensures that the proposal submitted was created by the multi-party leader. This is necessary to prevent the block proposal interface from being a denial-of-service vector.
Multi-party anchor block signatures: multi-party block commitments via a multi-party contract. The multi-party contract checks the transaction sender and the supplied signatures to confirm that at least the required-signers count of approvals has been achieved before using a block commit.
The current implementation of microblocks assumes a single-signer setup. The hyperchain node is being updated to support federated miners producing microblocks.
Help Us Test Hyperchains
We also recently released 4 new Clarinet features that offer hyperchains functionality. Namely, the command clarinet integrate lets developers spin up a hyperchain node and test locally. We'd love your help testing hyperchains. To try the NFT use case on devnet, please follow the instructions on GitHub here. To spin up your own hyperchain node, follow these instructions. If you experience difficulties in the process, open an issue on GitHub.
Spinning up this demo and a hyperchain node requires familiarity with Clarity contracts and the Stacks blockchain. If you're new to the Stacks ecosystem, you may want to start with Stacks documentation and the Stacks blockchain GitHub.
Join the Hyperchains AMA on August 11th
We know everyone is excited about scaling solutions for Stacks. Bring your questions and join the conversation on August 11th 3pm EST for the Hyperchains AMA on Twitter (you will find the event at our account @hirosystems). You can submit questions live in the Twitter Spaces event or ahead of time on the Hyperchains AMA Discord channel. Set a reminder for the event here.
Future Work for Hyperchains
We have delivered the core functionality aspects of a hyperchain so far and made lots of progress to reach this point, and as we enter the next phase in hyperchains development, there are several projects we are working on, including:
- API integration and Stacks.js support: Hyperchains introduces new events for deposits and withdrawals, which means additional support needs to be added to our API. We are updating our API by adding event handlers that react to the deposits and withdrawals on a hyperchain. This also involves updating the current burnchain references from Bitcoin to Stacks for hyperchains.
- Performance benchmarking & improvements: We are running a benchmarking exercise to gauge the current performance of hyperchains in terms of both throughput and latency. Improvements to MARF are also underway, which will further improve throughput. More details to follow after our benchmarking exercise.
- Abstraction layer for plug and play: We are working on building an abstraction layer - abstracting lib clarity's public functions that will allow for VM-specific implementations. This will be the foundation for the ability to plug and play with different flavours of VMs on a hyperchain.
- Rollups: What is a hyperchain without rollups? While still in the inception phase, we believe rollups will further increase the throughput of hyperchains. Optimistic rollup operators will bundle multiple off-chain transactions together in large batches before submitting them to the main chain. We will continue to post more updates on rollups architecture and next steps as we tease out further details on their implementation on a hyperchain near you.
Conclusion
That’s it for this update. We’ll share more progress on hyperchains in the coming weeks. If you are interested in learning more about our progress, join the Hyperchains AMA on August 11th at 3pm EST on Twitter. Set a reminder for the event here and submit your questions ahead of time on the Hyperchains AMA Discord channel.
