How to Use Bitcoin Data in Clarity Unit Tests

One of the biggest value propositions of Stacks is its connection to Bitcoin. The Stacks blockchain has 100% Bitcoin finality and offers a native BTC-yielding product through stacking. Now, the network also enables sBTC, a new trust-minimized Bitcoin-backed asset.

But one important piece of this Bitcoin connection that isn’t talked about as much is that Stacks’ smart contract language Clarity has read-access to Bitcoin. That read-access unlocks powerful use cases because it means Clarity smart contracts can react to Bitcoin L1 transactions.

For example, a few years ago, along with some other Stacks devs, I worked on a primitive called “catamaran swaps” that leveraged this read access and enabled users to trustlessly swap between assets on Stacks and Bitcoin.

Clarity smart contracts can react to Bitcoin L1 transactions.

In essence, this swap contract takes a Stacks asset (sBTC, STX, etc.) into escrow and holds it until a matching Bitcoin transaction is submitted. The contract can read Bitcoin’s state and can verify that the Bitcoin transaction was mined. If verified, the contract will then release tokens on Stacks from escrow to the user. An entirely trustless asset swap.

You can imagine all kinds of use cases for this kind of swap: lending protocols that rely on Bitcoin deposits, minting new tokens, the list goes on. 

So how does this read-access actually work?

Verifying Mainnet Bitcoin Transactions With Clarity

With the consensus mechanism Proof of Transfer, Stacks has a view of the Bitcoin network. That view is limited to the block header hashes of each Bitcoin block, but that limited view is still useful—you can use this information in a Clarity smart contract to verify that a Bitcoin transaction was actually included in a Bitcoin block.

The Clarity Bitcoin library, a stateless contract deployed on mainnet, does just that. It enables you to verify that a transaction was mined in a certain Bitcoin block.

At a high level, the contract’s two functions <code-rich-text>was-tx-mined-compact<code-rich-text> and <code-rich-text>was-segwit-tx-mined-compact<code-rich-text> take all data as arguments (proof data) that are required to rebuild the block header hash using the Bitcoin transaction. Rebuilding the block header hash involves merkle trees and hashing. Finally, the <code-rich-text>was-tx-mined<code-rich-text> functions compare whether the calculated hash is the same as the one provided by Stacks consensus. 

You can learn more about the specifics of how to verify Bitcoin transactions in the GitHub repo.

The Challenge of Testing The Bitcoin Connection

This library works on mainnet today, and it is used under the hood to power the catamaran swap contract. You can also see this library in play with the Bitcoin Primer course. However, this library was impossible to unit test.

Unit tests happen locally, where tools like Clarinet offer test environments that simulate execution environments (simnet). In this environment, Bitcoin data is mocked and functions like <code-rich-text>get-burn-block-data?<code-rich-text> return mock hashes. Until now, it was impossible to provide proof data that would match the mock hashes.

In other words, there wasn’t a way to write unit tests for contracts that use the Clarity Bitcoin library or for the library itself (because Clarinet didn’t simulate valid BTC transactions).

It was impossible to test Clarity’s read access to Bitcoin in Clarinet’s test environments because they generate mock data.

Instead, you were limited to testing cases where the proof didn’t succeed or you could change the contract by adding some mocking functions that would overwrite the Bitcoin data with some mocked data.

Both options are not satisfying because the tests would not take real data into account. However, with the latest release of Clarinet we have a much better option now.

Pulling Mainnet Data into Clarinet

With the new mainnet transaction simulation feature in Clarinet, mainnet data can be pulled directly into simnet.

In the case of verifying Bitcoin transactions, rather than generating mock data (where the block hash is just a random string and thus any attempt to verify a transaction with that hash fails), Clarinet now pulls in actual block data from mainnet, including Bitcoin data (the block header hash). So, we can provide mainnet proof data to verify real Bitcoin transactions in a simulated environment.

As a first step, I was able to add a unit test to the Clarity Bitcoin library that proves a Bitcoin legacy transaction was mined on Bitcoin mainnet and another test for a Bitcoin segwit transaction. 

The unit test shows in detail which verification steps are done when calling the contract functions <code-rich-text>was-tx-mined-compact<code-rich-text> or <code-rich-text>was-segwit-tx-mined-compact<code-rich-text>. The test uses Bitcoin transactions from Bitcoin block 883230 because that block doesn’t have too many transactions, and therefore the proof data can be generated more easily.

The corresponding Stacks block is 595050. This value goes into the remote data section in Clarinet.toml, like so:

[repl.remote_data]
enabled = true
api_url = "https://api.hiro.so"
initial_height = 595050

For the proof data of a legacy transaction, there are public APIs available like mempool.space that help to create the data. The test calls <code-rich-text>was-tx-mined-compact<code-rich-text>, which you can see in line 94 here, and the expected result is the transaction id.

For the proof data of a segwit transaction, there are tools like bitcoin-tx-proof and clarity-bitcoin-client. These tools create the required witness merkle proof and the coinbase merkle proof using a Bitcoin core node (can be pruned). We ran the tool once to fetch the blockchain data. Afterward, the logged proof data is stored as <code-rich-text>cachedProof.ts<code-rich-text> to speed up testing. In the future, these tools will probably be available as public APIs in the same way that proof data for legacy transactions is available today.

The data can be used directly in the contract calls after converting the JSON data to Clarity values. The helper functions <code-rich-text>proofToArray<code-rich-text> and <code-rich-text>wasSegwitTxMinedCompact<code-rich-text> in our unit test do this for us. The function <code-rich-text>was-segwit-tx-mined<code-rich-text> returns the wtxid (or hash) of the transaction. In the test, we see that the result matches the value that is calculated from the txProof data.

In a second step, I was also able to add those same unit tests for a btc-stx swap between Alice and Bob. In this test, Alice creates the swap, places STX tokens into escrow, and provides her address as a scriptPubKey.

Bob submits the same Bitcoin transaction we used before and receives the STX tokens (you can reference the unit tests for legacy transactions and segwit transactions here). The tests do not verify that the Bitcoin transactions were mined, but they show that submitting valid Bitcoin transactions triggers an action, in this case an actual STX transfer.

These unit tests help to better understand our contracts and increase confidence in their correctness. The tests described here were written for Clarinet epoch 2.1. Future versions of Clarinet will also support block data for more than one block (enabling contracts in epoch 3.0 and later to be verified). More tests will also be added in the future to cover even more cases, such as when the amount of transferred Bitcoin is less than the expected amount.

Learn more about unit tests in Hiro docs.

Example of Fetching Bitcoin Mainnet Data

In the code of the unit tests above, you can see comments noting how to fetch hex encoded transactions and block headers using the mempool.space API. The first call fetches the transaction:

https://mempool.space/api/tx/c1de234c01ecc47906117d012865ce3dabbbb081dc0309a74dbbae45e427aadc/hex

‍

And the second fetches the block header:

https://mempool.space/api/block/00000000000000000001c55626b85b4b3ecb33f67645356a2b01f4dfba893679/header

‍

Using the BitcoinJS library, we can then convert the transaction hex of a segwit transaction into the expected wtxid—that is the hash of that transaction. 

const bitcoinjsTx = bitcoinjs.Transaction.fromHex(txProof.transaction);
   const expectedWtxid = bitcoinjsTx.getHash(true).reverse().toString('hex');

‍

The BitcoinTxProof library has convenient methods to query a Bitcoin core node and fetch a decoded transaction:

const btcRPCConfig: BitcoinRPCConfig = {
  url: 'http://localhost:8332',
};

const btcRPC = new BitcoinRPC(btcRPCConfig);
const blockHash = await btcRPC.call('getblockhash', [blockHeight]);
const txObject = await btcRPC.call('getrawtransaction', [txid, true, blockHash]);

‍

Finally, there is also the complete call to get the proof data for a segwit transaction with the clarity-bitcoin-client:

const data: ProofRequest = await getProofData(req.params.txid, blockHash, getRpcParams());
const pgd: ProofGenerationData = getProofGenerationData(data);
const proof: TransactionProofSet = extractProofInfo(pgd, data);

Conclusion

With this feature, it’s now much easier to build safer smart contracts that tap into the Stacks <> Bitcoin connection because you can pull valid Bitcoin data directly into your test environment and make sure your code behaves as expected before deploying to mainnet.

Alongside verifying Bitcoin transactions, this new feature also allows you to write tests for smart contracts interacting with other deployed contracts, like an oracle.

If you have any questions, you can reach out to me aka friedger.btc at the #clarity channel on the Stacks Discord.