A walk into the Forest

Authored by Timothy Hao Chi Ho

The latest from ChainSafe's Filecoin implementation in Rust

Forest is ChainSafe's Rust implementation of the Filecoin protocol. Rust provides a foundation for building blockchains with memory safety and performance as a priority. Filecoin is an open-source blockchain-based decentralized storage network. Check out this primer by former Forest Tech Lead Austin Abell to learn more about the project.

Run, Forest, Run!

Thank you to Jonny Rhea for tweeting one of the best responses when ChainSafe announced in early March that our Forest client successfully synced up with the Filecoin mainnet. Our work, which began late 2019, resulted in a series of protocol-hardening efforts that culminated in our client finally participating in the very network it had been helping to enhance! And our gift was getting blessed with an epically relevant GIF.

🎊

Check out Forest Tech Lead and Chief Eric Officer Eric Tu demoing the sync here.

Of course, seeing a bunch of logs appearing in a terminal massively undersells the amount of effort that the Forest team had to put in to make this happen. So, in our first ever Forest project update, let's go through what had to happen to get to this point.

The road to syncing with Filecoin mainnet

Whereas Lotus, written in Go, is the reference implementation of the Filecoin protocol, developing Forest in Rust presented a different set of challenges:

• How do we verify the correctness of our implementation of the virtual machine (VM)?

• Does it actually match Lotus' functionalities given the language difference?

• And can it interoperate well with the other clients?

As it turns out, the only way to discover every single discrepancy is by syncing to one of the testnets (or even mainnet) to see where the interoperability fails, and debugging relentlessly until we have an interoperable client build. This process, to no surprise, is tedious and time consuming. It boils down to taking snapshots of Lotus and comparing the state tree at where the epoch and the state difference occurs, and finding out where in the code modifies this part of the state tree. We built a tool along the way to help us identify the two states exactly where the mismatches are.

On the importance of conformance vectors

One of our key learnings is how critical conformance vectors are in facilitating effective multi-client blockchain development. Conformance vectors are language-agnostic test vectors to check whether or not an implementation complies with a standard - in this case, whether Forest conforms with the Filecoin protocol and reference implementation. These tests are designed to be easily consumed by any implementation (regardless of language) because they are provided in a standard format like JSON. They are also meant to be easily updated, regenerated, and reviewed for protocol conformity. Conformance vectors are especially useful when changes are made to the reference implementation, since we can then generate a new set of test vectors to quickly check if our updates conform to the updated protocol.

Most protocols that have a multi-client ecosystem typically have conformance vectors. In Filecoin's case, since the protocol was still so early in its lifecycle, up until August 2020 only unit tests in Go were available. As such, for large periods of time, the Forest team struggled to test conformance to the reference implementation efficiently. Even when conformance vectors were finally released from the Filecoin team, it was difficult for test coverage to encompass every nuance of the state transition. This meant yet more manual testing to resolve consensus discrepancies.

On the debugging process

Specifically in regards to syncing Forest to mainnet, we had two options:

1. Sync directly to mainnet, or

2. Exporting the Space Race testnet chain and importing it into Forest

We could have attempted a sync directly to the Filecoin mainnet to see what happens. However, this would have been counterproductive. Our main focus was on verifying the correctness of our state transition before having to worry about ironing out networking issues.

Given that, it made sense to simplify by isolating and debugging instead. This meant exporting the Space Race testnet chain, and importing it into Forest. During the import, the Forest client would verify all the messages and perform the state transitions [essentially what happens during syncing]. Whenever errors appeared, we would troubleshoot them, and repeat this process until we were able to sync the entire chain.

Which we did.

On sharpening state transition and what that means for Forest's block production

One of the main reasons multiple implementations are written for a blockchain protocol is because client diversity improves network resilience. Since the most security-critical component of a blockchain is the state transition and verification, as a secondary implementation we focused on implementing and verifying the correctness of the virtual machine (which is a part of the state transition for Filecoin). The state transition must be matched bit for bit, which makes rewriting the virtual machine in another language no small task and our primary focus throughout development.

On the other hand, networking, block production, and storage & retrieval markets have more leeway in terms of implementation freedom. As a result, we aimed to reuse existing components from Lotus in order to focus our efforts on the VM. The goal was getting to a fully functioning node as quickly as possible.

Given this directive, the Forest team integrated the Storage Miner in Golang (built for Lotus) for mining blocks. Most of the work integrating the Lotus Storage Miner involved establishing a connection between the Forest process and the Lotus Storage Miner process over RPC, while also ensuring the RPC calls that the storage miner used were functioning as expected.

When you imagine the Forest software, you can look at it as this object that:

• Connects to the network;

• Verifies consensus with other nodes;

• Propagates blocks and messages via GossipSub;

• Maintains a Message Pool, etc.

Forest then passes off the block producing responsibility to the Lotus Storage Miner process, which includes computing PoSt, PoReps, and generating blocks.

With all this being said, we have only tested block production on Forest with the Lotus Storage Miner on a local devnet, since testing block production on mainnet would require our miner to meet the minimum 10 TiB storage power threshold for participation in consensus.

Updating RPC Methods

With syncing Forest making up a large part of our work, it would be easy to miss a lot of the other action as well!

Another thing we have been working on is updating our RPC methods to better match Lotus' implementation. This allows programs (like block explorers and wallets) that are already compatible with the Lotus API to have the ability to switch over to Forest. If the RPC interface is incompatible, this would prove to be difficult.

We have also added HTTP support to our JSON-RPC server, in addition to WebSockets. In order to do so, we had to refactor our RPC server to use the async-std compatible tide-websockets library. This allows us to offer a web server capable of handling multiple concurrent connections, which will be useful for either private or authenticated HTTP RPC API gateways.

Priming Forest for Network version 10 / Actors v3

Finally, the Filecoin network recently underwent a Network Upgrade to Network Version 10 (Filecoin's nomenclature for hardforks). This upgrade also introduced Actors v3.

[For an overview of the role of Actors, see this page. You can think of Actors as Filecoin's smart contracts - they are on-chain objects with their own state and set of methods.]

Actors v3 implements two Filecoin Improvement Proposals (FIPs): FIP-0007 and FIP-0010. The first of which improves the performance of the Hashed Array Mapped Trie (HAMT) and Array Mapped Trie (AMT), both of which are multi-block data structures abstracted as Interplanetary Linked Data (IPLD) HashMaps. The second proposal reduces gas consumption for SubmitWindowedPoSt messages by optimistically accepting proofs without verification and building in a dispute mechanism for off-chain verifiers.

[WindowedPoSt is one of two Proof of Space-time cryptographic challenges that miners have to submit to prove they are storing a copy of the sealed data. WindowedPoSt is Filecoin's guarantee that the miner has continuously maintained a copy of the data over time.]

The Forest team had to implement all these changes as the state transition had been fundamentally changed. This meant updating Forest's Actor and VM code to ensure parity with the protocol specs.

Next steps

Over the next month, the Forest team will be preparing for an audit of our codebase by Sigma Prime at the end of April. Prior to the audit, we will be updating our documentation for the audit team, which will have the added bonus of having it ready for public use post-audit.

On the development side, the main focus over the next little bit will be continuing to improve syncing performance and productionizing other parts of the code, especially as some parts are markedly better (VM and Actors) than others (syncing and message pool).

We are excited that Forest is inching ever closer to production. Once we have reliable sync and block production, it will be time to zero in on a thesis and a vision for Forest:

• How do we differentiate from other implementations?

• What subset of Filecoin users will Forest specialize for?

• How can we create synergies with other client implementations?

The ChainSafe team is eager to answer these questions for the community and we cannot wait to show y'all what we have in store. If you have any thoughts about Forest's vision, we would love to hear your thoughts on our Discord in the #forest channel!

Get Involved

Want to work with ChainSafe? COME JOIN US!!! Check out the new Careers section of our website and our open positions, and get in touch with us at careers@chainsafe.io if you are interested!

If you are interested in getting involved and contributing to the project, check out our Github. If you would like to get in contact with one of the Forest team members, feel free to drop by on Chainsafe's Discord, or email forest@chainsafe.io. We would love to know more about you, your team and your project!

For more details on Forest, please head to our documentation site.

Learn more about ChainSafe by visiting our website, through our Medium, via Twitter, or by visiting the ChainSafe GitHub.

Acknowledgements

Thank you to Amer Ameen, Eric Tu, & Hunter Trujillo. Your contributions were invaluable to the making of this article.