Gossamer: Into the Polkadot-verse Pt. 1

Authored by Timothy Hao Chi Ho

Syncing to Kusama

Pt. 1 - Syncing to Kusama

For those who are unfamiliar, Gossamer is an implementation of the Polkadot Host, formerly known as the Polkadot Runtime Environment. To learn more about what exactly that is, watch our introductory talk from DotCon during Berlin Blockchain Week.

As mentioned in our last Gossamer project update, ChainSafe's Gossamer team had been updating our Go implementation of the Polkadot Host to be compatible with Polkadot Runtime API v0.8. This sets the foundation for an important milestone in the Gossamer journey - synchronization with the Kusama network.

For the uninitiated, Kusama is Polkadot's value-bearing canary network (the "wild cousin") for fast experimentation and innovation. The value-bearing property of the network, along with the exploding ecosystem forming around Kusama, is an attestation to its staying power as a sovereign relay chain.

With this context, synchronizing the Kusama network using Gossamer-built nodes suddenly becomes a pivotal goalpost. Why? Success here will be a proving ground for our implementation's ability to interoperate with Substrate and other clients on a major network. The byproduct of this will be the further diversification of client implementations on Polkadot - an important aspect of the network's resiliency and security.

Current state of Gossamer

🎊 We are extremely proud to share that our Gossamer nodes have been able to successfully connect and sync with Kusama.

We have made many optimizations to node synchronization (which we shall discuss below) that were packaged together and released as a preliminary Kusama-compatible release under Gossamer v0.3.0.

That being said, the sync performance is still a bit slow, with a few outstanding issues that need to be smoothed out. This includes further optimizations to sync algorithms in order to sync fully to the head of the chain. However, the progress we have made is extremely promising.

For a video rundown of our Gossamer progress so far, be sure to check out Elizabeth Binks' talk from CSCON[0]: Gossamer: a Polkadot Journey in Go!

Weaving the path to full sync

Synchronizing a network like Kusama is no easy feat. High block heights, storage requirements, transaction throughput, and interoperability requirements make optimizing the node synchronization process a challenging problem to tackle. As well, we have to ensure functionality is consistent across all Gossamer nodes and the runtime is upgraded correctly.

Optimizing node storage

As the block height increases, more data must be validated & stored when syncing from genesis for each Gossamer node, which puts increased stress on the node itself. To address this performance bottleneck, we've added data structure optimizations to the Merkle-Patricia trie and database (which stores the state of the chain at each block), and made improvements to storage operations. We have also refactored in-memory storage trie to eliminate the storage of duplicated data, improving storage and retrieval speeds and reducing the overall memory usage. All of these factors allow Gossamer nodes to get to a performant full sync.

Here's a look at the numbers:

• Before the refactoring, Gossamer nodes were producing <1 block/second.

• Post-optimizations, the nodes are able to produce ~15 blocks/second

Our goal is to have parity with Substrate nodes, which produce ~40–80 blocks/second.

Choosing a Wasm interpreter

Amongst other optimizations, choosing a performant Web Assembly (wasm) interpreter, a core component of the Polkadot Host, has also been a priority for the Gossamer team. The interpreter is required to interpret the network's runtime, which is the state transition logic of the chain. The core logic of Polkadot is written in Rust and compiled to Wasm - in order to use the runtime with Gossamer, we need a Wasm interpreter.

The plan is to benchmark three interpreters: Wasmer, Perlin/Life, and Wasmtime. So far, we have been developing exclusively with Wasmer, which has shown itself to be stable and usable under Runtime API v0.8. That being said, Gossamer has been designed such that it is trivial to switch between supported interpreters through a config option. Once we have all interpreters updated to have compatibility with the latest Runtime API version, we will be able to flexibly test each one for the best performance.

Node interoperability

Aside from that, there are some important interop considerations on the road to a fully syncing node on Kusama. Network standards must be adhered to so that messages exchanged on a multi-client network (in this case, on Kusama between Substrate & Gossamer nodes) are universally understood. To account for this, we have made updates to support the current Substrate networking protocols. We have also moved over two message types (BlockRequest and BlockResponse) to protobuf encoding and decoding, which is a specification requirement for interoperability on Kusama.

Polkadot.js support, and adding JSON-RPC support and integrations

We are currently able to connect to Polkadot.js' web UI and see blocks through the interface. However, there are some issues with creating accounts when using the Polkadot.js UI that we are still resolving. Otherwise, using Polkadot.js through node.js works as expected!

Finally, the Gossamer team is continuing to add additional remote procedural call (RPC) support, which is used by polkadot.js. We are working to implement the remaining API endpoints and methods that are part of Polkadot Standards Proposal 6 (PSP-6). These updates will allow external services to interact with Gossamer nodes through a standardized interface.

Next steps

Now that we know we can discover, stay connected, and communicate with other Substrate nodes on the Kusama network, we will continue making improvements to the sync performance. In its current form, it is not perfect, and we plan to be able to sync to the head of the chain in the near future.

Beyond that, we will create a devnet of Gossamer-only nodes to further test performance. Once we are satisfied with this phase of our development, we will introduce testnets involving both Substrate and Gossamer nodes to test for interoperability.

Finally, if you want to learn more about our implementation, please head to our Gossamer documentation site. We will be updating it over the next few weeks with the latest updates we have made.

Be sure to stay tuned for the next announcement once we are ready for the community to run and test our Gossamer nodes!

Get Involved

If you are interested in getting involved and contributing to the project, check out our Github. If you would like to get in contact with one of the Gossamer team members, feel free to drop by on Chainsafe's Discord, or email info@chainsafe.io. We would love to know more about you, your team and your project!

For more details on Gossamer, please head to our documentation site.

Learn more about ChainSafe by visiting our website, through our Medium, via Twitter, or by visiting the ChainSafe GitHub.

Acknowledgements

Thank you to Elizabeth Binks, Dustin Brickwood, Ed Mack, Amer Ameen, & Aidan Hyman. Your contributions were invaluable to the making of this article.