Gossamer: Into the Polkdot-verse Pt. 2

Authored by Ben Adar

Achievement unlocked! Full sync to Polkadot, and other updates from the Gossamer team

For those who are unfamiliar, Gossamer is an implementation of the Polkadot Host, formerly known as the Polkadot Runtime Environment. To learn more about what exactly that is, watch our introductory talk from DotCon during Berlin Blockchain Week.

We recently celebrated our most significant milestone: achieving full sync with Polkadot and Kusama!

With this achievement, we are a step closer to a multi-client reality on Polkadot. But just as importantly, we are a step closer to a productionized Gossamer, which means Gossamer will be responsible for block production and finality alongside Substrate nodes on a live network. This means we must ensure implementation of the consensus matches with Substrate's! To do this, we will test consensus interop on a multi-client devnet (more on this later).

Nevertheless, we've made great strides towards a production-ready state in the past few weeks, including refactoring our documentation page to reorganize and update a lot of the outdated content.

Finally, Gossamer Tech Lead Elizabeth Binks recently gave a talk at the annual Polkadot Decoded event. She ran a workshop on how to run a custom Polkadot-compatible chain with Gossamer. You can check out the workshop [here](https://www.youtube.com/watch?v=xyiwCiv9JtE). We also wrote a supplementary blog post/tutorial, which you can try for yourself here.

Node Optimizations

Since our last update, we have made incredible progress in optimizing the Gossamer client and its performance:

• Offline pruning

• Memory optimizations

• Gossip algorithm

• Sync speed

Starting us off, the Gossamer team implemented offline state pruning to reduce the disk space used by the node. The pruning algorithm is "offline," which means the node must be stopped for the pruning of the database to occur. The offline state pruning algorithm used by Gossamer is similar to that of go-ethereum and utilizes bloom filters to check if historical trie nodes are still in use.

A Bloom filter is a data structure for efficiently testing set inclusion. Using the bloom filter, elements can be added to the filter in O(1), and elements can be tested for inclusion in O(1). It operates probabilistically to determine if an element is definitely not in the set or if it probably is in the set with some fixed error probability for false positives.

The pruning algorithm firstly constructs a bloom filter by traversing the database for each trie node in the recent state window, by default, the most recent 256 blocks. Each node in each block state trie gets added to the bloom filter.

The algorithm works as follows:

1. For each block in the state window, traverse its state trie and add all nodes to the Bloom filter.

2. Then, iterate through the database and check each node in the database for inclusion.

• Nodes that are definitely not used inside the window get deleted.

• Otherwise, they are kept as-is.

We are currently implementing online state pruning, similar to Substrate's algorithm, so that node operators do not need to stop and restart their nodes to prune.

On top of the offline state pruning implementation, the team has improved Gossamer's memory usage. This work includes improving memory usage when syncing by only keeping the current state trie in memory and increasing the default garbage collection frequency.

The team has also improved the gossip algorithm by implementing Substrate's "polite gossip" algorithm, ensuring duplicate messages do not get sent to peers.

While node optimization is still an ongoing challenge for the team, with an improved pruning algorithm, memory usage, gossip algorithm and sync speed, Gossamer nodes are well on their way to be production-ready!

(･ω･)b

Establishing Internal Development Network

In other Gossamer news, the team has focused on updating GRANDPA, particularly the networking protocol related to GRANDPA, which has since been updated by Substrate since the Gossamer implementation was written. GRANDPA, which stands for GHOST-based Recursive Ancestor Deriving Prefix Agreement, is the finality gadget that each authority node runs to decide on block finality. To test the authority functionality (i.e. block production and finality) of Gossamer nodes, the team has created an internal development network running only Gossamer nodes. The team has established a set of test cases to work through to ensure the node is operating as intended.

These test cases include ensuring:

• Block production (BABE) functions as intended.

• Finality (GRANDPA) functions as intended.

• Runtime upgrades work as intended.

• Non-authority nodes can sync properly.

• Telemetry and RPC connections, key management, metrics, and monitoring are all in place and working as intended.

After the Gossamer-only devnet milestone has been completed, the next step is to create a cross-client internal development network to test that the Gossamer BABE and GRANDPA implementations interoperate with the Substrate implementations. Finally, we will be moving towards a Gossamer audit and productionizing the codebase after that is complete.

Keeping An Eye on the Metrics

As an ongoing process improving Gossamer, the team has implemented monitoring to track and report key metrics. Using Prometheus, the team will be monitoring and gathering metrics on network traffic. The team will also be using Grafana to visualize the data captured by the metrics.

With a closer look into the metrics, the team will further develop and improve Gossamer in the places that count.

As supplementary reading: our neighbourhood Eth2 Lodestar team also recently discussed the importance of metrics as a tool for improving node performance. You can check it out here!

General Updates

Leading up to having a fully productionized Gossamer, the team has refactored their SCALE implementation to be more idiomatic and performant. SCALE is the codec used to encode and decode data that is used by the Polkadot protocol.

The team has also continued work on the RPC layer and polkadot.js tooling. We are at a point where polkadot.js + UI is fully functional with v4.6 API, and we can submit RPC calls and visualize blocks and accounts.

Next Steps

Moving forward, the team will continue to work through development network test cases and identify issues and areas of improvement. The team will also be implementing online state pruning, where, in contrast to offline bloom filtering, the node does not need to be stopped for the pruning process to run. The team is also working towards an internal, then eventually public, cross-client testnet. This means we will have a test network containing both Gossamer and Substrate nodes authoring and finalizing blocks soon!

Get Involved

If you are interested in getting involved and contributing to the project, check out our Github. If you would like to get in contact with one of the Gossamer team members, feel free to drop by on Chainsafe's Discord or email info@chainsafe.io. We would love to know more about you, your team and your project!

For more details on Gossamer, please head to our documentation site.

Learn more about ChainSafe by visiting our website, through our Medium, via Twitter, or visiting the ChainSafe GitHub.

Thank you to Timothy Hao Chi Ho, Elizabeth, Dustin Brickwood and the entire Gossamer team for all the help writing this article!