Lodestar Post-Interop Update

Authored by Colin Schwarz

Written by Colin Schwarz, Marin Petrunić, Cayman Nava, Gregory Markou and Eric Tu

Last week the Lodestar team was privileged to participate in the Eth2 interop week up in Skeleton Lake Muskoka. The week was a huge success for Lodestar, for the other implementation teams and for the development of Eth2.0 in general. We accomplished a tremendous amount in a short period of time and are pleased with the number of implementations that could interoperate after only a week.

Pre-Interop Testing

The teams arrived at the cottage on Friday, September 6th (day 0) and stayed until the next Friday, September 13th (day 7). Going into the week, we felt a little behind some of the other teams. We had been building a bunch of modules, and we passed spec tests, but had never stood up a full node before. We had never run a full beacon node and our networking was also still unstable. As a result, one of our first priorities was to get the new networking spec working. This would enable us to connect two beacon nodes together and receive gossiped data over the network. Leading up to the retreat, we had been working hard on the validator API but still didn't have a stable one going into the week. This is an important component because the beacon chain relies on validators to drive the chain forward. If you don't have any validators and are not connected to any peers who do, the chain just sits there and there are no blocks to be processed. Early into interop week, we were able to get that API working which was critical for actually being able to use our validator client and try the chain. Greg nicely summed up our progress on day one: "By noon we had started the chain which we had never done. It was doing things and we were like ‘holy f**k, this works'. Then by the end of day one we started to attach validators to it and we were like ‘holy sh*t, this really works!" At this point there were a lot of errors but we knew what was causing them. It took the rest of the weekend to get to a point where we were confident that our implementation was internally stable. This is when we were able to begin interoperating with other clients

Interoperation

The first client that Lodestar interoped with was Lighthouse. The process was pretty straight forward although there were some issues with our different implementations of libp2p. After the first interop with Lighthouse we moved on to become interoperable with Nimbus and Artemis. By the end of the week Lodestar was also interoperable with Trinity, Harmony and Prysm. It was incredible to see so many teams successfully interoperating in such a short period of time. This is a feat that could only have been achieved so quickly with all the teams together in the same place, in a context that allowed them to focus exclusively on development. Having all of the other implementer teams constantly available and within steps of one another was invaluable for working through a lot of the issues that came up. For example, we found that everyone who implemented the Gossipsub spec themselves didn't implement message signing as part of the protocol. As a result, our implementation ended up rejecting others messages because they weren't signed! This is the kind of discovery that the interop week allowed for. There was no way we would have been able to check in with all the other teams before making every minor decision, so to get everyone in the same cottage constantly working together, testing and communicating was invaluable to being able to find a lot of issues that we otherwise might have missed.

Fulfilling the vision of libp2p.

One of the biggest challenges of the week was to get five different implementations of libp2p interoperating. A lot of translation and interpretation had to go into these various implementations which were written in Javascript, Golang, Python, Java and Rust. Over the week we put in a lot of work to get different implementations of gossipsub (subprotocol within libp2p) working in harmony and discovered a lot of minor inconsistencies through the process. For example, the Rust implementation didn't flush the buffer when it was doing an initial identity handshake so our implementation was hanging, waiting for a response that never came. Some other implementations (Java and Python) implemented the identity protocol that week! Cayman explained how this is a good example of finding "...little things that you wouldn't immediately think about, little nuances that you have to just work through... I don't know rust libp2p but the guy who wrote rust-libp2p (Lighthouse's fork) was right there." Eric elaborated: "You might not even have found it until way later on when you actually try to connect the client, because if you're running Lodestar to Lodestar and if your sh*t is both wrong in the right ways it'll still work." Libp2p was thus another area in which the value of having everyone together in one space became really apparent.

When we started to interop with other teams, Lodestar blew up every client. This led to a running joke that whenever our team walked into a room, someone would say: "who called the fuzzy testers!" This was due to the fact that not every implementation followed the spec to the last word. This is mostly because unlike some other teams, we do not maintain our own fork of libp2p. Cayman explained: "Our libp2p implementation was kind of more beholden to a slower pace, a more conservative pace of development, because we already had IPFS in production using js-libp2p. So for instance, the lighthouse guys were just like, oh there's a bug in our Rust-libp2p and then went in a just fixed it and immediately republished, merged it pushed it up, and it's fixed." This is easy for lighthouse to do because they maintain their own fork of libp2p. For us it was a bit trickier because we don't have a deep understanding of the libp2p-js codebase (we only wrote the gossipsup-js codebase). Due to only having a week for interop, we worked directly with the libp2p team in order to fix issues across implementations. This added an extra step to the process but we are grateful for all the help and support we received from the folks at libp2p. Near end of week we started a big libp2p Telegram group with all relevant people including the libp2p team to work through all the incompatibilities and this went a long way to getting every implementation working together. Cayman explained: "I think that was probably one of the biggest immediate wins of the week, beyond getting these clients all talking together... I feel like we pushed a lot of the way forward on delivering on the promise of libp2p which is what people wanted in the first place: this interoperable, pluggable network."

Conclusion

Overall we are extremely happy with how the week went and what our team and others were able to accomplish. We would like to thank Danny Ryan and Diederik Loerakker for helping to lead to processes and giving a lot of excellent advice and of course Consensys for their generosity in funding the retreat.

We are excited to move forward with the next stages of Lodestar's development which will include a productionized beacon chain, further optimizations, research into phase 1 and of course, research and development of a light client.

Stay tuned!