Lodestar’s Pectra Retrospective and Future Fork Ideas

In response to Ethereum’s request for reflection on the Pectra hard fork cycle, the Lodestar team put together some reflections, recommendations and some opinionated ideas on how we can work better with the broader Ethereum protocol community. Our main goal is to provide some perspective on what we observed, experienced and can potentially do better to ensure Ethereum’s ambitious roadmap continues to serve the ecosystem’s needs without compromising the stability of Ethereum’s quality of service over the last decade, a main feature of our network.

Observations from the Pectra development cycle

Due to the nature of the current hard fork coordination process, we have to realize that some of the failures are not necessarily technical in nature but rather psychological, as side effects of how we organize this process of implementing features for a hard fork. 

Inconsistent perspectives on fork scope

The problems originated when there was broad consensus to keep Pectra a “small fork” to be shipped in 2024. Without this enforcement throughout the process, we allowed Pectra to balloon into a large enough hard fork that it had to eventually split. The process was flawed from the onset as this created alternative perspectives of what should have been included in a hard fork due to a shorter than normal turnaround time of up to a maximum of 8 months rather than 12+ months. In retrospect, we had many iterations of Pectra, which consisted of enough scope creep that many popular memes emerged.

Feature cramming is a side effect of slow release cycles

Research velocity and the landscape of Layer 1 blockchains change the competitive environment faster than our hard fork implementation cycles. We must accept that researchers, out-of-protocol implementers and the competitive L1 landscape change the environment of ideas, leading to the reprioritization of efforts more often. Our infrequent hard fork cycles may lead to stale hard fork features and degrade the importance of a feature inclusion. We must also realize that the long hard fork development cycle incentivizes this heightened sense of urgency to include features in an upcoming hard fork. Unless we have a solution to address our ability to deliver hard forks in a shorter timeframe, we will inevitably receive “urgent” EIP inclusion proposals from everyone.

Lodestar believes that with some changes to how we coordinate together, we can aim for shorter hard fork development cycles, which may help alleviate this problem. 

Scoping failures delaying implementation

Proposals often look simple from the start, but the implementation phase of adding the feature EIP into codebases generally tends to uncover complexities that aren’t fully realized until the period of implementation, which leads to context switching. Context switching is a logistical burden on client teams as it’s not always easy to drop a feature once it’s implemented, and the engineering hours spent on it become a sunk cost. We need a more thorough understanding of how included proposals affect client team’s codebases, ideally before they get scheduled for inclusion (SFI). EIPs should also not scope creep during the implementation process, but rather batched together upfront to target an objective. When blob targets increased via EIP-7691, it needed to be bundled with EIP-7623 to increase calldata cost for better effect. Objectives should be batched together with their relevant EIPs to prevent further implementation creep. 

EIPs need a standard of readiness for consideration

Currently, there are no guidelines or standards for inclusion criteria. Lodestar had advocated for EIP-7495 SSZ StableContainerand EIP-7688 Forward Compatible Consensus Data Structures with a clear and fixed feature scope, a devnet with two viable consensus clients demonstrating its usefulness and multiple mentions on various ACD calls. However, this wasn’t enough to get included in the hard fork, while other premature EIPs considered for inclusion (CFI), like  EIP-7702, were still debating major EIP design decisions, such as in-protocol revocation during its implementation. The team should have finalized design choices and community feedback before implementation. The rush to deliver this important feature made it a prime example of feature cramming due to slow development cycles. This approach creates an uneven standard for applying proposals to a hard fork, and there are no clear directions on how anyone can do the proper due diligence necessary to have an equal shot at inclusion.

Recommendations for future improvements

Although there will be some tradeoffs and disagreements with our recommendations, it should be made clear that Lodestar does not want to see any changes made that will compromise Ethereum's high standard of operational excellence in stability and uptime of mainnet. Increased disruptions to users of the Ethereum mainnet increase the likelihood of engineering burnout and require additional resources and diversions from sprint objectives. 

Shorter hard fork development cycles, smaller scopes

We should develop a faster cadence to shipping hard forks by keeping scopes smaller. One of the big issues with Pectra is its large number of changes—it becomes harder to test, and edge cases become more abundant by introducing new states that interact with other features in unknown ways. By isolating features from each other via hard forks, we can minimize the vector for errors and be able to conduct adequate testing to ensure mainnet stability. We believe that one coordinated EL+CL hard fork cycle is achievable every six months with tighter enforcement of timelines. This will also allow us to achieve some stability in thinking/planning for n+2 forks ahead with greater certainty. 

Planning n+2 forks ahead

Client teams like Lodestar need greater certainty of what is contained within the n+1 hard fork to organize our code branches for better feature development. If the dependent fork consistently changes, a lot of time gets spent rebasing branches. Having a locked-down n+1 fork specification will allow us to merge fork branches into our development branch more aggressively to move faster. We suggest that the n+1 hard fork’s meta EIP becomes frozen sometime before the mainnet deployment of fork n. 

Dedicate greater resources to specification design, testing and fuzzing

To maintain the high standard we have for quality assurance, we suggest:

• Dedicating more resources to specification hardening for greater edge case discoveries
• Dedicating more resources for testing, fuzzing and simulations to uncover failure scenarios of the implementations
• Dedicating more resources to maximize test case coverage

Keeping the scope tight for a specific hard fork makes the testing environment more predictable and can be more effective at uncovering devastating bugs.

Streamline specific, light upgrades in parallel to coordinated hard forks

There have been bottlenecks to the entire development cycle of a coordinated execution and consensus hard fork. As the ecosystem continues to grow, potentially with additional client teams, turnaround will increase as we prefer to have all client software upgraded simultaneously for network stability. A process must be in place to allow for streamlined, layer-specific hard forks to ensure we can quickly respond to network demands such as blob increases for the consensus layer. Further into this post is a framework to think about how we can have a parallel stream to better respond to network demands for isolated upgrades that bypass the main coordination bottleneck of multi-layer hard forks.

A framework for thinking about Ethereum L1 development

This framework is intended to outline an idea of how we could think about softly organizing the Layer 1 development process, using some ideas from other open-source development organizations, such as Kubernetes, as floated in the Ethereum Magicians retrospective thread. 

The goal of this framework is to consider how we can better organize roles and responsibilities to improve coordination and decision-making and achieve a smoother, faster hard fork development cycle for Layer 1.

The hardest coordination efforts are proposals which involve multiple layers, especially when the speed of readiness varies because the scope of work is different between execution and consensus clients. This is generally a bottleneck for development and prevents us from moving faster on proposals such as the blob-parameter only forks, which can be done as a CL Only fork with EIP-7742 in parallel to multi-layer hard forks. The idea of having three parallel streams would allow for the inclusion of more layer specific EIPs and achieve faster delivery where isolated layer changes can be decoupled. 

N+1 fork should be at a stage where it is fully scoped with the specification frozen. No additional EIPs can be added, no significant design deviations should occur, and they are all considered scheduled for inclusion (SFI). Client teams can then implement and achieve draftnets (idea envisioned by Danno), where the cycle of spec implementation details, coordination of draftnets, modification of spec tests and proactive stress testing (e.g. fuzzing, simulations, audits, etc.) can occur. There are still opportunities to decline for inclusion (DFI) should unforeseen complications evolve, which would significantly delay the fork. The n+1 fork should be the main focus of AllCoreDevs to strive for delivery of the scoped fork starting at month 0 of 6 in the cycle.

During this time, we are simultaneously working with a potential steering committee (consisting of multiple stakeholders TBD) to form consensus on the n+2 fork. There are two main purposes for the existence of this group:

• Consider new Proposed for Inclusion EIPs from special interest groups for consensus on CFI status
• Organize CFI’ed EIPs to scope the n+2 fork for Scheduled for Inclusion status

The steering committee receives EIPs from special interest groups (SIGs) with their proposed for inclusion EIPs. They are responsible for having a cohesive vision of their objectives and concrete design choices in their EIP(s) to minimize scope creep if CFI’ed. These proposals should ideally come with initial spec tests. Generally, the steering committee should also sort through other CFI’ed EIPs which may already be partially implemented in clients and organize what should be included with the n+2 fork by asking some example questions:

• How/where does this EIP fit with Ethereum's current mission and vision?
• What is the degree of inter-connectedness or complexity of the implementation scope for individual client teams? 
• Is this CFI’ed EIP in a state where it can be shipped within 6 months?
• How does this EIP affect future roadmap objectives? 
• Is this EIP a requirement to enable full functionality of a hard fork objective?
• Does this EIP become stale and less useful over time if delayed?

Special Interest Groups are more of a formal title that groups people together to achieve a specific implementation goal for Ethereum. This is already happening organically, where frequent interop groups, breakout meetings and coordination efforts happen externally to the ACD process. EVMMAX, Stateless, FOCIL, ePBS, L2 Interop, PeerDAS and EOF are examples of these special interest groups. The bulk of specification development, design choices, proof of concept devnets and community feedback should be done within these groups.

We hope that this framework for addressing the hard fork process will help us coordinate Ethereum L1 implementations better going forward. We will support the decisions that make the most sense to ensure Ethereum remains competitive yet stable and reliant, for securing billions of dollars in value. 

Contribute to Lodestar! 

At ChainSafe, we’re constantly seeking exceptional talent to join our teams and welcome contributors to the Lodestar consensus client. To any TypeScript developers looking to take on challenges and push the boundaries of the JavaScript ecosystem, get in touch!

To get involved, visit our GitHub repository

If you wish to contact the team, join Chainsafe's Discord in the #lodestar-general channel or explore our job openings page. Alternatively, you can email us at info@chainsafe.io.

Remember to visit our official website and follow us on Twitter for more updates.

Lodestar is built and maintained by ChainSafe, a leading blockchain R&D firm.

Website | Youtube | Twitter | Linkedin | GitHub