Don’t rebuild for Canton. Translate to it.

Building on Canton Network used to mean a 100% smart contract rewrite in Daml. For Ethereum teams, that’s a massive technical tax and months of lost momentum.

We’re ending that bottleneck.

Introducing the ChainSafe ERC-20 / CIP-56 Middleware: A high-fidelity translation layer that lets you deploy to Canton using your existing EVM workflows, MetaMask signatures, and infrastructure.

Canton, meet Ethereum

We’ve built a high-fidelity interoperability layer that functions as a professional interpreter between the EVM and Canton’s privacy-first architecture.

The middleware consists of two core components designed for institutional reliability:

The API Translation Layer

This is the "brain" of the operation. It generates the specific data required so that a wallet like MetaMask perceives it is communicating with a standard Ethereum node. When you sign a transaction in your wallet, the middleware translates the EVM command into a native Canton call in real time.

The Dedicated Relayer Server

To maintain total asset integrity, we deploy a bespoke, white-label bridge for each partner. This server acts as the central authority for your specific token, locking assets 1:1 on Ethereum and mirroring them as CIP-56 standard tokens on Canton.

The problem: The Daml rewrite

Canton is the premier privacy ledger for the world’s largest financial institutions (G-SIBs), but it has been a silo. To tap into the $10 Trillion tokenization wave or settle against JPM Coin, you previously had to:

• Abandon your EVM stack for a total Daml rewrite.
• Spend months on technical debt before even launching.
• Retrain your entire team on a new VM.

The solution: High-touch, low-friction integration

We’ve been building custom dapps and complex smart contracts on Ethereum for 8 years. We aren’t just handing you a piece of middleware or a sign-in dashboard; we’re offering a bespoke co-development partnership to get you to market first.

ChainSafe holds supervalidator status on Canton Network and offers a dedicated infrastructure team that can help you deploy on Canton and navigate the ecosystem's privacy and security requirements.

Middleware specs:

• ERC-20 / CIP-56 mapping: Native translation between Ethereum standards and Canton’s institutional token standard.
• Supervalidator status: As a Canton Network Supervalidator, we provide the dedicated infrastructure to help you navigate specific privacy and security requirements.
• MetaMask native: Sign Canton transactions without switching wallets.
• Institutional rails: Immediate, native interaction with Canton-resident liquidity, including JPMD and DTCC settlement rails.

We specialize in high-complexity partnerships. Our goal is to handle the technical heavy lifting so your team can focus on the product, not the plumbing.

Fast-track stack: Middleware and Daml Autopilot

Most bridging solutions try to help you avoid Daml entirely. We help you master it at 10x speed.

Because we are a Supervalidator with a specialized AI context (the MCP Server), our middleware isn't a black box. It is part of a holistic development stack that allows you to write, validate, and translate complex Daml logic that traditional EVM-wrappers simply cannot replicate.

Automate Canton Network Development with Daml Autopilot

ChainSafe’s DAML Autopilot features 3,600+ compiler-validated patterns, built-in safety checks, and CI automation to streamline DAML smart contract development.

ChainSafeEvan Wells

Daml Autopilot

Precision tools for Daml smart contract development

While Autopilot handles the heavy lifting of writing and validating your custom Daml logic in record time, the middleware provides the production-ready plumbing to connect that logic to the world's most liquid markets. It’s the difference between learning a new language from scratch and having a professional interpreter.

Liquidity onboarding

This middleware allows institutions to lock standard ERC-20 assets (like USDC, EURC, or tokenized treasuries) on Ethereum and mirror them 1:1 as CIP-56 assets on Canton.

Unified wallet workflows

By translating ERC-20 commands to CIP-56, teams can manage Canton-resident assets using MetaMask and existing internal signing tools.

Accelerated market entry

Deploy your asset onto Canton today using your existing EVM logic and expertise, effectively skipping months of development latency and technical debt.

Claim your first-mover advantage

The window to be an early mover on the world’s most private institutional ledger is closing. We provide the support your team needs to move assets onto Canton without changing your internal security rules or retraining your staff.

Let’s discuss your project. We are ready to scope your engagement and start building immediately. Skip the rewrite and start integrating today.

✉️ Contact ChainSafe

Today, we are announcing a complete overhaul of the Canton Network development workflow.

ChainSafe is proud to launch Daml Autopilot, a tool that includes the first Model Context Protocol (MCP) server for the Canton Network, including CI automations to kickstart app development.

With built-in safety and compliance verification, Daml Autopilot acts as both a safety layer and an automated creation platform for any Daml smart contract project. Our assistant is grounded in 3,600+ verified canonical patterns from Canton Network to ensure your code is sound.

Canton's brand-new development workflow

For developers new to Daml or the Canton ecosystem, the learning curve is steep.

Canton Network's environment requires developers to be both code specialists and compliance experts simultaneously, a challenging (and often error-prone) balancing act.

ChainSafe has designed a dynamic context layer that sits directly between your Daml smart contracts and the Canton Network. LLMs alone are not intelligent by design. They are extremely good at completing patterns, but lack the guardrails to prevent probabilistic guesses from becoming production liabilities. 

Daml Autopilot grounds every suggestion in compiler-validated patterns, not statistical hunches. It handles the complexity and surfaces potential issues early, so your team can focus on core business logic.

How the Daml Autopilot MCP Server works

• AI-powered Daml development assistant with progressive context building.
• Access 3,600+ verified canonical patterns with semantic similarity search.
• MCP Server for LLM-based authorization model extraction and code analysis.

Most AI coding assistants rely on probability-based guesses. In financial systems, “probably right” means “definitely vulnerable.” Daml Autopilot treats code generation like a manufacturing process. It enforces safety gates that require patterns to compile and contain safety annotations before recommending them to you.

Daml Autopilot uses an MCP server that connects directly to the Integrated Development Environment (IDE) of your choice (e.g. Cursor, Claude Code, etc). It’s grounded in a Daml-safe-by-construction philosophy, unlike generic LLMs.

Instead of hallucinating syntax, the MCP server leverages a library of 3,600+ compiler-validated patterns sourced directly from official Daml and Canton repositories and documentation.

Using Daml Autopilot is a seamless, chat-based experience, but under the hood, the MCP Server includes two unique tools that assist in code development:

Daml Reason & Daml Automator.

Daml Reason

Daml Reason handles all pattern recognition, authorization model analysis, and safety reasoning. Daml Reason has built-in progressive context building, meaning it constantly updates its knowledge base from the latest versions of the Canton and Daml documentation.

Daml Reason ensures minimal hallucinations. It only analyzes what it actually sees, providing a safety layer that catches authorization errors and compliance issues, helping your developers catch issues before code review.

Daml Automator

Daml Automator guides developers through environments, CI/CD, and builds. Daml Automator is the component that creates and executes the code in local test environments before finalizing and pushing it.

Client-side orchestration enables developers to monitor code development in progress, keeping them in control of their work. No surprises, just instructions.

When you ask for help, the server executes a rigorous safety process: 

1. Pattern Matching: It searches established patterns for asset swaps, bonds, or options.
2. Logic Validation: It runs validate_daml_business_logic to check your code against compiler-validated structures.
3. Security Extraction: It utilizes debug_authorization_failure to analyze why a transaction failed, providing fixes based on Daml’s internal authorization model. It will not rely on random guesses.

How CI Automation Works

• Production-ready GitHub Actions for Daml and Canton CI workflows
• Eliminate manual setup with one-line installations
• Automate testing with Daml Sandbox environments

As a bonus, we are also providing CI Automations for those of you who work in a standalone capacity.

CI Automation eliminates the manual labour of installing Daml or Canton SDKs and configuring sandbox nodes for every new contributor. Instead, replace the tedious setup time with GitHub Actions that automatically standardize your pipeline.

Skip the pain. Drop these workflows into your repository and get automated Daml testing, builds, and sandbox environments on every push.

CI Automation does not require the Daml Autopilot MCP server. Together, though, they provide the best Daml development experience possible. Perfect for teams who want fast CI/CD without the steep learning curve.

Pricing

The Daml Autopilot MCP Server uses x402 micropayments in Canton Coin to cover operational costs (API calls, cloud infrastructure, blockchain transaction fees). You're paying for the infrastructure that processes your requests, not for code quality, accuracy, or reliability.

Fund your Canton Coin wallet, connect it when prompted, and you're ready to go. Each request costs a fraction of a cent, so even heavy usage is affordable.

The code analysis service is provided as-is without warranty. All generated code must be independently reviewed. See our Terms of Service for details.

Become a Daml power-user

Upgrade your Canton workflow. Instantly integrate Daml Autopilot to add a context-aware safety layer to your local environment. Accelerate your learning curve, automate your first round of safety checks, and start shipping faster.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development & operations, and co-development.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, and more, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

It was a concerning metric to know that when Flashbots publicized data on how node operators' client diversity changed throughout the year, we felt it was important to reiterate the importance of staking on multi-node combination setups or outright primarily running minority clients to protect the delegated Ethereum stake of your clients. It has always been a difficult question to answer for us at Lodestar, but if node operators were, in fact, reconfiguring their node infrastructure to select a new primary consensus client, why were they considering further centralizing their stake in majority clients, putting further undue risk on Ethereum?

In addition, throughout our time meeting with large node operators at Devconnect in 2025, it was noted that Lodestar (a TypeScript consensus client currently transitioning to the Zig programming language) was often misconstrued as being unsafe due to the nature of the ecosystem we currently support on mainnet. We want to reassure those who passively dismiss Lodestar as being an unsafe client to reconsider and allow us to demonstrate why our client's priority on safety surpasses the risk of switching to a majority client.

The JavaScript and TypeScript ecosystem has faced unprecedented supply chain attacks in recent years, raising legitimate concerns about the security of production systems built on NPM dependencies. Lodestar operates critical blockchain infrastructure securing millions of dollars in staked Ethereum today through ChainSafe. This makes the question of supply chain security not just important—it's existential. Yet, despite running on the Node.js runtime and depending on the node package manager ecosystem, Lodestar implements a comprehensive, multi-layered security strategy that makes it remarkably resilient against supply chain attacks.

This blog post explains why Lodestar is safe for node operators to use in production, even in the face of sophisticated NPM supply chain threats. We'll examine real-world attacks that have compromised the ecosystem, detail the specific protections Lodestar employs, and demonstrate how our security-first architecture provides robust defense-in-depth against malicious dependencies.

A History of NPM Supply Chain Attacks

Before diving into Lodestar's defenses, it's crucial to understand the threat landscape. The NPM ecosystem has experienced a dramatic escalation in both the frequency and sophistication of supply chain attacks over the past several years. Most recently, the "Shai-Hulud" Worm and a phishing campaign against the maintainer "qix".

Notable NPM Compromises: Learning from the Past

The event-stream Incident (2018): Perhaps the most infamous NPM supply chain attack began when the maintainer of event-stream, a popular package with 2 million weekly downloads, transferred ownership to a malicious actor who appeared to be a legitimate contributor. Over 2.5 months, this attacker added a dependency called flatmap-stream, containing obfuscated malicious code specifically targeting the Copay Bitcoin wallet. The attack remained undetected for months, resulting in approximately 8 million downloads of the compromised package before discovery.

This incident revealed a critical vulnerability: maintainer fatigue and social engineering could compromise even well-established packages. The malicious code was surgically targeted, demonstrating that attackers invest significant resources in understanding their targets.

ua-parser-js Compromise (October 2021): In October 2021, the widely used ua-parser-js package (7 million weekly downloads) was compromised when attackers hijacked the maintainer's NPM account. Three malicious versions were published, containing cryptocurrency miners and credential-stealing trojans. The attack was live for approximately 4 hours before detection, during which the compromised versions were downloaded thousands of times.

Forensic analysis revealed that threat actors had advertised access to a developer account for an NPM package matching ua-parser-js's profile on Russian hacking forums for $20,000 two weeks before the attack, suggesting a deliberate targeting of high-value packages.

The September 2025 Phishing Attacks: On September 8, 2025, the JavaScript ecosystem experienced one of its most significant supply chain attacks when sophisticated phishing campaigns compromised multiple maintainer accounts. Attackers registered the domain "NPMjs.help" and sent urgent security notifications claiming 2FA updates were required. Josh Junon, maintainer of critical JavaScript infrastructure packages, described receiving what appeared to be legitimate emails during a stressful morning—a perfect example of how social engineering exploits human factors.

The attack compromised at least 18 widely-used packages, including debug, chalk, and ansi-styles, collectively receiving over 2.6 billion weekly downloads. The malicious code was active for only about 2 hours before being removed. Still, it targeted cryptocurrency transactions using sophisticated address replacement algorithms that employed the Levenshtein distance to create similar-looking addresses.

Shai-Hulud: The Self-Replicating Worm (2025): Perhaps most concerning is the Shai-Hulud campaign, which represents an evolution in attack methodology. This self-replicating worm specifically harvests NPM publishing credentials to automatically spread itself to other packages. By targeting the pre-install phase, it executes before static scanning tools can analyze the code, and it requires zero human interaction beyond the initial npm install command.

The Shai-Hulud 2.0 campaign has compromised hundreds of packages, including the widely used @ctrl/tinycolor library, which receives millions of weekly downloads. The automated propagation mechanism means the scope continues to expand, making this an ongoing threat rather than a discrete incident.

Common Attack Patterns and Vulnerabilities

Analysis of these incidents reveals consistent attack vectors that NPM's default behavior enables:

1. Account compromise through phishing or credential theft: Attackers increasingly target maintainer accounts rather than exploiting code vulnerabilities
2. Abuse of postinstall scripts: Historically, over 90% of malicious NPM packages used postinstall or other lifecycle scripts to execute code immediately upon installation
3. Transitive dependencies: Malicious code often enters through dependencies-of-dependencies, which developers rarely audit
4. Speed as an attack enabler: The NPM ecosystem's default "always install latest" behavior means malicious packages propagate to production systems within minutes of publication
5. Detection windows: Most compromised packages are discovered and removed within 1-4 hours, but that's often enough time for widespread installation

Lodestar's Multi-Layered Defense Strategy

Understanding these threats, Lodestar has implemented a comprehensive security architecture that goes far beyond simply hoping our dependencies remain secure. Our approach creates multiple independent layers of defense, ensuring that even if one layer fails, others provide protection.

A TL;DR of some of these mitigations are explained in our new Lodestar Devlog.

Layer 1: pNPM Package Manager—Not Standard NPM

One of the most significant security decisions made recently in Lodestar's architecture is our use of pNPM rather than standard NPM for dependency management. This isn't just a preference—it's a fundamental security choice that provides numerous advantages against supply chain attacks.

Why pNPM Matters for Security: While NPM and yarn have improved their security features over time, pNPM was designed from the ground up with a security-conscious architecture. Its content-addressable storage model and strict dependency resolution provide inherent protections that other package managers lack.

Automatic Postinstall Script Blocking: Since pNPM v10, postinstall scripts are disabled by default in dependencies. This single feature eliminates approximately 90% of historical NPM supply chain attacks, which relied on automatically executing malicious code during installation.

When Lodestar does need to allow build scripts for trusted dependencies, we explicitly whitelist them using the allowBuilds configuration. This means if a previously safe dependency is compromised and suddenly adds a postinstall script, it won't execute, providing immediate protection against precisely the type of attack that compromised ua-parser-js.

Preventing Lockfile URL Injection Attacks: Before using pNPM, Lodestar would use lockfile-lint in our CI build pipeline to detect security issues on our yarn.lock file. Using pNPM architecture now prevents this from happening natively because pNPM does not store the download URL for registry packages in its lockfile by default. With the pnpm-lock.yaml file, there isn't a URL to point it to a malicious source.

Content-Addressable Storage with Integrity Verification: Unlike NPM's traditional node_modules structure, pNPM uses a content-addressable storage system where packages are stored once globally and hard-linked into projects. Each package is verified against SHA-512 integrity hashes in the lock file, making it virtually impossible for attackers to silently replace package contents.

If a malicious actor somehow gained access to a package and modified its contents without changing the version number (a sophisticated attack vector), pNPM's integrity verification would detect the mismatch and refuse installation.

Layer 2: The minimumReleaseAge Protection—A Time-Based Defense

In September 2025, pNPM v10.16 introduced a groundbreaking security feature that directly addresses the speed problem in NPM supply chain attacks: minimumReleaseAge.

This setting defines the minimum number of minutes that must pass after a package version is published before pNPM will install it. The Lodestar team now employs this feature to introduce a deliberate delay between publication and installation.

How minimumReleaseAge Creates a Protective Buffer: Looking at the attack timeline data, we see a clear pattern: the September 2025 attack was live for 2 hours, ua-parser-js for 4 hours, and even the most successful attacks are typically discovered within 24 hours. By setting minimumReleaseAge: 2880 (48 hours), we create a buffer during which:

1. Security researchers and automated scanning tools detect malicious code
2. The NPM security team removes compromised versions
3. The community raises alerts and creates security advisories
4. Lodestar's dependencies remain on known-safe versions

Practical Impact: This simple configuration would have completely prevented Lodestar from being affected by the September 2025 attack (2-hour window), the ua-parser-js compromise (4-hour window), and the majority of other supply chain attacks. As cybersecurity expert Brian Fox noted, even a 24-hour delay "dramatically lowers the chance you'll ever install a compromised package."

Flexibility for Urgent Security Patches: Critics might argue that delaying updates could prevent timely security patches. pNPM addresses this through minimumReleaseAgeExclude, which allows us to whitelist specific packages that should update immediately. For critical security dependencies or our own ChainSafe-maintained packages, we can bypass the delay while maintaining protection for the broader dependency tree. Lodestar currently does not whitelist any specific packages, and we've rarely found the need to urgently update dependencies, thanks to our conservative dependency tree and lock files, as explained below.

Layer 3: Lock Files—Immutable Dependency Resolution

Lock files are often misunderstood or underutilized, but they represent a critical security boundary in dependency management. Lodestar's pnpm-lock.yaml file serves as a cryptographic contract defining exactly which package versions are installed.

How Lock Files Prevent Supply Chain Attacks: When Lodestar's CI/CD pipeline builds the client, it uses pnpm install --frozen-lockfile, which:

1. Installs only the exact versions specified in the lock file
2. Verifies each package against its SHA-512 integrity hash
3. Refuses to proceed if any package hash doesn't match
4. Prevents "unexpected" updates that could introduce malicious code

This means even if an attacker compromises a dependency and publishes a malicious version, Lodestar won't install it unless a developer explicitly updates the lock file—providing a critical checkpoint for human review.

Lock Files as Audit Trails: Beyond security, lock files create a transparent audit trail. Every dependency change generates a diff in version control, allowing reviewers to see exactly what packages changed, when, and in which pull request. This visibility is essential for catching suspicious updates.

Protecting Against Weak Hash Algorithms: Older NPM lock files used SHA-1 hashing, which is vulnerable to collision attacks. pNPM's lock files use SHA-512, a cryptographically strong algorithm that makes it computationally infeasible for attackers to create a malicious package with the same hash as a legitimate one.

Layer 4: Lean Dependency Tree with Conservative Upgrades

One of Lodestar's most important, yet often overlooked, security features is our intentionally minimal dependency footprint and conservative upgrade philosophy.

Principle of Minimal Dependencies: Every dependency is a potential attack vector. Lodestar's monorepo architecture allows us to share code between packages without creating unnecessary external dependencies. By implementing core functionality ourselves rather than pulling in third-party packages, we dramatically reduce our attack surface.

This contrasts sharply with the "NPM install all the things" culture that has led to projects with thousands of transitive dependencies—each one a potential entry point for malicious code.

Conservative Update Strategy: While staying current with security patches is crucial, Lodestar doesn't chase every minor version bump. Our update philosophy prioritizes stability and thorough review:

1. Security updates are applied promptly after verification
2. Minor updates are batched and tested comprehensively
3. Major updates undergo extensive testing and code review
4. Breaking changes are carefully evaluated before adoption

This approach means Lodestar doesn't automatically pull in the "latest and greatest" version of every dependency the moment it's published—providing natural immunity against attacks that target newly-published versions.

Monorepo Advantages for Dependency Management: Lodestar's monorepo structure managed by pNPM workspaces provides additional security benefits:

• Centralized dependency versioning: All packages use the identical versions of shared dependencies, reducing the complexity of security auditing
• Workspace protocol: Internal package dependencies use workspace:*, ensuring they resolve to local versions and preventing dependency confusion attacks
• Single point of dependency review: Changes to dependencies affect the entire project, making them visible and subject to team review

Layer 5: Institutional Security Practices

Beyond tooling, Lodestar benefits from ChainSafe's institutional commitment to security best practices that extend across the entire development lifecycle.

Continuous Integration Security Checks: Every pull request to Lodestar triggers automated security scans:

• Dependency vulnerability scanning detects known CVEs
• License compliance checks ensure no malicious license changes
• Automated testing verifies no regression or unexpected behavior
• Code review by experienced protocol engineers on the team

Production Installation Guidance: Importantly, Lodestar's documentation actively discourages using npm install for production deployments. Instead, we recommend:

1. Docker containers: Pre-built, verified images from Docker Hub
2. Build from source: Clone the repository and build directly
3. Official binaries: Download signed releases from GitHub

This guidance acknowledges that, despite our defensive measures, eliminating all NPM supply chain risk requires controlling the entire build and distribution pipeline.

Two-Factor Authentication and Access Controls: All Lodestar developers with publishing access to NPM packages are required to use two-factor authentication. This directly addresses the attack vector that compromised the September 2025 packages. Additionally, all of our published packages must have reviewer approval, and no developer can accidentally or maliciously push code to main branches.

Trustless Publishing and Provenance: Lodestar packages are now published trustlessly to NPM through OpenID Connect (OIDC), as NPM has forced revocation on its classic long-lived access tokens.

Layer 6: Community-Driven Transparency and Client Diversity

Lodestar's open-source nature and role in Ethereum's client diversity strategy provide unique security advantages that proprietary software cannot match.

Open Source Transparency: Every line of code, every dependency, and every change to Lodestar is publicly visible on GitHub. This radical transparency means:

• Security researchers can audit our dependency choices
• The community can report suspicious changes
• Malicious modifications would be immediately visible
• Trust is built on verifiability, not blind faith

Client Diversity as a Security Feature: Lodestar is one of six production-ready Ethereum consensus clients, and the only one written in TypeScript. This diversity is itself a security feature: if a bug or vulnerability affects one client, the Ethereum network continues operating because validators using other clients remain functional.

The same principle applies to supply chain attacks. If an NPM supply chain attack specifically targeted Rust or Go dependencies (affecting other clients in different languages), Lodestar would be unaffected. Conversely, if a TypeScript-specific attack occurred, the network's other clients provide continuity.

Community Vigilance: The Ethereum community actively monitors client health and security. With node operators running Lodestar in production, from solo stakers to institutional operators, any anomalous behavior or suspected compromise would be quickly detected and reported.

Why TypeScript on Node.js is Actually an Advantage

Some might argue that using TypeScript on Node.js creates inherent supply chain risks compared to compiled languages like Rust or Go. However, this perspective overlooks several key advantages that TypeScript offers for blockchain development.

Type Safety for Critical Infrastructure: TypeScript's static type system catches entire classes of bugs at compile time that would become runtime errors in JavaScript. For blockchain infrastructure handling real financial value, this additional verification layer is crucial.

Smart contracts and consensus clients operate in an environment where bugs can be catastrophic, resulting in the loss of funds, breaking consensus, or enabling exploits. TypeScript's type-checking provides an additional safety net that reduces these risks.

Accessible Security Auditing: TypeScript code is significantly more readable and auditable than compiled languages. Security researchers, external auditors, and community contributors can more easily understand and verify Lodestar's implementation, improving the quality of security reviews.

Ecosystem Maturity: The size of the TypeScript/JavaScript ecosystem means that security tools, scanners, and monitoring solutions are extremely mature. Projects like Snyk, Socket Security, and GitHub's Dependabot provide sophisticated supply chain security specifically for NPM packages.

Developer Accessibility: TypeScript is one of GitHub's most popular languages, making it accessible to a broad developer community. This accessibility means more eyes on the code, more potential contributors for security improvements, and a larger pool of developers who can audit and verify Lodestar's security.

Practical Recommendations for Node Operators

For node operators considering Lodestar, here are practical steps to maximize security:

1. Use Docker, Signed Binaries or Build from Source: For production deployments, use ChainSafe's official Docker images, run GitHub-signed binaries or build Lodestar from source rather than using npm install.
2. Enable Dependency Scanning: Use tools like pnpm audit or GitHub Dependabot to monitor for known vulnerabilities in dependencies.
3. Review Lock File Changes: When updating Lodestar, carefully review changes to pnpm-lock.yaml to understand what dependencies changed.
4. Follow Our Channels: Follow Lodestar's X account and GitHub security advisories for Lodestar.
5. Contribute to Client Diversity: By choosing Lodestar alongside other clients, you strengthen the entire Ethereum network's resilience against client-specific bugs and attacks.

Why TypeScript/Lodestar Isn't Safe is Irrationally Justified

Supply chain security encompasses a spectrum of risk management strategies, rather than being a binary proposition of safe or unsafe. While no system can claim absolute immunity from sophisticated attacks, Lodestar's multi-layered defense architecture demonstrates that a TypeScript/Node.js infrastructure can achieve security postures comparable to or exceeding those of other compiled language alternatives.

By combining pNPM's inherent security advantages with time-based protections through minimumReleaseAge, cryptographic integrity verification via lock files, minimal dependencies, conservative upgrade practices, and the transparency of open-source development, Lodestar has constructed a robust defense-in-depth strategy against NPM supply chain attacks.

The Ethereum network's security, and the billions of dollars it secures, depends on the resilience of its consensus clients. Lodestar's security-focused architecture, institutional practices, and community-driven transparency make it a safe, reliable choice for node operators seeking to participate in Ethereum's consensus while contributing to critical client diversity. If you don't believe us, you can check our statistics on Rated.Network as we continue to stake thousands of Ether as a Lido Node Operator with Lodestar architecture.

The threat of supply chain attacks is real and evolving, but so are our defenses. Through continued vigilance, conservative dependency management, and leveraging the security features of modern tooling, like pNPM, Lodestar demonstrates that a TypeScript consensus client can operate at the highest levels of security, performance and reliability demanded by critical blockchain infrastructure.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development & operations, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, and more, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

We’re excited to share that ChainSafe is rolling out Orbitor, a new gateway for IPFS Mainnet, the IPFS public network.

Gateways play a key role in making IPFS accessible. They translate HTTP requests to IPFS, fetch content from the IPFS public network, and return that content back via HTTP. This allows anyone access to the decentralized public IPFS network directly through a standard web browser, with no separate installation required.

Regional endpoints:

• LATAM: https://latam.orbitor.dev
• EU: https://eu.orbitor.dev
• APAC: https://apac.orbitor.dev

Global endpoint:

https://ipfs.orbitor.dev/ipfs/[object CID here]

ChainSafe Infrastructure

Through our continued work in open decentralized infrastructure, such as building and maintaining client implementations including Forest (Filecoin), Lodestar (Ethereum), and Gossamer (Polkadot) while also operating production-grade Filecoin bootstrap nodes, Ethereum validators, and infrastructure spanning multiple decentralized networks, we’ve seen firsthand how geographic diversity significantly enhances both performance and resilience in public systems.

We’re excited to support the IPFS network in its goal of accessible, decentralized content distribution by operating a new IPFS public gateway designed to deliver region-aware, low-latency access to IPFS.

The Orbitor Gateway

Orbitor will operate across three regional endpoints: South America, Europe, and Asia-Pacific. 

These endpoints are backed by highly available IPFS nodes and monitored for reliability, speed, and abuse protection. This approach reduces latency by serving content closer to users and increases robustness against regional network disruptions or DDoS events.

With fully integrated monitoring, scheduled GC-aware caching on upstream Kubo nodes, and abuse mitigation coordination with trusted ecosystem partners, the gateway is built to be a reliable and transparent participant in the broader IPFS network.

If we want IPFS to be accessible to everyday users, gateways can’t be bottlenecks. They need to be open on-ramps built for scale, resilience, and public good.

Learn more about Orbitor
Explore our Observability Dashboards

Gateways as a Transition Tool

While gateways serve as an important bridge today, they're designed as a transition tool. IPFS is actively investing in browser-native approaches that eliminate the need for gateways or additional installs. With the recent launch of Helia's service worker support, developers can now bundle IPFS directly into web applications, enabling peer-to-peer content access from any standard browser.

In the coming year, ChainSafe will be collaborating with the IPFS Foundation and community to determine the future role of gateways as these browser-native approaches mature.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development & operations, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, and more, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

As the Fusaka hard fork nears deployment on mainnet, the Lodestar team continues to push towards their Zig transition alongside participating in scoping the Glamsterdam hard fork. We made it clear that Glamsterdam was an important upgrade in optimizing Ethereum's slot restructuring via EIP-7732 and the focus now shifts towards additional EIPs for inclusion.

TL;DR

• EIP-7805: We strongly believe that censorship resistance needs to be included with Glamsterdam. We believe delaying Glamsterdam and/or simplifying ePBS so we can include a censorship resistant mechanism with ePBS is important for the protocol. We currently estimate an additional engineering effort of ~2 weeks for implementation of rebasing FOCIL on top of ePBS for our client only
• EIP-7688: We support inclusion of compatible consensus data structures for reducing maintenance churn for important staking ecosystems relying on beacon data in smart contracts
• EIP-8045: We support inclusion of excluding slashed validators from proposing for chain health in non-finality situations, as demonstrated by the Holesky rescue
• EIP-8071: is overall a low priority as it doesn't create a security issue for the chain, even though it is a design flaw in how consolidations are used. It is however, a simple fix to an unintended consequence if we choose to include it
• We overall do not support the following maintenance EIPs and would prefer to see a more thorough Meta-EIP to fix flaws in Heka:
  • EIP-8061: We do not support increasing churn limits until more research has been completed on tradeoffs and more consensus is formed by other parts of the community affected by changing staker dynamics. We believe in a security-first approach over staking UX and the better question to primarily ask is "what is a safe weak subjectivity period?" However, we would be open to having a separate churn limit for consolidation because its purpose is different from entry/exit queues
  • EIP-8062: This is overall a low priority and has bad optics on burning/charging fees on the consensus layer, which has no precedence. In addition, there is no clear data to show that this will be impactful for 0x02 migration and consolidations are potentially lagging in adoption due to downstream governance delays from staking entities
  • EIP-8068: We do not support neutral effective balance design because the proposed thresolds feel overly complex, the severity of the issue is unclear without data on its gamification and we would much prefer a broader plan for how to deal with consolidations rather than band-aid solutions such as this

The importance of censorship resistance: Why FOCIL matters now.

Understanding Censorship Resistance on Ethereum

Censorship resistance ensures Ethereum remains open and permissionless. This blog explores what censorship resistance is, why it matters, and the ongoing efforts to keep Ethereum decentralized.

ChainSafePhil Ngo

Ethereum has a unique property unlike any other Layer 1 blockchain. The community and the technological structure of Ethereum's protocol are meant to ensure we preserve its permissionlessness and decentralization ethos.

Block production is an important function of the blockchain but is naturally, a centralizing mechanism that is ripe for capture. Whether it's the lack of diverse builders, relays or stakers, it is clear that a "winner take most/all" is inherently incentivized within the protocol for maximum rewards. Off-protocol block production such as MEV-Boost generally capture a consistent ~90% of blocks on mainnet because of its larger rewards for stakers, which also birthed the concept of smoothing pools for sharing these rewards. This also means that builders who can build the most valuable blocks, generally squeeze out the competition and capture the ability to dictate what makes it into a block and what doesn't, hence, censorship.

As we work towards enshrined proposer-builder separation (ePBS) via EIP-7732, we confirm what we've tried so hard to avoid: Admitting that local block building is not viable long-term on Ethereum. In the early designs of Ethereum's proof of stake system, we hoped that block production and block validation would be run by a diverse group of individuals alongside professional node operators. That diversification is what gives Ethereum its edge in remaining the world's "hardest" (irrevocable and irreversable) global settlement layer for digital assets. Over time, fewer solo node operators have emerged than originally hoped for. Secondly, we need to counteract builder centralization urgently.

This is why we feel it is important for us to include EIP-7805: Fork-Choice enforced Inclusion Lists (FOCIL) as part of Glamsterdam. Delaying this EIP induces uncertainty with its inclusion at all as we keep having to rebase it on other changes such as six second slots (anticipated headliner of the Heka hard fork). This is a huge risk and the engineering effort is worth the additional time required to ship Glamsterdam. We currently estimate the additional time at ~two weeks based on the specification details for implementation on top of ePBS.

In addition, the benefits of FOCIL accrue to other layers of the protocol such as Layer 2s which would allow for optimistic rollups to reduce their challenge window. This creates another desirable effect, making finality faster on Layer 2s where a lot user-based activities live.

We generally believe that FOCIL implementation is worth an additional delay in shipping Glamsterdam.

Other EIPs for Inclusion Consideration

EIP-7688: Forward compatible consensus data structures

We are in favor of including EIP-7688, which has now been rebased on top of Gloas. This EIP introduces generalized indices across forks, reducing the maintenance churn (involving security councils) for builders that consume beacon data in their smart contracts, and is mostly benefiting staking pools (e.g., Rocket Pool, Diva, Lido) and our ethos of trustlessness.

EIP 8045: Exclude slashed validators from proposing

We are in favor of including EIP-8045. This is a straight forward change that will improve chain liveness should there be unforeseen events such as a mass slashing. We want to maximize including any EIPs which will positively impact chain performance, especially during periods of instability. Every block proposed on an unstable chain is valuable and we want to minimize opportunities for missed slots because a validator scheduled to propose has already been slashed.

EIP 8071: Prevent using consolidations as withdrawals

We are weakly in favor of including EIP-8071. This simple fix in the protocol is an oversight and the unintended consequence is that the consolidation queue is being used for a purpose it was not designed for to circumvent the withdrawal queue. Although, we believe that it makes sense for us to correct this, this is not a priority as it does not pose a threat to security.

Other EIPs for Exclusion or Deferral

EIP 8061: Increase churn limits

We believe that this EIP requires more research to identify additional unintentional or predictable consequences downstream to the protocol before inclusion. This change introduces tradeoffs in chain security for staking experience with no additional benefit to Ethereum ethos, such as decentralization. Entering or exiting the protocol should be constrained by the volume of those looking to enter or exit the validator set for the preservation of chain security. Native staking should always been seen as a long-term commitment and not a predictable mechanism to make capital as efficient as possible. There are offchain markets (e.g. liquid staking tokens, CEXs) which serve and also take the risk of offering faster churn whilst the protocol itself remains as secure as possible.

This EIP also prematurely sets parameters that require additional time to investigate. We need to thoroughly understand all/most of the risks of reducing the weak subjectivity period and finding a minimum period required which balances staking experience and security of the chain. Then, we can derive optimal limit sizes for queues. We need to answer, "what is the right duration for the weak subjectivity period" first.

In addition, this EIP will also affect the incentives related to staking issuance and rewards which require additional scrutiny from its changing staking dynamics. We currently do not have any data, feedback and opinions from other parts of the community to justify the large scope of this change. It is not a purely technical decision to change these queues.

A research discussion was also raised about potentially investigating a priority-fee mechanism for faster withdrawals. There is currently no proposed design or research to cite. However, there may be future options in the near future to debate how to solve for faster queues without sacrificing chain security.

EIP 8062: Add sweep withdrawal fee for 0x01 validators

We are not in favor of including EIP-8062. Although we believe that migrating to a smaller active validator set is important for Ethereum's roadmap, we do not think there is definitive proof that this EIP would force migration to 0x02 and this would create precedence for adding fees on the consensus layer for regular operations, which is optically negative.

EIP 8068: Neutral effective balance design

We are not in favor of including EIP-8068. The gamification identified by this EIP needs to be supported with more data that this is occurring. The proposed thresolds feel overly complex and without the data, the severity of this issue is unknown. We would much prefer to see a plan to overhaul the consolidation process with a simpler design, which includes addressing issues such as this for Heka.

The problem

My Chrome tab is using so much memory!

You might be complaining about your Chrome tab using 2 GiB of memory, but have you ever tried running a Filecoin node? The minimum for mainnet starts at 8 GiB, and the more features you want to use, the more memory you will need. Some examples:

• Network upgrade for NV22 Smaug required at minimum 64 GiB; both Forest and Lotus. For the curious, it was due to a relatively complex state tree migration.
• Snapshot export in Forest requires ~16 GiB of memory. In Go implementations, it is significantly more. I don’t have the exact numbers because I don’t have enough RAM to test it. Sufficient to say, it’s substantially more than 64 GiB.
• Some niche JSON-RPC calls require a lot of memory, like Filecoin.StateMarketDeals - given the API is not paginated, it will try to load all deals into memory, which can easily go over 64 GiB. There are some efforts to make the API more usable, see this experimental API PR

With all this in mind, it is essential to account for every single gigabyte of memory the node uses. After all, it’s one of the selling points of Forest: run a Filecoin node with fewer resources. Hell, run it on a Raspberry Pi! This is where memory analysis comes in handy.

But Rust prevents memory leaks, right?

Yes, Rust prevents memory leaks in the sense that it will not allow you to allocate memory and then forget about it, unless you use Box::leak, std::mem::forget or depend on faulty crates. For the latter, you might want to use cargo-geiger. However, Rust will not prevent you from creating a data structure and keep appending to it indefinitely, eventually leading to a beautiful OOM. A typical case in a long-running application that performs non-trivial operations is that you have a cache and keep adding to it. If it’s unbounded, you will have an OOM, sooner or later.

Setup

You need to compile your program with debug symbols to get the most out of every tool. It makes sense to create a custom profile for this, e.g., profiling, in your Cargo.toml:

[profile.profiling]
inherits = "dev"
opt-level = 1
debug = true

This helps if your custom profile isn’t exactly the same as dev. For example, you need some optimisations so that the program runs fast enough to be useful, but you still want debug symbols.

Another thing to keep in mind is that some tools might work better with the system allocator rather than Rust’s default or your custom one, like jemalloc (the actual default on some platforms) or mimalloc. You can introduce a feature flag for this, e.g., system-allocator, and then use it in the code:

cfg_if::cfg_if! {
    if #[cfg(feature = "rustalloc")] {
      // uses the default Rust allocator
    } else if #[cfg(feature = "jemalloc")] {
        use tikv_jemallocator::Jemalloc;
        #[global_allocator]
        static GLOBAL: Jemalloc = Jemalloc;
    } else if #[cfg(feature = "system-alloc")] {
        use std::alloc::System;
        #[global_allocator]
        static GLOBAL: System = System;
    } else {
        compile_error!("You must enable one of the following features: `rustalloc`, `jemalloc`, or `system-alloc`");
    }
}

And in your Cargo.toml

[features]
...
system-alloc = []

Compile your program with the profiling profile and the system allocator feature:

cargo build --profile profiling --features system-alloc # you might need to add `--no-default-features`, depending on your setup

heaptrack

heaptrack - heaptrack-1.5.0

It’s the simplest tool to use. It works by running your program under heaptrack, which will record all memory allocations and deallocations, and then you can analyse the results with its interactive viewer. It also supports generating a flamegraph. It used to be my go-to tool for memory analysis, but it has a very high memory overhead, so it might not be suitable if your program is already memory-intensive, like a Filecoin node. In my attempts, it mostly crashed with an OOM error, even with a 64 GiB RAM machine. Sometimes my Fedora would hang, and I had to reboot it: instant +10 to burnout.

You can run it with heaptrack <program>; once closed, it will generate a file like heaptrack.<pid>.gz. You can then analyse it with heaptrack <file>. The analysis might take quite a while, so be patient. The output will look something like this:

Theoretically, you can attach to a running process with heaptrack --pid <pid>, but there’s an explicit warning that it might open a Pandora’s box of issues, so I wouldn’t recommend it.

When it works, it’s a great tool and my go-to for memory analysis. Unfortunately, it has some issues with memory overhead, so I had to look for alternatives.

gperftools & pprof

gperftools - gperftools-2.16

I’ve had quite a good experience with this tool. The setup is minimal, and there’s hardly any overhead. Every once in a while, it dumps the heap profile to a 1 MiB file, which can be inspected (without interrupting the program) with pprof.

Dumping heap profile to /tmp/gperfheap.forest.prof_10047.0167.heap (171024 MB allocated cumulatively, 184 MB currently in use)

You need to link your program with tcmalloc to set it up. You can do it, e.g., via a build script:

fn main() {
    println!("cargo:rustc-link-lib=tcmalloc");
}

Now, in your Makefile, you can add targets to build and run your program:

gperfheapprofile = cargo build --no-default-features --features system-alloc --profile=profiling --bin $(1); \
	ulimit -n 8192; \
	HEAPPROFILE_USE_PID=t HEAPPROFILE=/tmp/gperfheap.$(1).prof target/profiling/$(1) $(2)

gperfheapprofile.forest:
	$(call gperfheapprofile,forest, --chain calibnet --encrypt-keystore=false)

Run it with make gperfheapprofile.forest to start the program. It will dump the heap profile periodically, and you can inspect the dumps at any time. Let it run for a while and put some typical load on it to accumulate enough data to analyse.

Now, analyse the dump using pprof. I prefer the web interface, but you might tinker with other options.

# Note that you need the binary with debug symbols, so use the profiling profile mentioned earlier.
pprof -http=localhost:8080 target/profiling/forest gperfheap.forest.prof_10047.0167.heap

This will start a web server on localhost:8080 where you can interactively inspect the memory usage of your program.

Here, you can see that quite a bit of memory is used from the load_required_tipset (and a bunch of others - one needs to follow the call stack with the code to understand what is going on and mentally filter all tokio tasks). This makes sense, because there’s a cache:

    /// Loads a tipset from memory given the tipset keys and cache. Semantically
    /// identical to [`Tipset::load`] but the result is cached.
    pub fn load_tipset(&self, tsk: &TipsetKey) -> Result<Option<Arc<Tipset>>, Error> {
        if !is_env_truthy("FOREST_TIPSET_CACHE_DISABLED")
            && let Some(ts) = self.ts_cache.lock().get(tsk)
        {
            metrics::LRU_CACHE_HIT
                .get_or_create(&metrics::values::TIPSET)
                .inc();
            return Ok(Some(ts.clone()));
        }

        let ts_opt = Tipset::load(&self.db, tsk)?.map(Arc::new);
        if let Some(ts) = &ts_opt {
            self.ts_cache.lock().put(tsk.clone(), ts.clone());
            metrics::LRU_CACHE_MISS
                .get_or_create(&metrics::values::TIPSET)
                .inc();
        }

        Ok(ts_opt)
    }

An environmental variable can turn off the cache at a cost of performance. In this case, it’s discouraged as the performance impact is so great that the node might not be able to keep up with the network.

dhat crate

dhat - dhat-0.3.3

This requires a different setup than what was mentioned at the beginning. All information is in the dhat documentation. The gist is that you must add dhat to your dependencies, set it up as a global allocator, and start it at the beginning of your program. Note that the dhat allocator is slower than the standard allocator, so it should be used only for profiling.

It might be that the slowdown is not acceptable for your use case, or you want to profile a specific part of the code, in which case you might want to use ad hoc profiling. This allows you to annotate the hot spots in your code with dhat::ad_hoc_event.

The output is a bit old-school, similar to valgrind output, but it can do the job. See the DH manual.

A unique feature of the crate is that you can write tests with it to check the amount of memory allocations. For example, you can write a test that asserts that the code doesn’t allocate more than 10 MiB of memory:

 #[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;

#[test]
fn test() {
    let _profiler = dhat::Profiler::builder().testing().build();

    some_function_that_should_not_allocate_more_than_10_mib();

    let stats = dhat::HeapStats::get();

    // At the point of peak heap size, no more than 10 MiB should've been allocated.
    dhat::assert!(stats.max_bytes <= 10 * 1024 * 1024, "Memory usage exceeded 10 MiB");
}

In case of failure:

---- test stdout ----
dhat: Total:     11,534,336 bytes in 1 blocks
dhat: At t-gmax: 11,534,336 bytes in 1 blocks
dhat: At t-end:  0 bytes in 0 blocks
dhat: The data has been saved to dhat-heap.json, and is viewable with dhat/dh_view.html

thread 'test' panicked at src/main.rs:20:5:
dhat: assertion failed: stats.max_bytes <= 10 * 1024 * 1024: Memory usage exceeded 10 MiB

This is a great way to ensure that your code doesn’t accidentally allocate too much memory, and it can be used in CI to catch regressions. Even better, you won’t have to read posts like this one!

valgrind - massif

valgrind - valgrind-3.25.1

It’s more of an honourable mention. I failed to get it working with Forest; it’d just hang indefinitely. I decided not to spend more time on it. You might have better luck with it. Add this to your Makefile:

memprofile-massif = cargo build --no-default-features --features system-alloc --profile=profiling --bin $(1); \
                   ulimit -n 8192; \
                   rm massif.out.*; \
                   valgrind --tool=massif target/profiling/$(1) $(2); \
                   ms_print massif.out.* > /tmp/massif.$(1).txt

memprofile-massif.mybin:
  $(call memprofile-massif,mybin)

Then run it with make memprofile-massif.mybin and check the output in /tmp/massif.mybin.txt. Sample output is lengthy, but you can check it here. The entire allocation is done in the allocate_memory function:

use std::alloc::System;
#[global_allocator]
static GLOBAL: System = System;

fn allocate_memory(size: usize) -> Vec<u8> {
    vec![0; size * 1024 * 1024] // Allocate `size` MB of memory
}

// massif should report 708 MiB allocated in total
fn main() {
    let _v1 = allocate_memory(42); // 42 MiB
    let _v2 = allocate_memory(666); // 666 MiB
}

Massif output:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 16        352,413      742,399,888      742,391,808         8,080            0
 17        352,594      742,399,888      742,391,808         8,080            0
100.00% (742,391,808B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->100.00% (742,391,808B) 0x40020B4: std::sys::alloc::unix::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::alloc_zeroed (unix.rs:36)
| ->100.00% (742,391,808B) 0x4002611: __rustc::__rust_alloc_zeroed (main.rs:3)
|   ->100.00% (742,391,808B) 0x4030786: alloc::raw_vec::RawVecInner<A>::try_allocate_in (in /tmp/tst/target/profiling/tst)
|     ->100.00% (742,391,808B) 0x4001E64: <u8 as alloc::vec::spec_from_elem::SpecFromElem>::from_elem (mod.rs:447)
|       ->100.00% (742,391,808B) 0x400273E: alloc::vec::from_elem (mod.rs:3220)
|         ->100.00% (742,391,808B) 0x4002493: tst::allocate_memory (main.rs:6)

Quick sanity check:

The math checks out. Oof!

You can also use the massif-visualizer. It’s not as feature-rich as heaptrack, but it can be useful for quick analysis.

Other tools

Other than the tools mentioned above, I got some potential recommendations from the community, but I haven’t had a chance to try them (yet):

• bytehound - some legwork needed to set it up, as it requires a specific, old version of mimalloc library on the host for compilation. It looks promising, though it hasn’t received any updates since 2023. According to the author, it is finished.
• Tracy Profiler + Tracing-Tracy - supposedly great combo for profiling asynchronous tokio code annotated with tracing spans.
• eBPF - this was recommended several times on the Reddit post as the modern way, with tooling like OTEL eBPF, Aya, bpftrace (easy but limited) and bcc (powerful but more painful). I haven’t tried it, but it looks like a powerful toolbox for profiling. The fact that there’s even a ePBF book about it makes it pretty daunting!

Hic Sunt Dracones

The profilers will not report everything. For example, memory-mapped files will not be reported. This is annoying because these tools might tell you that your program uses 200 MiB of memory, but then you check the memory usage with htop, and it shows over 2 GiB.

You can use smaps to check the mappings of your process. With a simple smaps Ruby wrapper, you can see what is happening:

hubert@fuzzy:~/prj$ sudo ruby cached_mappings.rb 10047 10

Top 10 file-backed memory mappings (by cached memory):
 Memory (MB)    Rss (MB)  Mapping Path
------------------------------------------------------------
       237.6       237.6  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_9a
       148.5       148.5  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_f3
       141.8       141.8  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_ed
       129.1       129.1  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_fc
        93.2        93.2  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_ab
        64.0       156.0  /home/hubert/no-kad-test/mainnet/0.27.0/index_00_17
        58.7        58.7  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_99
        57.8       660.0  /home/hubert/prj/forest/target/profiling/forest (deleted)
        57.3        57.3  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_56
        42.9        42.9  /home/hubert/no-kad-test/mainnet/0.27.0/table_00_ff
------------------------------------------------------------
      1030.8      1725.0  TOTAL (Top 10)

At least in Forest’s case, it’s not a huge deal because the memory-mapped files are used for caching and are not leaked. The system will reclaim the memory when needed. It is still essential to be aware of this, though, as it can lead to confusion (yes, speaking from experience) when analysing memory usage.

Avoiding memory issues

Memory analysis aside, having all persistent collections bound is a good idea so that you could skip the analysis altogether. Some suggestions based on the memory analysis done in Forest:

• Use an lru or its variants for caching. Don’t implement them yourself; use battle-tested libraries like lru or cached.
• Monitor the size and length of your caches, e.g., via Prometheus metrics. Calculating the total size of a data structure can be tricky, but some libraries can help, such as get-size2. This will require some manual annotation, but it is worth it.
• Don’t use unbounded channels, like flume::unbounded, unless you are confident they won’t grow indefinitely. Ban their usage in your codebase with clippy’s disallowed-methods.
• Use reference-counted types like Arc or Rc to limit the number of copies of the data you have in memory.
• Consider using pointers, e.g., Box, for keys and values in your data structures, especially if the data types are large. This will reduce the memory taken via calls like with_capacity if the entire capacity is not always filled.
• Consider using linked lists for more efficient memory usage. You can use hashlink for an LRU cache that uses linked lists internally. Note, this might save memory at the cost of performance, so you need to benchmark it for your use case.

All these suggestions won’t prevent all possible memory issues (see this one), but they will help you avoid the memory analysis legwork.

Conclusion

Memory analysis in Rust is not as straightforward as in some other languages, e.g., JVM-based. Still, plenty of tools and techniques can help you understand and optimise memory usage in your Rust programs. Once you’re done with the analysis, you will significantly reduce your program’s memory usage, leading to better performance and lower resource consumption. Who knows, maybe you’ll even convince your boss to rewrite their bloated Java application in Rust! 🦀

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

Sequencers sit at the heart of almost every Layer 2 and blockchain rollup. They collect, order, and submit transactions from the L2 mempool, decide whether to execute or discard them, and broadcast the information to other nodes.¹ The transaction information then gets batched together, along with proofs, and sent to the base Layer 1 the L2 lives on. The sequencer is what effectively rolls up the transactions to support scaling blockchain networks. 

As is the case with every novel technology, though, sequencers have their own growing pains. Issues like bugs stalling transactions, censorship eroding user trust, and limited ability to scale slow them down. But sequencers are still an essential part of almost every Layer 2. 

So the question becomes: How do we design sequencers in order to minimize the potential setbacks? 

To answer that, we need to look at what might go wrong.  

Some of the risks we will look at have already appeared in the industry and are not limited to any single design. But this exploration isn't about pointing fingers. Instead, it's about stepping back to better understand the unique ways that sequencing architectures can fail in order to design new solutions. Ideally, in a way that lets developers choose how to manage and mitigate the risks most relevant to their needs. 

Risks

Sequencers are vulnerable to risks ranging from mild interruptions to existential threats from bugs. Below, we will cover a few of the ones we have noticed already popping up in many sequencer models, along with a few others that scare us the most. 

Fallback mechanisms

Every rollup design accounts for the possibility of sequencer outages. Fallback mechanisms, or “escape hatches,” are built to ensure users can still include transactions if the sequencer fails. On paper, these mechanisms look simple. In practice, they often fail when needed most.

For example, a rollup may offer a Layer 1 fallback that forcibly includes user transactions during a sequencer failure. In some designs, this only activates after a 24-hour delay or depends entirely on a small set of governance-approved proposers. In a crisis, transactions stall, markets move, and users cannot act until it is too late.

In centralized systems, governance teams may hold exclusive control and can be slow to respond or unreachable during emergencies. In decentralized systems, fallback access may be restricted by protocol keys, high costs, or strict timelocks. In both cases, users face the same problem: no way to act when immediate access matters most.

A slow or restricted fallback fails its purpose. These designs need to be built for urgent, reliable use, not as token safety measures.

Sequencer stalls

Moderate sequencing risks often start with small mistakes but have severe consequences. These risks include stalls caused by software bugs, depleted gas fee balances, or basic operational errors.

In one example, a rollup sequencer batches and posts transactions to Ethereum’s Layer 1 using a single wallet to pay gas fees. If the balance runs out, either from oversight or a sudden spike in gas prices, the sequencer stops, and transaction batches freeze indefinitely.

In centralized systems, this often comes down to manual wallet management and human error. Hours-long outages can occur if the operations team fails to top up the wallet on time. Automation and clear separation of duties can prevent this, but many monolithic designs do not include these safeguards.

Decentralized systems face the same risk in a different form. Multiple validators may share the responsibility of posting batches, but without clear incentives and coordination, no one may step in during a fee spike. The result is the same: stalled transaction processing.

These stalls delay transactions, freeze assets, and erode user trust. Some rollups reduce the impact with fallback logic. In the OP Stack, if a sequencer fails to submit batches within a fixed window, the system inserts an empty Layer 1 block. This triggers a one-block reorg that removes the stalled state and allows the chain to continue. The tradeoff is a brief rollback instead of a complete freeze.

Redundancy, automated economic management, and separation of operational duties turn catastrophic stalls into manageable incidents. Fallback mechanisms like OP Stack’s can limit downtime to a minor disruption.

Transaction censorship

A rollup sequencer with unilateral transaction-ordering power can censor activity without oversight or transparency. It can delay certain transactions indefinitely or discard them entirely. Censorship can result from compliance requirements, operator bias, or internal policy. Users may mistake repeated failures for normal congestion, unaware that the exclusions are deliberate.

In centralized systems, censorship can be explicit or implicit. Users must appeal directly to the operators, who may be slow to respond or unwilling to explain decisions.

Decentralized systems face similar risks. While explicit collusion is generally harder because it requires coordinating a threshold of participants, implicit censorship can still arise from shared incentives or operator overlap and is often more difficult to detect. Without built-in routing alternatives, users have few ways to bypass a censoring sequencer. Whether censorship is more or less likely depends on validator diversity and concentration, MEV dynamics, and whether users have alternate submission routes or forced-inclusion guarantees.

Censorship often appears as delays, missing transactions, or silence. Transparent inclusion rules and proactive monitoring are essential to detect and prevent it.

Emergency halts after hacks

A rollup sequencer suffers a severe security breach, compromising user funds. Operators respond with an emergency halt and block transactions from exploit-linked wallets. This can limit further damage, but also concentrates decision-making power.

In centralized systems, operators have direct control and can act quickly. The trade-off is reduced user autonomy and transparency, with users relying entirely on the operator’s judgment.

In decentralized systems, control is distributed, but governance processes can be slow, divided, or influenced by conflicting incentives. Coordination challenges can delay action and create inconsistent responses, making recovery harder.

In any model, a crisis places operator judgment under intense scrutiny. Users have little ability to bypass such halts. Clearly defined and transparent emergency governance policies are essential.

Silent bugs and accidental exclusion

A subtle software bug prevents certain users or addresses, potentially millions, from submitting transactions. The sequencer appears operational, and transactions seem normal, but a large group of users is quietly locked out.

Centralized sequencers are especially vulnerable to these silent failures because a single point of control magnifies the impact. Users often remain unaware until patterns of exclusion emerge, by which time trust may be irreparably damaged.

Decentralized designs can sometimes catch such bugs earlier through redundancy and validator oversight, but they are not immune. Added complexity creates more points of failure, communication challenges, and greater opacity, which can make silent exclusions harder to detect.

Operator trust depends on competence, oversight, and transparency. Users who cannot bypass a faulty sequencer remain exposed. Rigorous testing, clear monitoring, and transparent processes are essential.

Latency sinkholes

All rollups, whether centralized or decentralized, face latency tradeoffs. Users expect near-instant finality, but delivering it under load is challenging.

A single centralized sequencer node can handle transactions efficiently in normal conditions. However, when network activity spikes, from market volatility or a popular NFT mint, it can become a bottleneck. This causes delays, stalled transactions, and degraded performance because the sequencer cannot scale quickly enough.

Decentralized sequencing distributes transaction ordering across multiple nodes to reduce bottlenecks. Under heavy load, validator coordination, ordering agreements, and prover consensus add latency and communication overhead. In modular or highly composable decentralized designs, the complexity can grow further, leading to “cross-rollup clashes” where rollups compete for shared resources and slow each other down.

Every architecture trades between speed, cost, simplicity, fairness, and scalability. Defining and prioritizing these tradeoffs early is essential for delivering consistent, predictable performance.

Consensus stalls

Decentralization can increase resilience, but it also adds complexity that creates coordination risks.

In a consensus-based sequencing design, using protocols like Byzantine Fault Tolerance, Proof-of-Stake, or DAG-based models, multiple validators share responsibility for ordering transactions. This approach removes single points of failure but introduces new risks. Validators must maintain accurate, timely communication, which becomes difficult during network partitions, software bugs, or validator churn.

When validators disagree on the network state, they trigger view-change events or halts while reconciling differences. Even small issues, like latency, software incompatibility, or an overloaded network, can lead to prolonged stalls and delayed consensus. Transactions remain in limbo, frustrating users.

Centralized sequencing does not face this type of coordination failure because a single node makes all ordering decisions, but it is more exposed to downtime from outages or attacks, and users must trust the operator not to censor or delay transactions.

Decentralization improves trust minimization and resilience, but it increases complexity and potential downtime. Builders must choose the balance of uptime and trust minimization that best fits their network’s priorities.

Sequencer risks are a design choice

Sequencer failures have already occurred and, in some cases, caused major disruptions. They are directly linked to design decisions made early in a rollup’s architecture.

Centralized systems tend to fail in visible ways. A single node under stress becomes a bottleneck, and the system stops. The source of the problem is clear and easier to address. Decentralized systems can fail more quietly, often due to coordination issues or fragility at the network’s edges. These vulnerabilities are harder to detect and can cause widespread disruption before they are noticed.

Many rollups still adopt a default sequencer stack without evaluating whether it fits their needs. Sequencing has no universal blueprint. Each design carries different risks, and the right choice depends on the specific trade-offs a network is willing to accept.

Toward modular sequencing

Sequencing should be a framework, not a fixed black box. Developers should be able to configure their own:

• Fallback design for resilience during downtime.
• Ordering logic tailored to their transaction types and use cases.
• Economic guarantees that align incentives and protect network health.
• Decentralization thresholds that define how much trust and coordination is distributed.

The goal is to choose the risks a network is willing to accept and design around them. The right sequencer is the one that fits the network’s needs and trade-offs.

Modular, customizable sequencing will not remove all risks, but it allows more intentional designs that align with the diverse requirements of today’s decentralized ecosystems.

To ground this in practice, here’s a snapshot of how different networks are approaching sequencer risks today. Think of it as a menu of design choices, each with its own strengths and trade-offs, to help guide what might work best for you.

Each project will need to find the mix that fits its users, risk tolerance, and regulatory surface. Think of your configuration like a living threat model, one you revisit as conditions change.

We can keep arguing about the best sequencer. Or we can build sequencers that are the right fit for each network.

Make policy explicit. Make operations explicit. Make each piece replaceable.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, and Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

July was a big month across the ChainSafe ecosystem, with progress spanning privacy tech, infrastructure upgrades, and core client development. From bringing shielded ZEC to MetaMask to major client releases, our team has built features and improvements that make web3 faster, more private, and more reliable for everyone.

Shielded ZEC in MetaMask

ChainSafe released the Zcash Snap for MetaMask, letting users send and receive shielded ZEC right in their browser.

The work started in 2024 with a feasibility study, moved through two grants, and included audits to ensure its security. Now, shielded ZEC is easier to use for anyone with MetaMask.

ChainSafe developed WebZjs to make this possible, enabling core functionalities:

• zk-SNARK generation/verification
• shielded transaction handling
• Sapling key management

Zcash in the browser: Bringing shielded ZEC to MetaMask

After months of research, prototyping, and iteration, we’re proud to release the Zcash Snap: a fully functional MetaMask Snap that lets users send, receive, and manage shielded ZEC directly from the browser. The challenge Most cryptocurrencies work fine in a browser. Zcash isn’t most cryptocurren…

ChainSafeBen Adar

Infrastructure

In July, the Infra team focused on strengthening our systems and refining our processes. From network upgrades to expanding our operational expertise, we laid the groundwork for smoother performance and future growth.

• Rolled out commit-boost to replace mev-boost: this upgrade brings in richer validator performance metrics, giving us more insight to fine-tune operations and improve network efficiency.
• Joined an Obol DV cluster with Lodestar: three infra team members gained hands-on experience running distributed validator nodes, strengthening our expertise in DV tech.
• Moved our Canton deployment into a Kubernetes cluster: this migration boosts scalability, resilience, and maintainability for the service.
• Became an official Monad operator: further expanding our validator footprint and contributing to the decentralization of the Monad network. See our listing below:

gmonads.com

gmonads.com — Built for the Monad community. Get real-time insights into validator activity, block production, and network performance.

gmonads.com

Lodestar

In July 2025, Lodestar saw two major releases that improved performance, reliability, and licensing. The v1.32.0‑rc.0 (pre-release) and v1.32.0 (stable) introduced key features like increased gas limits, async Rust-based proof verification, and a switch to Apache-2.0 licensing, alongside performance boosts in block building and proposer metrics.

Building on this momentum, v1.33.0, released August 6th, continues to enhance Lodestar’s capabilities with important bug fixes, optimizations, and additional support for upcoming Ethereum protocol improvements.

Glamsterdam strategic proposal

Lodestar outlined their strategic proposal for Ethereum’s Glamsterdam upgrade, focusing on faster slot times through smarter structuring. The key idea is to implement EIP-7732 (Execution Payload Block Separation, or EPBS) to decouple consensus (beacon blocks and blobs) from execution payloads. This separation enables:

• More efficient pipelining of slot operations.
• Parallel development paths for consensus and execution layers.
• Boosted network stability and liveness, thanks to extended validation time and better bandwidth use.
• Collection of real-world data to inform future slot time optimizations.

Lodestar’s Glamsterdam headliner vision

Lodestar’s vision for Glamsterdam’s headliner is an opportunity to optimize Ethereum’s slot structuring, separate consensus and execution concerns, and ensure we set ourselves up for future success in shortening slot times with data on overhead concerns.

ChainSafePhil Ngo

Forest

v0.28.0 “Denethor’s Folly” Highly recommended release. Denethor's Folly introduces quality-of-life improvements for archival snapshot operations and resolves a memory leak issue affecting long-running nodes.

Key updates for developers:

• v0.28.0 “Denethor’s Folly” — Memory leak fix for long-running nodes and smoother archival snapshot operations.
• FRC-0108 snapshot format — F3 finality certificates embedded in CAR snapshots for much faster node bootstrap and reduced bandwidth.
• filecoin-requests-builder tool — Quickly generate real chain RPC calls for reproducible benchmarks across node implementations.
• Improved RPC conformance testing — Structured, machine-readable reports with detailed failure info for easier CI integration and debugging.

Other improvements:
The Forest team also added out-of-the-box Grafana monitoring, selectable snapshot export modes with speed estimates, and richer JSON decoding for Init and System actor messages.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

After months of research, prototyping, and iteration, we’re proud to release the Zcash Snap: a fully functional MetaMask Snap that lets users send, receive, and manage shielded ZEC directly from the browser.

Zcash Shielded Wallet

Customize your web3 experience with Zcash Shielded Wallet.

MetaMask Snaps Directory

The challenge

Most cryptocurrencies work fine in a browser. Zcash isn’t most cryptocurrencies.

Its shielded pool relies on zero-knowledge proofs, complex cryptographic systems that use custom circuits, massive proving keys, and memory-heavy computations. That’s never been a good fit for browser environments.

To make things trickier, wallet frameworks like MetaMask were never built to support shielded assets out of the box. There was no drop-in solution. If we wanted Zcash to work in a browser, we’d need a whole new approach. One that preserved its privacy guarantees without requiring users to install a custom node or give up control.

First, we asked, “Is this even possible?”

In Summer 2024, before writing a single line of code, we partnered with Zcash Community Grants on a feasibility study. The question was simple: Can shielded ZEC run in the browser?

We dug into the technical constraints—circuit size, memory limits, and proving time—and tested whether a subset of the proving system could realistically run client-side without compromising performance or correctness.

The answer? Yes. With the right optimizations, browser support wasn’t just possible, it was within reach.

That led to two grants:

• One from Zcash Community Grants to build a browser-native Zcash library.
• One jointly funded by the MetaMask Grants DAO and Zcash Community Grants to create a Snap using that library.

The first in-browser Zcash shielded transaction

As a proof of concept, we used an early prototype of the library (what would become WebZjs) to send the first-ever shielded transaction entirely in the browser.

Shielded ZEC, live in a browser tab. No custodians. No custom software. Just code and cryptography.

That first transaction proved it could be done. The next step was making it repeatable.

WebZjs: A Zcash library for browsers

We built WebZjs to provide all the core functionality developers need:

• Generate and verify zk-SNARKs
• Build and sign shielded transactions
• Interface with Sapling
• Manage wallet state and addresses

All in the browser. No compromises on privacy.

With WebZjs, developers can integrate shielded ZEC directly into web apps and wallet interfaces, making private transactions more accessible without lowering the security bar.

Wrapping it in a Snap

Next, we used WebZjs to build the MetaMask Zcash Snap.

Snaps let developers extend MetaMask with custom logic, safely sandboxed from the core wallet. That made it the perfect platform for Zcash: custom cryptography, privacy-preserving logic, and a familiar user experience.

The Zcash Snap supports:

• Fully shielded send/receive
• Sapling key management
• Native ZEC support in MetaMask
• A no-install, fully private workflow

And all of it works in a regular browser, with the same privacy guarantees Zcash is known for.

Ivan Rubido was able to demo the snap at ZconVI ⤵️

Iteration, testing, and community feedback

Throughout development, we worked closely with Zcash Community Grants and the broader Zcash community to test the Snap in real-world conditions. Users ran shielded transactions, verified outputs, and confirmed that privacy was preserved, just with better UX.

After several rounds of feedback, bug fixes, and a full audit by Hacken, we finalized the Snap and made it public.

It’s now live and ready for anyone to use.

Try the Snap here.

Zcash Shielded Wallet

Customize your web3 experience with Zcash Shielded Wallet.

MetaMask Snaps Directory

The community made this possible!

This work wouldn’t have been possible without the support and guidance of Zcash Community Grants and MetaMask Grants DAO.

Their vision for privacy-preserving infrastructure and willingness to push technical boundaries were critical at every step.

At ChainSafe, we believe privacy is a public good. Building tools like the Zcash Snap is part of our broader commitment to accessible, user-owned, and decentralized technology.

We’re proud to have brought shielded ZEC to the browser, and we can’t wait to see what builders do with it next.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

Lodestar's vision for Glamsterdam's headliner is an opportunity to optimize Ethereum's slot structuring, separate consensus and execution concerns, and ensure we set ourselves up for future success in shortening slot times with data on overhead concerns. We believe that shortening slot times is a great win for user experience (UX), but how we get there is also important.

In our proposal, we outline why using the slot time we have more efficiently is the first step to achieving the goal of shortening the length of slot times. Then, we outline the wins with EIP-7732 and why it matters. In addition, we can then parallelize development on both layers with an execution layer headliner.

Why we support EIP-7732

EIP-7732, also known as Execution Payload Block Separation (EPBS), achieves multiple goals, which we believe are worth the effort over competing headliners. These goals are:

• Maximally efficient pipelining of slot operations.
• Separate layer concerns between consensus beacon blocks/blobs and execution payloads.
• Larger positive impact on the liveness/stability of Ethereum via extended validation timeframes and improved bandwidth utilization.
• Allowing for the collection of real-world data to support the shortening of slot times.

Why is separating beacon blocks and execution payloads important?

Every second counts. Minimizing the dependencies of operations will help us better optimize, structure and utilize every millisecond of a slot operation effectively. We do this by decoupling execution validation from consensus validation, both logically and temporally. Currently, we have to acquire multiple pieces of data, which vary in size, thus affecting transmission times:

• Consensus block
• Execution payload
• Blobs

Then, we run the state transition function (processing the execution payload transactions to determine validity) so we can attest to the head block. As payloads get larger, the overhead on transmission and validation grows in time duration. Separating the dependencies unlocks massive efficiencies in how we utilize every millisecond of a slot.

With EPBS, attestations can be submitted if the consensus block is valid without the dependencies of all the pieces of data listed above, nor having to acquire and check validity of the execution payload itself (we remove the ExecutionPayload field from the BeaconBlockBody and replace it with a signed commitment as SignedExecutionPayloadHeader).

Consensus blocks are much smaller and faster to propagate and validate. Therefore, attestations can then be done earlier in the slot. Attestations will no longer need to be thrown out with invalid execution payloads, which minimize liveness failures if non-valid execution payloads are no longer being produced. For a network that majorily depends on out-of-protocol block production rather than local block building, there is a larger surface of possible malfunctions that can occur, risking Ethereum's liveness capabilities. This could be from builders submitting invalid blocks and/or trusted relays having various issues ranging from failure to demote invalid block builders to infrastructure failures:

• Post-Mortem: High Missed Block Rate Incident on 28/11/2024
• BloXroute - Feb 6th Post Mortem
• Titan Relay Incident Report: Withdrawals Root Validation Issue

EPBS ensures trustless builder accountability and helps to remove another vector for liveness failure.

In Delayed Execution (EIP-7886), the consensus block, execution payload and blobs still have the same propagation deadline, while EPBS, has different propagation deadlines and start times, which lead to better utilization of the slot.

If we also want to keep Ethereum as decentralized as possible, we need to consider supporting lower resourced nodes both in terms of compute and bandwidth. Supporting EPBS means we can restructure operators to efficiently use slots, reducing peak bandwidth requirements. Execution payloads can be transmitted over extended timeframes and executed with more time. Bandwidth is harder to scale due to physical network infrastructure requirements, so we must optimize its usage for scaling the network.

Invasiveness of shortening slot times

EIP-7782 for six second slot times are much more invasive on the client as it touches more aspects of the codebase than EPBS. There is also no compelling data at this point to show that client implementations can successfully upkeep a network like Ethereum mainnet that is now two times faster. Some clients may need to rework their architecture (such as their caches) to optimize for this future. Postponing EIP-7782 gives clients more time to ensure their clients are ready for this future.

We believe that data from the network and clients collected from restructuring the slot can inform us about changes we can make to the slot time in the H* hard fork.

Bonus integration of FOCIL?

Although we believe the priority on EPBS is higher than Fork-choice Enforced Inclusion Lists (FOCIL), research developments have allowed FOCIL to find a clean integration into EPBS. FOCIL can make a great non-headliner addition to Glamsterdam and is complimentary to the integration of EPBS. This would make for a great secondary goal to the headliner if achievable.

For those who are concerned about the degradating of UX due to increased average transaction inclusion time, there is a mitigation in the works for a dual-deadline PTC vote in EPBS where the Payload Timeliness Committee (PTC) observes for the availability of the payload first at four seconds and then for blob data availability at ten seconds.

Execution Layer Headliner: EIP-7919 Pureth

With EPBS, we can separate consensus and execution headliners for parallel workflows

If EPBS becomes the headliner for Glamsterdam, the work is scoped to be 100% for consensus layer clients. An added side benefit of this is testing for the most complex and impactful feature is constrained to consensus, minimizing coordination and testing surface overhead. This will then allow execution client teams to focus on their own main headliner which focuses mostly on execution.

We believe an execution headliner like EIP-7919 Pureth, will unlock massive UX benefits for infrastructure with minimal disruptions to EPBS implementation on the consensus layer. Pureth removes current issues with consuming chain data and includes a few minor tasks for consensus clients. These tasks include updating SSZ definitions for execution payload and helps future-proof changes to the beacon state container.

Other resources

There are multiple other benefits that we agree with which make EIP-7732 very appealing as a headliner for Glamsterdam. The resources are below for additional context:

• Potuz: EIP-7732 the case for inclusion in Glamsterdam
• Sigma Prime: Glamsterdam's Headliner
• Prysm Team's Glamsterdam Headliner

It's summer (at last!), and even though "summer break" is more of a memory than an expectation these days, ChainSafers are getting out there and making the most of the warm weather. Our team has been travelling the world, speaking at web3 conferences like EthCC in Cannes, Protocol Berg in Berlin, and the aptly named Berlinterop this June.

This month’s roundup highlights the speaking engagements ChainSafe has been proud to participate in so far, as well as updates to our validator node infrastructure and protocol implementations.

If you're going to be at Gamescom next month, be sure to look out for ChainSafers!

Infrastructure

In June, our infrastructure team made significant strides: we fully deployed Kubernetes clusters to orchestrate our Canton deployment with improved scalability, seamless continuous deployment, and auto-healing features.

Plus, our Monad node is live as a validator in Testnet-2! This deployment highlights our robust, production-ready infrastructure capabilities with Kubernetes and our expanding role in the Monad network.

EthCC in Cannes, France

This year, ChainSafe made a strong presence at EthCC in Cannes, France, with our VP of Engineering, Belma, delivering a standout solo presentation and participating in a high-profile panel.

On July 2, she explored “Open‑Source Development – What Are We Doing Wrong?”, highlighting common pitfalls in Web3 projects and sharing actionable strategies to foster more welcoming and collaborative open source communities.

The next day, she joined industry peers on the panel “Who Captures the Value: The Protocol or the dApp?” to debate how value accrues in the Ethereum stack and its implications for builders. Belma was joined by Lane Rettig from NEAR Foundation, Dani O. of EthDenver and MetaWebVC, Vangelis Andrikopoulos from Sei Labs, Guy Wuollet from a16z crypto, and Andy from The Rollup!

Belma’s talks captured the spirit of ChainSafe’s mission—advocating for open, collaborative ecosystems and pushing critical conversations that shape the future of web3.

Lodestar

In June, the Lodestar team laid out their plan to transition from a pure JavaScript client focused on browser compatibility to a performance-first, hybrid JavaScript and Zig architecture.

Read about the new direction for the Lodestar team here:

Lodestar’s next chapter: Blending Zig and JavaScript

Lodestar’s future: transitioning from a pure JavaScript client focused on browser compatibility to a performance-first, hybrid JavaScript and Zig architecture.

ChainSafeChainSafe

EthCC and the Ethereum Protocol Fellowship

While representing ChainSafe at EthCC, Lodestar lead Phil Ngo (@philngo_) has picked up four Ethereum Protocol Fellowship (EPF) fellows to work with and on our Ethereum implementation.

The EPF is a six-month program that gives developers the opportunity to contribute directly to Ethereum’s core protocol through hands-on projects and mentorship. The four fellows will work on Lodestar-related projects, working directly with the Lodestar team.

Protocol Berg Berlin

Phil Ngo and Nico Flaig travelled to Berlin to talk about one of ChainSafe's shining moments from this year—the Holesky hard fork rescue:

How client diversity saved Holesky: Lessons for the future

Holesky, the largest Ethereum public testnet experienced one of the best, unplanned events for chaos testing as it hard-forked to Pectra. With an invalid block, consistent forking and a non-finality period spanning a duration of three weeks, client and infrastructure teams were pushed to Ethereum’s…

ETHBerlin

Read about the Holesky fork and the need for client diversity on our blog:

Lodestar Holesky Rescue Retrospective

When the Holesky hard fork went sideways, validators struggled through hidden bugs, misconfigured clients, and unexpected chain splits. Here’s how quick thinking, precise fixes, and community teamwork turned a chaotic experience into a valuable lesson for future forks.

ChainSafePhil Ngo

Forest

ChainSafe's Filecoin implementation in Rust pushed a swath of updates last month. June's updates mean faster, more efficient performance for Forest users, especially for developers and node operators. RPC calls like StateWaitMsg are now significantly quicker, chain data is easier to inspect with new CLI tools, and the client is more aligned with the broader Filecoin ecosystem thanks to the Common Node API. Here's a look at Forest's June upgrades:

• Release: Forest v0.27.0 now live.
• F3 Snapshots: Preliminary design complete; FRC open for review.
• Common Node API: Merged! Aligns JSON-RPC behaviour across Filecoin clients.
• RPC Performance: Added dedicated caches for receipts/events → big speedup for StateWaitMsg & StateGetReceipt.
• Power Table Optimization: New blockstore read cache slashed query time from 1.7s to 0.7s.
• Snapshot Automation: New CLI command auto-generates and syncs snapshots to S3.
• CLI Parity with Lotus: New chain list command shows detailed gas and message stats per epoch.
• Actor Param Decoding (PoC): Early support for decoding parameters in Miner, Account, and EVM actors via StateDecodeParams.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

Written by Aleksei Krasnoperov

Filecoin is built on a vision of decentralization and client diversity, where multiple implementations coexist, driving innovation, resilience, and security across the ecosystem. However, for this vision to flourish, a fundamental problem needs to be addressed: interoperability.

Right now, Lotus is the primary reference implementation for Filecoin. Builders, miners, and applications widely adopt its JSON-RPC interface. But there's no formally agreed-upon standard for this interface. As a result, building applications compatible across multiple Filecoin nodes becomes an exercise in reverse engineering and guesswork.

Enter FRC-0104: The Common Node API.

What's the problem?

As the de facto standard, the Lotus RPC API offers over 280+ methods. However, since these methods were developed naturally over time by the Lotus team, some of them are core to the entire network, while others are more Lotus specific. This makes it a bit painful for alternative client developers to determine which methods are which.

Lack of standardization means:

• Slowing down development as builders spend resources deciphering interfaces.
• Leaving developers uncertain about API stability and compatibility.

How FRC-0104 addresses this

FRC-0104 proposes a clearly defined standardized API for Filecoin nodes using an OpenRPC schema. It focuses on methods actively used in real-world applications, distilled down from the existing Lotus API.

FRC-0104 includes a subset of methods carefully identified through extensive exploration with the Lotus team. The selected methods reflect actual, widespread needs, eliminating redundancy and ensuring clarity for implementers.

This proposal categorizes RPC methods into four essential groups:

• State queries: Methods to access chain, actor, and network states.
• Miner operations: Essential methods miners need for consensus participation.
• Transactions: Methods for submitting transactions and managing the mempool.
• Node operations: Methods for basic node management and inspection.

Complete list of Common node API methods

By defining a common node API in FRC-0104, we’re unlocking the ability to build robust, automated conformance testing tools. These tools can check whether implementations adhere to the standard structurally, behaviorally, and functionally. To support this, the Forest team is externalizing their RPC Snapshots Testing Suite, making it easier for any team to validate their implementation against FRC-0104.

The result? A minimal, predictable, and universally compatible API standard.

Why OpenRPC?

With FRC-0104, we now have complete JSON schemas, detailed method descriptions, and consistent documentation that aligns precisely with implementation.

By proposing a normative OpenRPC schema, FRC-0104 provides a standardized, language-agnostic, and machine-readable definition of each method. This approach enables a simplified process for syntax compliance checks, method specificity, conformance validation, streamlined tooling across diverse programming languages and environments, and, critically, reducing redundant effort across teams.

Real-world impact

With FRC-0104 in place, developers can confidently build Filecoin applications that work seamlessly across Lotus, Forest, Venus, and any future implementations. Allowing node operators and miners flexibility in selecting the client implementation best suited to their needs, knowing their applications will remain compatible.

Implementing the Common Node API will also encourage clearer security considerations, allowing node operators to implement standardized access control and auditing measures.

What's next?

Establishing this standard specification encourages greater accountability among node implementers and, hopefully, more thoughtful consideration of future iterations. FRC-0104 will help mitigate rapid, unchecked iterations of the API, reducing technical debt over the long term.

By embracing a unified API standard, Filecoin can strengthen client diversity, streamline development, and move towards a robust, interoperable future.

Let's help Filecoin speak the same language!

Forest

The Filecoin Implementation in Rust

Forest

Forest is a Rust implementation of the Filecoin protocol, offering fast performance with minimal hardware requirements. Use it to validate and access chain data, run an RPC node, generate snapshots, and manage FIL with a built-in wallet.

Forest is built and maintained by ChainSafe

Website | Youtube | Twitter | Linkedin | GitHub

Written by Cayman Nava.

Thanks to Matthew Keil for feedback.

TLDR: Lodestar is rewriting core modules in Zig and switching to Bun as the primary runtime.

Since 2019, the Lodestar team has been an active contributor in the Ethereum R&D ecosystem. Over the years, we’ve built a robust production-grade beacon node and validator in TypeScript, which now powers a notable portion of the Ethereum network, especially among solo stakers.

As Ethereum’s protocol has evolved, so has our client. Quietly, we’ve begun replacing performance bottlenecks in JavaScript with native code. Today, we're excited to publicly share our vision for Lodestar’s future: transitioning from a pure JavaScript client focused on browser compatibility to a performance-first, hybrid JavaScript and Zig architecture. Don’t worry, browser support for key libraries the community depends on isn’t going anywhere.

Our JavaScript journey

In the early days-before the Beacon Chain launched (remember the Eth2 meme?)-we anticipated strong demand for Ethereum tooling in the browser. Naturally, we chose TypeScript.

But what we’ve learned is that Ethereum browser tooling is a niche-important, but not where a client team has the most impact. The biggest value we provide is running performant infrastructure that supports mainnet. Our true contributions are the EIPs we write, the opinions we share, and the consensus we support. That’s where Lodestar has influence in the ecosystem.

The challenge? JavaScript is not designed for high-performance applications. You can’t control memory layout. You can’t share structured data across threads. And the runtimes have deep, often unpredictable performance quirks.

But that didn't stop us. Despite those hurdles, we built a production-grade beacon node. Our client is now consistently a top performer: our Lido validators routinely operate within 0.1% effectiveness of the best-performing validators on mainnet.

These hard-won victories shaped the skills and mindset now driving our next leap forward.

Tradeoffs in the bouncy house

Trivia time:

Q: How many bytes does a 32-byte Uint8Array take in Node.js?
A: 217 bytes. 😭

Q: What’s the fastest way to iterate through the bits of a BigInt in Node.js?
A: Convert it to a base-2 string and iterate over the characters. 🤪

These are the kinds of realities we’ve had to work around. JavaScript is bound—gloriously and painfully—to its runtime.

Every language poses unique challenges when building a robust client. In JavaScript’s case, those challenges pushed us to get clever: squeezing every last drop of performance out of a leaky abstraction.

Over the past year, the biggest wins have come from moving bottlenecked workloads into native code-C, Rust, and now Zig. That strategy has shaped our roadmap. We’re confident that doubling down will continue to raise Lodestar’s reliability, efficiency, and long-term maintainability.

Zig: Control and clarity

We’ve been experimenting with Zig for over a year and we've grown to love its laser focus. Zig is designed to be the simplest tool for writing optimal low-level software without surprises or hidden costs.

Its combination of simplicity, control, and performance makes it a perfect fit for building robust Ethereum infrastructure.

Zig’s ecosystem is promising, but still young. The library landscape isn’t yet battle-tested. We’re helping improve that-by contributing upstream, building Ethereum-specific tooling, and integrating with C libraries.

We see our participation in the Zig ecosystem as a long-term investment. We're already collaborating with teams like Zeneth and Zeam, and we’re excited to help shape a vibrant, reliable Zig-powered stack for Ethereum infra.

Active Zig projects we’re working on:

• ssz-z – feature-rich SSZ toolkit
• zbuild – Zig build configuration using zon
• snappy – Zig bindings for Google’s Snappy
• hashtree – Zig binding for hashtree
• blst-z – Zig binding for blst
• zig-discv5 – Zig implementation of discv5 (in progress)
• bun-ffi-z – Utility to build/publish Zig FFI modules for Bun

Bun: Performance edge & integrated tooling

Bun is a modern JavaScript runtime, written in Zig and built for speed. It’s a compelling alternative to Node.js, offering:

• Lower latency
• Higher throughput (via JavaScriptCore)
• A tight and fast FFI
• First-class tooling (bundler, test runner, etc.)

For Lodestar, this means less overhead and faster execution, especially for CPU-bound tasks like cryptographic operations and serialization. In our benchmarks, Bun outperforms Node.js by up to 20% on these workloads.

While Node's N-API is a solid tool for native integration, Bun's leaner FFI unlocks tighter, more efficient interoperability with our native Zig code.

And for those of you relying on our browser tooling: nothing’s changing. We’ll continue supporting our TypeScript libraries and shipping WASM builds for critical modules.

Lodestar: Ship of Theseus

If a ship has all its parts replaced, is it still the same ship?

The architecture of Lodestar is solid. But JavaScript’s inherent limits aren’t going away. Rather than contorting around them, we’re choosing to transcend them.

Bit by bit, module by module, we’re rebuilding Lodestar-starting with cryptography, hashing, SSZ types, and the state transition-in Zig.

This unlocks architectural improvements we’ve long wanted but couldn’t justify. Many performance bottlenecks will simply disappear.

In the near term, we’ll release Lodestar 2.0, which targets Bun and includes several breaking changes.

Longer term, we plan to integrate a native networking stack. Eventually, we might leave JavaScript behind entirely-but only if it helps us build the best possible Ethereum client.

Our north star is not any particular language or runtime, it’s impact. It’s doing what needs to be done to build infrastructure that Ethereum can depend on.

What does this mean for you?

• Lodestar 2.0 will "just work™" in Docker and binary releases.
  • For source installs: just swap Node.js for Bun.
• Browser builds live on. WASM+TypeScript tooling stays, but it no longer leads the roadmap.
• Cold-start times and first-slot performance are dramatically improved.
• New ways to contribute:If you know Zig, we need you.Curious about Bun FFI, we need you.Want to benchmark or fuzz, we need you.

Conclusion: building for impact

Six years. Hundreds of thousands of lines of code. Thousands of mainnet epochs. And a few gray hairs.

What we’ve learned is this: the best way to meet Ethereum’s evolving challenges is by aggressively optimizing our hottest code paths with the best tools available.

Zig gives us precision, predictability, and fearless concurrency. Bun gives us developer experience we can love-without compromising speed.

Together, they allow Lodestar to shine brighter than ever.

This isn’t about chasing hype. It’s about building the leanest, most trustworthy Ethereum consensus client we can imagine.

If that mission resonates with you, jump into our Discord, open an issue, or fire off a PR. Whether you cheer us on or tell us why we’re wrong, you’re helping steer the ship.

Onward to Lodestar 2.0 - lighter, sharper, and ready for the next 10 million slots.

Hey everyone, we've got a lot to share this month! From a fresh new website to exciting protocol updates, validator milestones, and a sneak peek at what’s coming next, the ChainSafe team has been busy across the stack.

ChainSafe.io's redesign

We are thrilled to announce the launch of our redesigned website! 

The new site design offers improved navigation and detailed information about our protocol, infrastructure, and co-development projects. Everything ‘ChainSafe’ is front and center on our new website, including two new initiatives we’re thrilled to share with you:

New Optimism Rust OP-Supervisor

Built to keep the Superchain connected and secure, the OP-Supervisor validates messages, tracks block safety, and makes sure everything stays in sync across chains. Acting as a reliable L1 data source, it handles rollup verification, re-orgs, and state consistency so interoperability stays smooth, safe, and ready to scale. 

Stay tuned for updates!

ChainSafe Staking

We’ve proven our chops as infrastructure operators, so we’re opening the door (soon!) to clients who want to delegate funds to our staking operations. 

ChainSafe is currently active on these networks, with plans to add more:

• Lido
• Celestia
• Namada
• Kusama
• Polkadot (coming soon)

Infrastructure

May was all about tightening access, testing portability, and deepening validator involvement. The infra team kept things moving behind the scenes—and even went on the road with the Forest team. All the while, the Infrastructure team is working hard to open our staking services for new business!

FilDevSummit Toronto

Infra was represented at FilDevSummit where team lead Josh (@joshdougall) demoed a cool piece of handiwork: a portable Raspberry Pi 5 running a Forest RPC node over battery and 5G hotspot. Find more info on that our socials soon :)

Validator milestones

We were accepted into the Namada Validator Foundation Delegation Program and successfully deployed a Canton validator, deepening our integration into the ecosystem.

Improved access with OPKSSH

We rolled out OPKSSH, a single sign-on SSH tool leveraging Google Workspace sign-ins and OpenID Connect. This enhances security by managing SSH access via identities and Google Groups, replacing long-lived keys with temporary tokens and MFA.

Lodestar

Lodestar shipped two back-to-back releases focused on stability, usability, and future-proofing for Ethereum’s evolving roadmap.

v1.30.0 (May 14)

• Focused on improving the validator experience and smoothing out compatibility issues with recent network changes.
• Delivered a cleaner, more reliable staking journey for both advanced users and newcomers.

v1.31.0 (May 28)

• Continued UX refinements and resolved lingering bugs.
• Recommended upgrade for anyone on older versions to ensure optimal performance and readiness for upcoming protocol changes.

Forest

May was a big month for Forest, featuring new tooling, CLI enhancements, and direct engagement with the community at FIL Dev Summit Toronto. The team returned with valuable feedback already influencing Forest’s roadmap.

Development Highlights

• Introduced a Snapshot Garbage Collector, enabling users to prune the chain and export lite snapshots with a new forest-cli chain prune snap command.
• Improved tooling transparency via the forest-tool archive info command, now displaying index size for better snapshot visibility.
• Aligned RPC token handling with Lotus, ensuring CLI interactions save tokens by default for smoother usage.

Discoverability Boost

• The Forest Faucet saw SEO improvements, making it easier for users to find and use this critical onboarding tool.

Shoutout to @barbaraperic for leading SEO improvements on the Forest Faucet!

More details are in the changelog.

Gossamer

This month, the Gossamer team made meaningful strides across Polkadot protocol support, storage improvements, and JAM/PVM implementation, while engaging with the community in person at JAM Experience Lisbon.

Gossamer joined JAM Experience Lisbon 2025, where the team gave a spontaneous project update, shared ideas with fellow JAM implementers, and participated in local “JAM Toaster” events.

Be sure to check this out: Gossamer’s Kirill and Jimmy appeared for an interview with Pala Labs, discussing Gossamer’s role in JAM:

Core Development Progress

• Continued integration of the GRANDPA finality gadget into Gossamer.
• Advanced the Polkadot storage layer by integrating a refactored database backend for improved performance and maintainability.

Polkadot ELVES Implementation

• Built out key subsystems for collator-validator communication, unbacked candidate distribution, and validator group gossip protocols.
• Began implementing a relay-parent–aware data structure for improved statement routing across the network.

JAM & PVM Integration

• Developed core JAM protocol logic, including block import routines and host functions for PVM communication.
• Improved memory management in the PVM to support multiple pages, and explored strategies for PVM recompilation and JAM-NP research

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub

At its core, Ethereum was designed with one intention: decentralization. But as Ethereum grows, staying true to that intention becomes harder.

One of the thorniest challenges? MEV (Maximal Extractable Value). Introducing technical vulnerability and incentive misalignment, MEV reshapes how blocks are built, who builds them, and who benefits from them.

MEV forces the protocol to answer uncomfortable questions: Should Ethereum give up on local block production? Should Ethereum hand over block construction to sophisticated external builders? Should Ethereum enshrine those builders in-protocol?

On May 13, 2025, a discussion between the Lodestar and Flashbots teams offered a window into how these questions are being tackled at the protocol level and at the very ethos of Ethereum's values. The conversation was about what kind of network Ethereum wants to become and what it might have to give up along the way.

ePBS: Fixing the relay problem, or locking in a new one?

When Ethereum introduced PBS (Proposer-Builder Separation) via MEV-Boost, it did so off-protocol, using relays as trusted intermediaries to pass signed blocks from builders to proposers. The decision was made in the interest of time. It allowed Ethereum validators to access more competitive, high-MEV blocks without requiring them to run sophisticated infrastructure themselves. But in doing so, it introduced a new layer of complexity and risk.

Relays aren’t part of the protocol. They can fail silently, be censored, or gamed.

ePBS (enshrined PBS) is meant to fix that. By baking PBS into the protocol itself, ePBS would remove the need for relays entirely. Builders would communicate directly with proposers in a trustless, standardized way. Payments would be enforced. Timing games, like waiting to see other bids or delaying payload publication, would be harder to pull off. 

The current ePBS design introduces a Payload Timeliness Committee (PTC), a selected group of validators who vote on whether a builder’s execution payload arrived on time. Their vote determines whether the block is treated as valid and full, or ignored as empty, discouraging strategic delays. In simpler terms: the committee acts as a check on builder behaviour. If a builder stalls, the protocol can penalize them by excluding their block or reducing its reward, keeping things fair and timely. 

It’s a clever design. One trust assumption, relays, is replaced by another, but this time inside the protocol, with cryptoeconomic enforcement.

Sounds like a win… Until you zoom out.

“Many MEV researchers... are quite nervous about enshrining any PBS solution at this point in time because it's still changing very rapidly. You run a very high risk, frankly, of enshrining something suboptimal.”
—Hasu, Strategy Lead, Flashbots 

Hasu’s worry isn’t about the goal of ePBS, it’s about the timing. Ethereum’s block-building market is still in flux. New designs like pre-confirmations, based rollups, and encrypted mempools are just beginning to take shape.

Hasu’s worry isn’t about the goal of ePBS, it’s about the timing. Ethereum’s block-building market is still in flux. New designs like pre-confirmations, based rollups, and encrypted mempools are just beginning to take shape. 

That’s the deeper tension. Today, builders are offchain actors. Their role is shaped by market forces, not protocol mandates. Validators can choose to use them or build locally. But once PBS is codified into the protocol, that flexibility shrinks. The builder-proposer relationship becomes part of Ethereum’s core logic. And in doing so, it might crowd out other designs entirely.

By embedding one version of PBS into the protocol, Ethereum risks freezing out other paths, like deterministic ordering, encrypted mempools, or preconfirmations, before they’ve had a chance to mature. Enshrining PBS now might lock-in the wrong assumptions into the protocol.

“ePBS basically means we give up on... solutions [like deterministic ordering] and just hand it over to the builder network.”
—Nico Flaig, Protocol Engineer, Lodestar

Enshrining PBS might solve one trust problem (relays) while hardcoding a bigger one: permanent dependence on external builders, most of whom are private, profit-maximizing entities.

BuilderNet: De-risking our dependence on private builders

Flashbots don’t want to enshrine PBS. But they do want to fix the builder market.

Their proposal? BuilderNet, a decentralized block building network that runs on hardware-secured enclaves, known as Trusted Execution Environments (TEEs).

The idea is ambitious: keep the performance and sophistication of external builders, but remove their opacity and control. Use hardware to prove that builders aren’t front-running users. Use privacy to stop sandwich attacks. Use structure to return value to users and validators alike.

On paper, it sounds like the best of both worlds: privacy, transparency, decentralization, and efficiency.

“If your transaction creates any MEV or pays any excess transaction fee, you can be refunded… when BuilderNet generates any surplus against other builders, anybody who contributed to that surplus can get a refund.”
—Hasu

That surplus comes from order flow. If BuilderNet can attract high-quality transaction flow from wallets, dapps, and users, then win blocks with it, it can split the rewards among the people who made that order flow possible. That includes users too, not just validators and builders.

It’s a new kind of auction. One that handles blockspace and attention.

Still, BuilderNet asks for trust in a different place. Instead of trusting a builder’s incentives, you’re trusting the guarantees of the TEE: a black box of silicon, manufactured by Intel or AMD, running unobservable code. It’s verifiable in theory but fragile in practice. SGX, Intel’s last TEE, was retired. Flashbots now uses Intel TDX and hopes to diversify across manufacturers for redundancy. But for now, trust in the hardware is table stakes.

The other risk? Success. If BuilderNet works too well, if it becomes the only place builders want to build, does Ethereum just recreate the same centralization under a new banner?

That’s the paradox BuilderNet is trying to navigate: use centralizing forces (speed, scale, specialization) to create decentralizing outcomes (user refunds, public infrastructure, transparency). Whether it works will depend on a combination of tech and governance. Hasu was clear:

“BuilderNet is intended to become its own decentralized network with open contribution, several teams, a governance process, BuilderNet improvement proposals.”
—Hasu

It’s a protocol by another name, but unlike ePBS, it isn’t enshrined. It can evolve in the open or fail fast without taking Ethereum with it.

The builder dilemma

For a long time, building blocks was the validator’s job. You ran a node, had a mempool, and ordered transactions. It wasn’t always the most profitable, but it was yours.

MEV changed that. As the value of block construction increased, so did the pressure to specialize. Enter block builders, external actors with fast infrastructure, deep exchange access, and custom logic to extract value from every transaction. Suddenly, the validator’s job wasn’t to build, it was to outsource its right to propose a block to someone else in exchange for a share of the profits.

For some, this tradeoff feels unavoidable. For others, it feels like capitulation.

“The builders are antithetical to the ethos of why we all join the blockchain space... to build a better future that is more egalitarian and more meritocratic.”
—Matthew Keil, Protocol Engineer, Lodestar

Keil’s concern isn’t about performance. It’s about power. When only a few players can build blocks, they set the rules for who gets in and who doesn’t.

Hasu sees it differently. In his view, builders are already solving real problems: they keep the validator set decentralized, distribute MEV more evenly, and improve user experience by reducing gas costs and sandwich attacks. To him, the question isn’t whether builders should exist, it’s how to constrain them. In his view, MEV should be minimized at the wallet and app layer, where users can assert control, and harnessed at the builder layer, where incentives and enforcement can be standardized.

And then there’s a third perspective. Beyond ideals, what’s actually feasible?

“How far do you think that we can scale local builders and still keep good liveness rates?”
—Data Always, Researcher, Flashbots

In other words, even if you want decentralization, can you actually run it onchain, on time, on hardware that doesn’t cost a fortune?

The deeper question here is: What kind of decentralization do we want, and who gets to participate in it? Is it about making sure every validator can build blocks locally on a Raspberry Pi? Or is it about making sure users, not only builders, benefit from block production? Ethereum may not be able to have both.

Building on shared ground

Even if they don’t agree on how to get there, there was definitely one major consensus among both teams: keep Ethereum credible, decentralized, and performant. 

And in some areas, the paths converge.

Inclusion Lists (FOCIL) came up again and again. Both Lodestar and Flashbots saw them as a pragmatic step, one that could improve censorship resistance without overhauling the builder market.

Inclusion lists allow block proposers to require certain transactions to be included in blocks. Used wisely, they can reduce the worst effects of MEV and reintroduce some protocol-native accountability.

Lower slot times also drew agreement. The logic is straightforward: when blocks happen faster, the window for MEV games gets smaller.

Hasu noted that lower latency reduces MEV in decentralized exchanges, especially in backrunning scenarios. Nico pointed out that faster slots could improve user experience while aligning with rollup-centric futures. No one saw it as controversial, just incomplete. A useful tool, but no cure.

Ethereum doesn’t need everyone to think the same. It needs people to build things that challenge each other’s assumptions and make the network stronger in the process.

Where it goes from here

No one left the conversation thinking the problem was solved. If anything, the discussion sharpened the uncertainty.

BuilderNet might work. It might not. ePBS might prove essential. Or it might ossify too early. Local block building might fade. Or it might stage a comeback, but realistically this would only happen if we can give it better incentives. Right now, it’s just so much more rational to outsource to builders who can extract more MEV. 

What’s clear is that Ethereum is at a fork, in the protocol and in its design philosophy. Does the network want to stay flexible, relying on external markets and modular upgrades? Or does it want to fold its infrastructure inward, codifying roles and responsibilities at the protocol layer?

BuilderNet, for now, represents an attempt to have it both ways: a system that isn’t enshrined, but still aspires to protocol-level trust. 

It’s a bold bet that buys us some time. Time that Ethereum could use to keep experimenting without rushing into permanence.

The work isn’t done. But it’s clearer now where the edges are.

About ChainSafe

ChainSafe is a leading blockchain research and development firm specializing in protocol engineering, infrastructure development, co-development and web3-enabled gaming.

Alongside its contributions to major ecosystems such as Ethereum, Polkadot, Filecoin, ChainSafe creates solutions for developers and teams across web3. 

As part of its mission to build accessible and improved tooling for developers, ChainSafe embodies an open source and community-guided ethos to advance the future of the internet.

Website | Youtube | Twitter | Linkedin | GitHub