ethrex

ethrex is a minimalist, stable, modular, fast, and ZK native Ethereum client built from the ground up with zero-knowledge proving in mind. Whether you're running an L1 node or building an L2, ethrex provides the foundation for verifiable Ethereum execution.

Why ethrex?

zkVM Integrations

ethrex integrates with multiple zero-knowledge virtual machines, giving you flexibility in how you prove Ethereum execution.

Quick Start

Run an L1 node:

# Install ethrex
cargo install ethrex

# Start syncing mainnet
ethrex --network mainnet

Deploy an L2:

# See the full deployment guide
# https://docs.ethrex.xyz/l2/deployment/overview.html

Architecture Highlights

ethrex's architecture is optimized for both traditional execution and ZK proving:

• Stateless execution - Block execution can run with only the necessary witness data, enabling efficient proving
• Modular VM (LEVM) - Our EVM implementation is designed for clarity and easy auditing
• Optimized tries - Merkle Patricia Trie operations are tuned to reduce zkVM cycle counts
• Precompile patches - Cryptographic operations use zkVM-accelerated implementations when available

Learn More

• zkVM Integrations - Detailed guide to supported proving backends
• Benchmark Comparisons - Performance data vs other implementations
• Case Studies - How teams are using ethrex
• Architecture Overview - Deep dive into ethrex internals

Get Involved

• GitHub - Star us, open issues, contribute
• Telegram - Join the community chat
• Blog - Technical deep dives and updates

Getting started

Ethrex is a minimalist, stable, modular and fast implementation of the Ethereum protocol in Rust. The client supports running in two different modes:

• ethrex L1 - As a regular Ethereum execution client
• ethrex L2 - As a multi-prover ZK-Rollup (supporting SP1, RISC Zero and TEEs), where block execution is proven and the proof sent to an L1 network for verification, thus inheriting the L1's security. Support for based sequencing is currently in the works.

Quickstart L1

Follow these steps to sync an ethrex node on the Hoodi testnet.

MacOS

Install ethrex and lighthouse:

# create secrets directory and jwt secret
mkdir -p ethereum/secrets/
cd ethereum/
openssl rand -hex 32 | tr -d "\n" | tee ./secrets/jwt.hex

# install lighthouse and ethrex
brew install lambdaclass/tap/ethrex
brew install lighthouse

On one terminal:

ethrex --authrpc.jwtsecret ./secrets/jwt.hex --network hoodi

and on another one:

lighthouse bn --network hoodi --execution-endpoint http://localhost:8551 --execution-jwt ./secrets/jwt.hex --checkpoint-sync-url https://hoodi.checkpoint.sigp.io --http

Linux x86

Install ethrex and lighthouse:

# create secrets directory and jwt secret
mkdir -p ethereum/secrets/
cd ethereum/
openssl rand -hex 32 | tr -d "\n" | tee ./secrets/jwt.hex

# install lighthouse and ethrex
curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-linux-x86_64 -o ethrex
chmod +x ethrex
curl -LO https://github.com/sigp/lighthouse/releases/download/v8.0.0/lighthouse-v8.0.0-x86_64-unknown-linux-gnu.tar.gz
tar -xvf lighthouse-v8.0.0-x86_64-unknown-linux-gnu.tar.gz

On one terminal:

./ethrex --authrpc.jwtsecret ./secrets/jwt.hex --network hoodi

and on another one:

./lighthouse bn --network hoodi --execution-endpoint http://localhost:8551 --execution-jwt ./secrets/jwt.hex --checkpoint-sync-url https://hoodi.checkpoint.sigp.io --http

For other CPU architectures, see the releases page.

Quickstart L2

Follow these steps to quickly launch a local L2 node. For advanced options and real deployments, see the links at the end.

MacOS

# install ethrex
brew install lambdaclass/tap/ethrex
ethrex l2 --dev

Linux x86

# install ethrex
curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-linux-x86_64 -o ethrex
chmod +x ethrex
./ethrex l2 --dev

For other CPU architectures, see the releases page.

Where to Start

• Want to run ethrex in production as an execution client?

  See Node operation for setup, configuration, monitoring, and best practices.

• Interested in deploying your own L2?

  See L2 rollup deployment for launching your own rollup, deploying contracts, and interacting with your L2.

• Looking to contribute or develop?

  Visit the Developer resources for local dev mode, testing, debugging, advanced CLI usage, and the CLI reference.

• Want to understand how ethrex works?

  Explore L1 fundamentals and L2 Architecture for deep dives into ethrex's design, sync modes, networking, and more.

Hardware Requirements

NOTE: The guidance in this document applies to running an L1 (Ethereum) node. L2 deployments (sequencers, provers and related infra) have different hardware profiles and operational requirements — see the "L2" section below for details.

Hardware requirements depend primarily on the network you're running — for example, Hoodi, Sepolia, or Mainnet.

General Recommendations

Across all networks, the following apply:

• Disk Type: Use high-performance NVMe SSDs. For multi-disk setups, software RAID 0 is recommended to maximize speed and capacity. Avoid hardware RAID, which can limit NVMe performance.
• RAM: Sufficient memory minimizes sync bottlenecks and improves stability under load.
• CPU: 4-8 Cores.
  • x86-64 bit Processors must be compatible with the instruction set AVX2.

Disk and Memory Requirements by Network

L2

TBD

Installation

Ethrex is designed to run on Linux and macOS.

There are 4 supported methods to install ethrex:

• Binary distribution
• Package manager
• Docker image
• Building from source

After following the installation steps you should have a binary that can run an L1 client or a multi-prover ZK-rollup with support for SP1, RISC Zero and TEEs.

Install ethrex (binary distribution)

This guide explains how to quickly install the latest pre-built ethrex binary for your operating system.

Prerequisites

• curl (for downloading the binary)

Download the latest release

Download the latest ethrex release for your OS from the GitHub Releases page.

Linux x86_64

curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-linux-x86_64 -o ethrex

Linux x86_64 with GPU support (for L2 prover)

If you want to run an L2 prover with GPU acceleration, download the GPU-enabled binary:

curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-linux-x86_64-gpu -o ethrex

Linux ARM (aarch64)

curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-linux-aarch64 -o ethrex

Linux ARM (aarch64) with GPU support (for L2 prover)

If you want to run an L2 prover with GPU acceleration, download the GPU-enabled binary:

curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-linux-aarch64-gpu -o ethrex

macOS (Apple Silicon, aarch64)

curl -L https://github.com/lambdaclass/ethrex/releases/latest/download/ethrex-macos-aarch64 -o ethrex

Set execution permissions

Make the binary executable:

chmod +x ethrex

(Optional) Move to a directory in your $PATH

To run ethrex from anywhere, move it to a directory in your $PATH (e.g., /usr/local/bin):

sudo mv ethrex /usr/local/bin/

Verify the installation

Check that Ethrex is installed and working:

ethrex --version

Install ethrex (package manager)

Coming soon.

Installing ethrex (docker)

Run Ethrex easily using Docker containers. This guide covers pulling and running official images.

Prerequisites

• Docker installed and running

Pulling the Docker Image

Latest stable release:

docker pull ghcr.io/lambdaclass/ethrex:latest

Latest development build:

docker pull ghcr.io/lambdaclass/ethrex:main

Specific version:

docker pull ghcr.io/lambdaclass/ethrex:<version-tag>

Find available tags in the GitHub repo.

Running the Docker Image

Check the Image

Verify the image is working:

docker run --rm ghcr.io/lambdaclass/ethrex --version

Start an ethrex Node

Run the following command to start a node in the background:

docker run \
    --rm \
    -d \
    -v ethrex:/root/.local/share/ethrex \
    -p 8545:8545 \
    -p 8551:8551 \
    -p 30303:30303 \
    -p 30303:30303/udp \
    -p 9090:9090 \
    --name ethrex \
    ghcr.io/lambdaclass/ethrex \
    --authrpc.addr 0.0.0.0

What this does:

• Starts a container named ethrex
• Publishes ports:
  • 8545: JSON-RPC server (TCP)
  • 8551: Auth JSON-RPC server (TCP)
  • 30303: P2P networking (TCP/UDP)
  • 9090: Metrics (TCP)
• Mounts the Docker volume ethrex to persist blockchain data

Tip: You can add more Ethrex CLI arguments at the end of the command as needed.

Managing the Container

View logs:

docker logs -f ethrex

Stop the node:

docker stop ethrex

Building ethrex from source

Build ethrex yourself for maximum flexibility and experimental features.

Prerequisites

• Rust toolchain (use rustup for easiest setup)
• libclang (for RocksDB)
• Git
• solc (v0.8.31) (for L2 development)

L2 contracts

If you want to install ethrex for L2 development, you may set the COMPILE_CONTRACTS env var, so the binary has the necessary contract code.

export COMPILE_CONTRACTS=true

Install via cargo install

The fastest way to install ethrex from source:

cargo install --locked ethrex --git https://github.com/lambdaclass/ethrex.git

Optional features:

• Add --features sp1,risc0 to enable SP1 and/or RISC0 provers
• Add --features gpu for CUDA GPU support

Install a specific version:

cargo install --locked ethrex --git https://github.com/lambdaclass/ethrex.git --tag <version-tag>

Find available tags in the GitHub repo.

Verify installation:

ethrex --version

Build manually with cargo build

Clone the repository (replace <version-tag> with the desired version):

git clone --branch <version-tag> --depth 1 https://github.com/lambdaclass/ethrex.git
cd ethrex

Build the binary:

cargo build --bin ethrex --release

Optional features:

• Add --features sp1,risc0 to enable SP1 and/or RISC0 provers
• Add --features gpu for CUDA GPU support

The built binary will be in target/release/ethrex.

Verify the build:

./target/release/ethrex --version

(Optional) Move the binary to your $PATH:

sudo mv ./target/release/ethrex /usr/local/bin/

Running an Ethereum Node with ethrex

This section explains how to run an Ethereum L1 node using ethrex. Here you'll find:

• Requirements for running a node (including the need for a consensus client)
• Step-by-step instructions for setup and configuration
• Guidance for both new and experienced users

If you already have a consensus client running, you can skip directly to the node startup instructions. Otherwise, continue to the next section for help setting up a consensus client.

Connecting to a consensus client

Ethrex is an execution client built for Ethereum networks after the merge. As a result, ethrex must operate together with a consensus client to fully participate in the network.

Consensus clients

There are several consensus clients and all of them work with ethrex. When choosing a consensus client we suggest you keep in mind client diversity.

• Lighthouse
• Lodestar
• Nimbus
• Prysm
• Teku
• Grandine

Configuring ethrex

JWT secret

Consensus clients and execution clients communicate through an authenticated JSON-RPC API. The authentication is done through a jwt secret. Ethrex automatically generates the jwt secret and saves it to the current working directory by default. You can also use your own previously generated jwt secret by using the --authrpc.jwtsecret flag or JWTSECRET_PATH environment variable. If the jwt secret at the specified path does not exist ethrex will create it.

Auth RPC server

By default the server is exposed at http://localhost:8551 but both the address and the port can be modified using the --authrpc.addr and --authrpc.port flags respectively.

Example

ethrex --authrpc.jwtsecret path/to/jwt.hex  --authrpc.addr localhost --authrpc.port 8551

Node startup

Supported networks

Ethrex is designed to support Ethereum mainnet and its testnets

For more information about sync modes please read the sync modes document. Full syncing is the default, to switch to snap sync use the flag --syncmode snap

Run an Ethereum node

This guide will assume that you already installed ethrex and you know how to set up a consensus client to communicate with ethrex.

To sync with mainnet

ethrex --syncmode snap

To sync with sepolia

ethrex --network sepolia --syncmode snap

To sync with holesky

ethrex --network holesky

To sync with hoodi

ethrex --network hoodi

Once started, you should be able to check the sync status with:

curl http://localhost:8545 \
    -H 'content-type: application/json' \
    -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'

The answer should be:

{"id":1,"jsonrpc":"2.0","result":{"startingBlock":"0x0","currentBlock":"0x0","highestBlock":"0x0"}}

Run an Ethereum node with Docker

You can simply start a node with a Consensus client and ethrex as Execution client with Docker using the docker-compose.yaml

curl -L -o docker-compose.yaml https://raw.githubusercontent.com/lambdaclass/ethrex/refs/heads/main/docker-compose.yaml
docker compose up

Or you can set a different network:

ETHREX_NETWORK=hoodi docker compose up

For more details and configuration options:

• Configuration
• CLI reference

Configuration

This page covers the basic configuration options for running an L1 node with ethrex. Full list of options can be found at the CLI reference

Sync Modes

Ethrex supports different sync modes for node operation:

• full: Downloads and verifies the entire chain.
• snap: Fast sync using state snapshots (recommended for most users).

Set the sync mode with:

ethrex --sync <mode>

File Locations

By default, ethrex stores its data in:

• Linux: ~/.local/share/ethrex
• macOS: ~/Library/Application Support/ethrex

You can change the data directory with:

ethrex --datadir <path>

Ports

Default ports used by ethrex:

• 8545: JSON-RPC (HTTP)
• 8551: Auth JSON-RPC
• 30303: P2P networking (TCP/UDP)
• 9090: Metrics

You can change ports with the corresponding flags: --http.port, --authrpc.port, --p2p.port, --discovery.port, --metrics.port.

All services listen on 0.0.0.0 by default, except for the auth RPC, which listens on 127.0.0.1. This can also be changed with flags (e.g., --http.addr).

Log Levels

Control log verbosity with:

ethrex --log.level <level>

Levels: error, warn, info (default), debug, trace

Dev Mode (Localnet)

For local development and testing, you can use dev mode:

ethrex --dev

This runs a local network with block production and no external peers. This network has a list of predefined accounts with funds for testing purposes.

Monitoring and Metrics

Ethrex exposes metrics in Prometheus format on port 9090 by default. But the easiest way to monitor your node is to use the provided Docker Compose stack, which includes Prometheus and Grafana preconfigured. For that we are currently using port 3701, this will match the default in the future but for now if running the containers we expected to have the ethrex metrics exposed on port 3701.

Quickstart: Monitoring Stack with Docker Compose

1. Clone the repository:

   git clone https://github.com/lambdaclass/ethrex.git
   cd ethrex/metrics
   

2. Start the monitoring stack:

   # Optional: if you have updated from a previous version, stop first the docker compose.
   # docker compose -f docker-compose-metrics.yaml -f docker-compose-metrics-l1.overrides.yaml down
   docker compose -f docker-compose-metrics.yaml -f docker-compose-metrics-l1.overrides.yaml up -d
   

Note: You might want to restart the docker containers in case of an update from a previous ethrex version to make sure the latest provisioned configurations are applied:

3. Run ethrex with metrics enabled:

   Make sure to start ethrex with the --metrics flag and set the port to 3701:

   ethrex --authrpc.jwtsecret ./secrets/jwt.hex --network hoodi --metrics --metrics.port 3701
   

This will launch Prometheus and Grafana, already set up to scrape ethrex metrics.

Note: We depend on ethereum-metrics-exporter for some key metrics to define variables on the Grafana dashboards. For it to work properly we need the consensus client to expose its RPC endpoints. For example if you are running lighthouse you may need to add --http and --http-address 0.0.0.0 flags to it before the dashboards pick up all metrics. This won't be needed in the near future

Logs

Ethrex logs are written to stdout by default. To enable file logging, you must specify the --log.dir argument, with this you'll be able to have Promtail collect the logs and send them to Grafana Loki for log visualization.

• Promtail Configuration: metrics/provisioning/promtail/promtail.yaml

The promtail configuration expects by default that logs are stored in ./logs (relative to the repo root). To correctly see the logs in Grafana, ensure that Promtail can access the logs directory:

• If running via Docker, ensure you map a volume to the log directory and pass --log.dir to the container.
• If running standalone, use --log.dir ./logs when running ethrex.

ethrex --log.dir ./logs ...

If you choose to use a different directory, you must set the ETHREX_LOGS_DIR environment variable when running the metrics stack to point to your custom logs directory.

ETHREX_LOGS_DIR=/path/to/your/logs docker compose -f docker-compose-metrics.yaml -f docker-compose-metrics-l1.overrides.yaml up

You can view the logs in Grafana by navigating to the logs row in our dashboard.

Running Docker Container Manually

If you run the ethrex Docker container manually (e.g., docker run ...) or use a custom docker-compose.yaml outside of this repository, you must ensure the logs are accessible to the monitoring stack.

The ethrex container writes logs to the directory specified by --log.dir. You should mount this directory to a location on your host machine that Promtail can access.

Example:

docker run -d \
  --name ethrex \
  -v $(pwd)/logs:/data/logs \
  ghcr.io/lambdaclass/ethrex:main \
  --datadir /data \
  --log.dir /data/logs

If you are using the provided monitoring stack in metrics/, it expects logs to be in the logs directory at the root of the repository (or ../logs relative to the metrics folder). Ensure your volume mount matches this expectation or update the Promtail volume configuration.

Accessing Metrics and Dashboards

• Prometheus: http://localhost:9091
• Grafana: http://localhost:3001
  • Default login: admin / admin
  • Prometheus is preconfigured as a data source
  • Example dashboards are included in the repo

Metrics from ethrex will be available at http://localhost:3701/metrics in Prometheus format if you followed step 3.

For detailed information on the provided Grafana dashboards, see our L1 Dashboard document.

Custom Configuration

Your ethrex setup may differ from the default configuration. Check your endpoints at provisioning/prometheus/prometheus_l1_sync_docker.yaml.

Also if you have a centralized Prometheus or Grafana setup, you can adapt the provided configuration files to fit your environment, or even stop the docker containers that run Prometheus and/or Grafana leaving only the additional ethereum-metrics-exporter running alongside ethrex to export the metrics to your existing monitoring stack.

docker compose -f docker-compose-metrics.yaml -f docker-compose-metrics-l1.overrides.yaml up -d ethereum-metrics-exporter 

For manual setup or more details, see the Prometheus documentation and Grafana documentation.

L1 Architecture

This section covers the internal architecture of ethrex as an Ethereum L1 execution client. It explains how the different components interact, how blocks flow through the system, and the design decisions behind the implementation.

• System Overview - High-level architecture and component interactions
• Block Execution Pipeline - How blocks are validated and executed
• Sync State Machine - Full sync and snap sync algorithms
• Crate Map - Overview of all crates and their responsibilities

System Overview

This document provides a high-level overview of ethrex's L1 architecture as an Ethereum execution client.

Introduction

ethrex is a Rust implementation of an Ethereum execution client. It implements the Ethereum protocol specification, including:

• Block validation and execution
• State management via Merkle Patricia Tries
• P2P networking (devp2p stack)
• JSON-RPC API for external interaction
• Engine API for consensus client communication

High-Level Architecture

                                    ┌─────────────────────┐
                                    │   Consensus Client  │
                                    │  (Lighthouse, etc)  │
                                    └──────────┬──────────┘
                                               │ Engine API
                                               │ (JWT auth)
                                               ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                              ethrex Execution Client                          │
│                                                                               │
│  ┌─────────────┐     ┌──────────────┐     ┌────────────────────────────────┐ │
│  │   JSON-RPC  │     │  Engine API  │     │           P2P Network          │ │
│  │    Server   │     │   Handler    │     │  ┌────────┐  ┌──────────────┐  │ │
│  │             │     │              │     │  │DiscV4  │  │    RLPx      │  │ │
│  │ eth_*       │     │ engine_*     │     │  │        │  │  ┌────────┐  │  │ │
│  │ debug_*     │     │ forkchoice   │     │  │        │  │  │ eth/68 │  │  │ │
│  │ txpool_*    │     │ newPayload   │     │  │        │  │  │ snap/1 │  │  │ │
│  │ admin_*     │     │ getPayload   │     │  │        │  │  └────────┘  │  │ │
│  └──────┬──────┘     └──────┬───────┘     │  └────────┘  └──────────────┘  │ │
│         │                   │             └────────────────┬───────────────┘ │
│         │                   │                              │                 │
│         └───────────────────┼──────────────────────────────┘                 │
│                             │                                                 │
│                             ▼                                                 │
│  ┌───────────────────────────────────────────────────────────────────────┐   │
│  │                           Blockchain                                   │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐   │   │
│  │  │   Mempool   │  │  Payload    │  │ Fork Choice │  │   Block     │   │   │
│  │  │             │  │  Builder    │  │   Update    │  │  Pipeline   │   │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘   │   │
│  └───────────────────────────────────────────────────────────────────────┘   │
│                             │                                                 │
│                             ▼                                                 │
│  ┌───────────────────────────────────────────────────────────────────────┐   │
│  │                              EVM (LEVM)                                │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐   │   │
│  │  │Transaction  │  │   Opcode    │  │  Precompiled│  │    State    │   │   │
│  │  │ Execution   │  │   Handler   │  │  Contracts  │  │ Transitions │   │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘   │   │
│  └───────────────────────────────────────────────────────────────────────┘   │
│                             │                                                 │
│                             ▼                                                 │
│  ┌───────────────────────────────────────────────────────────────────────┐   │
│  │                             Storage                                    │   │
│  │  ┌───────────────────────────────────────────────────────────────┐    │   │
│  │  │                     Store (High-level API)                     │    │   │
│  │  └───────────────────────────────────────────────────────────────┘    │   │
│  │                    │                              │                    │   │
│  │         ┌──────────┴──────────┐        ┌─────────┴────────┐           │   │
│  │         ▼                     ▼        ▼                  ▼           │   │
│  │  ┌─────────────┐       ┌─────────────────┐       ┌───────────────┐    │   │
│  │  │  InMemory   │       │    RocksDB      │       │  State Trie   │    │   │
│  │  │  (Testing)  │       │  (Production)   │       │ (MPT + Flat)  │    │   │
│  │  └─────────────┘       └─────────────────┘       └───────────────┘    │   │
│  └───────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────────┘

Core Components

1. Network Layer

The network layer handles all external communication:

JSON-RPC Server (crates/networking/rpc)

• Implements the Ethereum JSON-RPC specification
• Namespaces: eth_*, debug_*, txpool_*, admin_*, web3_*
• Validates and broadcasts incoming transactions

Engine API (crates/networking/rpc/engine)

• Communication channel with the consensus client
• Handles engine_forkchoiceUpdatedV{1,2,3}, engine_newPayloadV{1,2,3}, engine_getPayloadV{1,2,3}
• JWT authentication for security
• Triggers sync when receiving unknown block hashes

P2P Network (crates/networking/p2p)

• DiscV4: Node discovery protocol for finding peers
• RLPx: Encrypted transport layer for peer communication
• eth/68: Block and transaction propagation protocol
• snap/1: Snap sync protocol for fast state download

2. Blockchain Layer

The blockchain layer manages chain state and block processing:

Blockchain (crates/blockchain)

• Orchestrates block validation and execution
• Manages the mempool for pending transactions
• Handles fork choice updates from the consensus layer
• Coordinates payload building for block production

Mempool

• Stores pending transactions awaiting inclusion
• Filters transactions by gas price, nonce, and validity
• Supports transaction replacement (EIP-1559 and EIP-4844)
• Broadcasts new transactions to peers

Fork Choice

• Implements Ethereum's fork choice rule
• Updates the canonical chain based on consensus client signals
• Handles chain reorganizations

3. Execution Layer

LEVM (Lambda EVM) (crates/vm/levm)

• Custom EVM implementation in Rust
• Executes smart contract bytecode
• Implements all EVM opcodes up to the latest hard fork
• Handles precompiled contracts

Block Execution Pipeline

1. Validate block header
2. Apply system-level operations (beacon root, block hash storage)
3. Execute transactions in order
4. Process withdrawals (post-Merge)
5. Extract requests (post-Prague)
6. Compute state root and verify against header

4. Storage Layer

Store (crates/storage)

• High-level API for all blockchain data
• Supports multiple backends: InMemory (testing), RocksDB (production)
• Manages block headers, bodies, receipts, and state

State Trie (crates/common/trie)

• Merkle Patricia Trie implementation
• Stores account states and contract storage
• Supports flat key-value storage for performance
• Handles trie node caching and persistence

Data Flow

Block Import (from P2P)

P2P Peer → Block Headers/Bodies → Syncer → Blockchain.add_block() → EVM.execute() → Store

1. Syncer requests headers from peers
2. Headers are validated (parent exists, timestamps, gas limits, etc.)
3. Bodies are requested and matched to headers
4. Blocks are executed in batches
5. State is committed to storage

Block Import (from Consensus Client)

Consensus Client → engine_newPayloadV3 → Blockchain.add_block_pipeline() → EVM.execute() → Store
                 → engine_forkchoiceUpdated → Fork Choice Update → Canonical Chain Update

1. Consensus client sends new payload via Engine API
2. Block is validated and executed
3. Fork choice update makes the block canonical
4. Sync is triggered if the block's parent is unknown

Transaction Lifecycle

User → JSON-RPC (eth_sendRawTransaction) → Mempool → Broadcast to Peers
                                                   → Include in Block

1. Transaction arrives via JSON-RPC or P2P
2. Validated for signature, nonce, balance, gas
3. Added to mempool if valid
4. Broadcast to connected peers
5. Eventually included in a block by the payload builder

Sync Modes

Full Sync

Downloads and executes every block from genesis (or a known checkpoint):

1. Request block headers from peers
2. Request block bodies for each header
3. Execute blocks in batches (1024 blocks per batch)
4. Commit state after each batch
5. Update fork choice when sync head is reached

Snap Sync

Downloads state directly instead of executing all historical blocks:

1. Download block headers to find a recent "pivot" block
2. Download account state trie leaves via snap protocol
3. Download storage tries for accounts with storage
4. Heal any missing trie nodes (state may have changed during download)
5. Download bytecode for contract accounts
6. Execute recent blocks (post-pivot) to catch up

See Sync State Machine for detailed documentation.

Concurrency Model

ethrex uses Tokio for async I/O with the following patterns:

• Async tasks for network I/O (RPC, P2P)
• Blocking tasks for CPU-intensive work (block execution, trie operations)
• Channels for inter-component communication (sync signals, mempool updates)
• RwLock/Mutex for shared state (mempool, peer table)

Configuration

Key configuration options:

See Configuration for the complete reference.

Next Steps

• Block Execution Pipeline - Deep dive into block processing
• Sync State Machine - Detailed sync algorithm documentation
• Crate Map - Overview of all crates and dependencies

Block Execution Pipeline

This document describes how ethrex validates and executes blocks, from receiving a block to committing state changes.

Overview

Block execution in ethrex follows the Ethereum specification closely. The pipeline handles:

1. Block header validation
2. System-level operations (beacon root contract, block hash storage)
3. Transaction execution
4. Withdrawal processing
5. Request extraction (post-Prague)
6. State root verification

Entry Points

Blocks enter the execution pipeline through two main paths:

1. P2P Sync (Syncer)

During synchronization, blocks are fetched from peers and executed in batches:

#![allow(unused)]
fn main() {
// crates/networking/p2p/sync.rs
Syncer::add_blocks() → Blockchain::add_blocks_in_batch() → execute each block
}

2. Engine API (engine_newPayloadV{1,2,3})

Post-Merge, the consensus client sends new blocks via the Engine API:

#![allow(unused)]
fn main() {
// crates/networking/rpc/engine/payload.rs
NewPayloadV3::handle() → Blockchain::add_block() → execute block
}

Block Header Validation

Before executing a block, its header is validated:

#![allow(unused)]
fn main() {
// crates/blockchain/blockchain.rs
fn validate_header(header: &BlockHeader, parent: &BlockHeader) -> Result<()>
}

Validation Checks

Execution Flow

┌─────────────────────────────────────────────────────────────────────┐
│                        Block Execution                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  1. ┌────────────────────────────────────────────────────────────┐  │
│     │             System Operations (post-Cancun)                 │  │
│     │  • Store beacon block root (EIP-4788)                       │  │
│     │  • Store parent block hash (EIP-2935)                       │  │
│     └────────────────────────────────────────────────────────────┘  │
│                              │                                       │
│                              ▼                                       │
│  2. ┌────────────────────────────────────────────────────────────┐  │
│     │              Transaction Execution                          │  │
│     │  For each transaction:                                      │  │
│     │  • Validate signature and nonce                             │  │
│     │  • Check sender balance                                     │  │
│     │  • Execute in EVM                                           │  │
│     │  • Apply gas refunds                                        │  │
│     │  • Update account states                                    │  │
│     │  • Generate receipt                                         │  │
│     └────────────────────────────────────────────────────────────┘  │
│                              │                                       │
│                              ▼                                       │
│  3. ┌────────────────────────────────────────────────────────────┐  │
│     │              Withdrawal Processing (post-Shanghai)          │  │
│     │  For each withdrawal:                                       │  │
│     │  • Credit validator address with withdrawal amount          │  │
│     └────────────────────────────────────────────────────────────┘  │
│                              │                                       │
│                              ▼                                       │
│  4. ┌────────────────────────────────────────────────────────────┐  │
│     │              Request Extraction (post-Prague)               │  │
│     │  • Deposit requests from logs                               │  │
│     │  • Withdrawal requests from system contract                 │  │
│     │  • Consolidation requests from system contract              │  │
│     └────────────────────────────────────────────────────────────┘  │
│                              │                                       │
│                              ▼                                       │
│  5. ┌────────────────────────────────────────────────────────────┐  │
│     │                  State Finalization                         │  │
│     │  • Compute state root from account updates                  │  │
│     │  • Verify against header.state_root                         │  │
│     │  • Commit changes to storage                                │  │
│     └────────────────────────────────────────────────────────────┘  │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Transaction Execution

Each transaction goes through the following steps:

1. Pre-Execution Validation

#![allow(unused)]
fn main() {
// crates/blockchain/validate.rs
fn validate_transaction(tx: &Transaction, header: &BlockHeader) -> Result<()>
}

• Signature recovery and validation
• Nonce check (must match account nonce)
• Gas limit check (must be <= block gas remaining)
• Balance check (must cover gas_limit * gas_price + value)
• Intrinsic gas calculation
• EIP-2930 access list validation
• EIP-4844 blob validation (if applicable)

2. EVM Execution

#![allow(unused)]
fn main() {
// crates/vm/levm/src/vm.rs
VM::execute() → Result<ExecutionReport>
}

The EVM executes the transaction bytecode:

1. Contract Call: Execute target contract code
2. Contract Creation: Deploy new contract, execute constructor
3. Transfer: Simple value transfer (no code execution)

During execution:

• Opcodes are decoded and executed
• Gas is consumed for each operation
• State changes are tracked (but not committed)
• Logs are collected
• Errors revert all changes

3. Post-Execution

After EVM execution:

#![allow(unused)]
fn main() {
// crates/vm/levm/src/vm.rs
fn finalize_transaction() -> Receipt
}

• Calculate gas refund (max 1/5 of gas used, post-London)
• Credit coinbase with priority fee
• Generate receipt with logs and status
• Update cumulative gas used

State Management

Account Updates

State changes are tracked as AccountUpdate structs:

#![allow(unused)]
fn main() {
pub struct AccountUpdate {
    pub address: Address,
    pub removed: bool,
    pub info: Option<AccountInfo>,       // balance, nonce, code_hash
    pub code: Option<Bytes>,             // bytecode if changed
    pub added_storage: HashMap<H256, U256>,
}
}

State Root Computation

After all transactions execute:

#![allow(unused)]
fn main() {
// crates/storage/store.rs
Store::apply_account_updates_batch(parent_hash, updates) -> StateTrieHash
}

This is one of the two merkleization backends (the other is used by add_block_pipeline):

1. Load parent state trie
2. Apply each account update to the trie
3. For accounts with storage changes, update storage tries
4. Compute new state root
5. Verify it matches header.state_root

Payload Building

When ethrex acts as a block producer (validator), it builds payloads:

#![allow(unused)]
fn main() {
// crates/blockchain/payload.rs
Blockchain::build_payload(template: Block) -> PayloadBuildResult
}

Building Process

1. Fetch transactions from mempool, filtered by:

   • Base fee (must afford current base fee)
   • Blob fee (for EIP-4844 transactions)
   • Nonce ordering (consecutive nonces per sender)

2. Order transactions by effective tip (highest first)

3. Execute transactions until:

   • Block gas limit reached
   • No more valid transactions
   • Blob limit reached (for blob transactions)

4. Finalize block:

   • Apply withdrawals
   • Extract requests
   • Compute state root
   • Compute receipts root
   • Generate logs bloom

Payload Rebuilding

Payloads are rebuilt continuously until requested:

#![allow(unused)]
fn main() {
// crates/blockchain/payload.rs
Blockchain::build_payload_loop(payload, cancel_token)
}

This maximizes MEV by including the most profitable transactions available.

Error Handling

Block execution can fail for various reasons:

Performance Considerations

Batch Execution

During sync, blocks are executed in batches (default 1024 blocks):

#![allow(unused)]
fn main() {
// crates/networking/p2p/sync.rs
const EXECUTE_BATCH_SIZE: usize = 1024;
}

This reduces database commits and improves throughput.

Parallel Trie Operations

Storage trie updates can be parallelized across accounts:

#![allow(unused)]
fn main() {
// Uses rayon for parallel iteration
account_updates.par_iter().map(|update| update_storage_trie(update))
}

State Caching

The EVM maintains a cache of accessed accounts and storage slots to minimize database reads during execution.

Hard Fork Handling

Block execution adapts based on the active hard fork:

#![allow(unused)]
fn main() {
// crates/common/types/chain_config.rs
impl ChainConfig {
    pub fn fork(&self, timestamp: u64) -> Fork { ... }
    pub fn is_cancun_activated(&self, timestamp: u64) -> bool { ... }
    pub fn is_prague_activated(&self, timestamp: u64) -> bool { ... }
}
}

Each fork may introduce:

• New opcodes (e.g., PUSH0 in Shanghai)
• New precompiles (e.g., point evaluation in Cancun)
• New system contracts (e.g., beacon root contract in Cancun)
• Changed gas costs
• New transaction types

Related Documentation

• LEVM Documentation - EVM implementation details
• Sync State Machine - How blocks flow during sync
• Crate Map - Overview of involved crates

Sync State Machine

This document describes the synchronization algorithms implemented in ethrex, including full sync and snap sync.

Overview

ethrex supports two synchronization modes:

Sync Manager Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        SyncManager                               │
│  • Receives sync targets from Engine API / P2P                   │
│  • Tracks current sync mode (Full / Snap)                        │
│  • Coordinates Syncer for actual sync work                       │
└──────────────────────────────┬──────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────┐
│                          Syncer                                  │
│  • Executes sync cycles                                          │
│  • Manages peer connections via PeerHandler                      │
│  • Handles both full and snap sync algorithms                    │
└─────────────────────────────────────────────────────────────────┘

Sync Triggers

Synchronization is triggered by:

1. Engine API: engine_forkchoiceUpdated with unknown head hash
2. P2P: Receiving block announcements for unknown blocks
3. Startup: When local chain is behind network

#![allow(unused)]
fn main() {
// crates/networking/rpc/engine/fork_choice.rs
match apply_fork_choice(...) {
    Err(InvalidForkChoice::Syncing) => {
        syncer.sync_to_head(fork_choice_state.head_block_hash);
        // Return SYNCING status to consensus client
    }
}
}

Full Sync Algorithm

Full sync downloads blocks from the network and executes each one to reconstruct the state.

State Machine

                    ┌─────────────────┐
                    │   START SYNC    │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
         ┌─────────│  Request Headers │◄─────────────┐
         │         └────────┬────────┘              │
         │                  │                        │
         │                  ▼                        │
         │         ┌─────────────────┐              │
         │         │ Validate Headers│              │
         │         └────────┬────────┘              │
         │                  │                        │
         │                  ▼                        │
         │         ┌─────────────────┐              │
         │         │ Found Canonical │──No──────────┘
         │         │   Ancestor?     │
         │         └────────┬────────┘
         │                  │ Yes
         │                  ▼
         │         ┌─────────────────┐
         │         │  Request Bodies │◄─────────────┐
         │         └────────┬────────┘              │
         │                  │                        │
         │                  ▼                        │
         │         ┌─────────────────┐              │
         │         │ Execute Batch   │              │
         │         │ (1024 blocks)   │              │
         │         └────────┬────────┘              │
         │                  │                        │
         │                  ▼                        │
         │         ┌─────────────────┐              │
         │         │  More Blocks?   │──Yes─────────┘
         │         └────────┬────────┘
         │                  │ No
         │                  ▼
         │         ┌─────────────────┐
         └─Error───│   SYNC DONE     │
                   └─────────────────┘

Algorithm Details

#![allow(unused)]
fn main() {
// crates/networking/p2p/sync.rs
async fn sync_cycle_full(sync_head: H256, store: Store) -> Result<()>
}

1. Find Chain Link

   • Request headers backwards from sync_head
   • Stop when reaching a canonical block (already known)
   • This identifies the fork point

2. Store Headers

   • Save all new headers to temporary storage
   • Headers are stored in batches during download

3. Download Bodies

   • Request bodies for stored headers
   • Match bodies to headers by hash
   • Maximum 64 bodies per request

4. Execute Blocks

   • Execute in batches of 1024 blocks
   • Each block is fully validated and executed
   • State is committed after each batch

5. Update Fork Choice

   • After all blocks executed, update canonical chain
   • Set new head, safe, and finalized blocks

Key Constants

#![allow(unused)]
fn main() {
const EXECUTE_BATCH_SIZE: usize = 1024;      // Blocks per execution batch
const MAX_BLOCK_BODIES_TO_REQUEST: usize = 64; // Bodies per request
}

Snap Sync Algorithm

Snap sync downloads state directly from peers instead of executing all historical blocks.

State Machine

┌─────────────────────────────────────────────────────────────────────────────┐
│                           SNAP SYNC STATE MACHINE                            │
└─────────────────────────────────────────────────────────────────────────────┘

    ┌──────────────┐
    │  START SNAP  │
    │    SYNC      │
    └──────┬───────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │   Download   │     │  Download headers to find sync head                  │
    │   Headers    │────▶│  Store hashes for later body download               │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │ Select Pivot │────▶│  Choose recent block as pivot (must not be stale)   │
    │    Block     │     │  Pivot block is target for state download           │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │  Download    │────▶│  Request account ranges via SNAP protocol           │
    │  Accounts    │     │  Store account states to disk as snapshots          │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │   Insert     │────▶│  Build account trie from downloaded leaves          │
    │  Accounts    │     │  Identify accounts with non-empty storage           │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │  Download    │────▶│  For each account with storage:                     │
    │  Storage     │     │  Request storage ranges and build storage tries.    │
    │              │     │  Includes a healing loop to fix state trie changes. │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │    Heal      │────▶│  Heal state trie (fill missing nodes)               │
    │    Tries     │     │  Heal storage tries for modified accounts           │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │  Download    │────▶│  Download bytecode for all contract accounts        │
    │  Bytecode    │     │  Match by code hash                                 │
    └──────┬───────┘     └─────────────────────────────────────────────────────┘
           │
           ▼
    ┌──────────────┐
    │  SNAP SYNC   │
    │   COMPLETE   │
    └──────┬───────┘
           │
           ▼
    ┌──────────────┐     ┌─────────────────────────────────────────────────────┐
    │   Switch to  │────▶│  Execute recent blocks from pivot to head           │
    │  Full Sync   │     │  Continue with full sync for new blocks             │
    └──────────────┘     └─────────────────────────────────────────────────────┘

Phase 1: Header Download

Download all block headers from current head to sync target:

#![allow(unused)]
fn main() {
// crates/networking/p2p/sync.rs
async fn sync_cycle_snap(sync_head: H256, store: Store) -> Result<()>
}

• Request headers in batches
• Store header hashes for later use
• Identify pivot block (recent block whose state we'll download)

Phase 2: Pivot Selection

The pivot block must be:

• Recent enough to have state available on peers
• Not "stale" (older than SNAP_LIMIT * 12 seconds)

#![allow(unused)]
fn main() {
// crates/networking/p2p/sync.rs
fn block_is_stale(header: &BlockHeader) -> bool {
    calculate_staleness_timestamp(header.timestamp) < current_unix_time()
}

const SNAP_LIMIT: usize = 128; // Blocks before pivot is considered stale
}

If the pivot becomes stale during sync, a new pivot is selected:

#![allow(unused)]
fn main() {
async fn update_pivot(block_number: u64, ...) -> Result<BlockHeader>
}

Phase 3: Account Download

Download all account states at the pivot block:

#![allow(unused)]
fn main() {
// Uses SNAP protocol GetAccountRange messages
peers.request_account_range(start_hash, end_hash, snapshot_dir, pivot_header)
}

• Accounts are saved to disk as RLP-encoded snapshots
• Each snapshot file contains a batch of (hash, account_state) pairs
• Process tracks code hashes for later bytecode download

Phase 4: Account Trie Construction

Build the account state trie from downloaded leaves:

#![allow(unused)]
fn main() {
async fn insert_accounts(store, storage_accounts, snapshots_dir, ...) -> (H256, accounts_with_storage)
}

For RocksDB backend:

• Ingest snapshot files directly via SST ingestion
• Build trie using sorted insertion algorithm
• Track accounts with non-empty storage root

Phase 5: Storage Download

For each account with storage, download storage slots:

#![allow(unused)]
fn main() {
peers.request_storage_ranges(storage_accounts, snapshots_dir, chunk_index, pivot_header)
}

• Multiple accounts can be requested per message
• Large accounts are downloaded in chunks
• "Big accounts" (>4096 slots) are marked for healing instead

Phase 6: Trie Healing

State may have changed while downloading. Healing fixes inconsistencies:

State Trie Healing:

#![allow(unused)]
fn main() {
async fn heal_state_trie_wrap(state_root, store, peers, deadline, ...) -> bool
}

• Walk trie from root
• Request missing nodes from peers
• Fill in gaps caused by state changes

Storage Trie Healing:

#![allow(unused)]
fn main() {
async fn heal_storage_trie(state_root, accounts, peers, store, ...) -> bool
}

• For each account marked for healing
• Request missing storage trie nodes
• Verify storage roots match account state

Phase 7: Bytecode Download

Download contract bytecode:

#![allow(unused)]
fn main() {
peers.request_bytecodes(&code_hashes)
}

• Code hashes collected during account download
• Bytecode downloaded in chunks (50,000 per batch)
• Verified by hashing and comparing to code_hash

Phase 8: Transition to Full Sync

After snap sync completes:

1. Store pivot block body
2. Update fork choice to pivot
3. Switch sync mode to Full
4. Execute any remaining blocks normally

P2P Protocols Used

eth/68 Protocol

Used for block header and body download:

snap/1 Protocol

Used for state download during snap sync:

Error Recovery

Recoverable Errors

These errors cause sync to retry:

• Peer disconnection
• Invalid response from peer
• Timeout waiting for response
• Database errors (transient)

Non-Recoverable Errors

These errors cause sync to abort with warning:

• Snapshot file corruption
• Database corruption
• State root mismatch after healing

#![allow(unused)]
fn main() {
// crates/networking/p2p/sync.rs
impl SyncError {
    pub fn is_recoverable(&self) -> bool {
        match self {
            SyncError::Chain(_) | SyncError::Store(_) | ... => true,
            SyncError::CorruptDB | SyncError::SnapshotDecodeError(_) | ... => false,
        }
    }
}
}

Performance Optimizations

Parallel Operations

• Account trie insertion uses Rayon for parallelism
• Storage tries built in parallel across accounts
• Bytecode downloads are batched

Disk I/O

• Snapshot files written in batches to reduce writes
• RocksDB SST ingestion for fast account loading
• Temporary directories cleaned up after sync

Network

• Multiple peers used concurrently
• Peer scoring based on response time and validity
• Automatic peer rotation for failed requests

Metrics

Sync progress is tracked via metrics:

#![allow(unused)]
fn main() {
// crates/networking/p2p/metrics.rs
METRICS.account_tries_inserted     // Accounts added to trie
METRICS.storage_leaves_inserted    // Storage slots added
METRICS.current_step               // Current sync phase
METRICS.sync_head_hash             // Current sync target
}

Configuration

Related Documentation

• Snap Sync Internals - Detailed snap sync documentation
• Block Execution Pipeline - How blocks are executed
• Networking - P2P protocol details

Note: For comprehensive snap sync documentation, see Snap Sync Internals.

Crate Map

This document provides an overview of all crates in the ethrex monorepo and their responsibilities.

Crate Dependency Graph

                              ┌─────────────────────────────────────┐
                              │           cmd/ethrex                │
                              │      (Main binary entry point)      │
                              └───────────────┬─────────────────────┘
                                              │
                    ┌─────────────────────────┼─────────────────────────┐
                    │                         │                         │
                    ▼                         ▼                         ▼
        ┌───────────────────┐     ┌───────────────────┐     ┌───────────────────┐
        │  networking/rpc   │     │  networking/p2p   │     │    blockchain     │
        │   (JSON-RPC API)  │     │  (P2P networking) │     │ (Chain management)│
        └─────────┬─────────┘     └─────────┬─────────┘     └─────────┬─────────┘
                  │                         │                         │
                  │                         │                         │
                  └─────────────────────────┼─────────────────────────┘
                                            │
                                            ▼
                              ┌─────────────────────────────┐
                              │           vm/levm           │
                              │    (EVM implementation)     │
                              └─────────────┬───────────────┘
                                            │
                                            ▼
                              ┌─────────────────────────────┐
                              │          storage            │
                              │     (Data persistence)      │
                              └─────────────┬───────────────┘
                                            │
                    ┌───────────────────────┼───────────────────────┐
                    │                       │                       │
                    ▼                       ▼                       ▼
        ┌───────────────────┐   ┌───────────────────┐   ┌───────────────────┐
        │    common/trie    │   │    common/rlp     │   │   common/types    │
        │ (Merkle Patricia) │   │ (RLP encoding)    │   │ (Core data types) │
        └───────────────────┘   └───────────────────┘   └───────────────────┘

Core Crates

ethrex-common

Purpose: Core data types and utilities shared across all crates.

Key Modules:

• types/ - Block, Transaction, Receipt, Account types
• trie/ - Merkle Patricia Trie implementation
• rlp/ - RLP encoding/decoding
• crypto/ - Keccak hashing, signature recovery

Notable Types:

#![allow(unused)]
fn main() {
pub struct Block { header: BlockHeader, body: BlockBody }
pub struct Transaction { /* variants for Legacy, EIP-2930, EIP-1559, EIP-4844, EIP-7702 */ }
pub struct AccountState { nonce: u64, balance: U256, storage_root: H256, code_hash: H256 }
}

ethrex-storage

Purpose: Persistent storage layer with multiple backend support.

Key Components:

• Store - High-level API for all blockchain data
• StoreEngine trait - Backend abstraction
• InMemoryStore - Testing backend
• RocksDBStore - Production backend

Stored Data:

ethrex-blockchain

Purpose: Chain management, block validation, and mempool.

Key Components:

• Blockchain - Main orchestrator for chain operations
• Mempool - Pending transaction pool
• fork_choice - Fork choice rule implementation
• payload - Block building for validators
• validate - Block and transaction validation

Public API:

#![allow(unused)]
fn main() {
impl Blockchain {
    pub fn add_block(&self, block: Block) -> Result<(), ChainError>
    pub fn add_block_pipeline(&self, block: Block) -> Result<(), ChainError>
    pub fn validate_transaction(&self, tx: &Transaction) -> Result<(), MempoolError>
    pub fn build_payload(&self, template: Block) -> Result<PayloadBuildResult, ChainError>
    pub fn get_payload(&self, id: u64) -> Result<PayloadBuildResult, ChainError>
}
}

ethrex-vm / levm

Purpose: Ethereum Virtual Machine implementation.

Key Components:

• VM - Main EVM execution engine
• Evm trait - VM interface for different contexts
• Opcode handlers (one per EVM opcode)
• Precompiled contracts
• Gas metering

Execution Flow:

#![allow(unused)]
fn main() {
impl VM {
    pub fn execute(&mut self) -> Result<ExecutionReport, VMError>
    fn execute_opcode(&mut self, opcode: u8) -> Result<(), VMError>
    fn call(&mut self, ...) -> Result<CallOutcome, VMError>
    fn create(&mut self, ...) -> Result<CreateOutcome, VMError>
}
}

ethrex-networking/rpc

Purpose: JSON-RPC API server.

Supported Namespaces:

• eth_* - Standard Ethereum methods
• debug_* - Debugging and tracing
• txpool_* - Mempool inspection
• admin_* - Node administration
• engine_* - Consensus client communication
• web3_* - Web3 utilities

Architecture:

#![allow(unused)]
fn main() {
pub trait RpcHandler: Send + Sync {
    fn parse(params: &Option<Vec<Value>>) -> Result<Self, RpcErr>;
    async fn handle(&self, context: RpcApiContext) -> Result<Value, RpcErr>;
}
}

ethrex-networking/p2p

Purpose: Peer-to-peer networking stack.

Protocol Layers:

1. DiscV4 - Node discovery
2. RLPx - Encrypted transport
3. eth/68 - Ethereum wire protocol
4. snap/1 - Snap sync protocol

Key Components:

• PeerHandler - Manages peer connections
• PeerTable - Tracks known peers and their scores
• Syncer - Synchronization state machine
• SyncManager - Coordinates sync operations

Supporting Crates

ethrex-common/trie

Purpose: Merkle Patricia Trie implementation.

Features:

• Standard MPT operations (get, insert, delete)
• Proof generation and verification
• Sorted insertion for snap sync
• Flat key-value store integration

ethrex-common/rlp

Purpose: Recursive Length Prefix encoding.

Traits:

#![allow(unused)]
fn main() {
pub trait RLPEncode {
    fn encode(&self, buf: &mut dyn BufMut);
    fn encode_to_vec(&self) -> Vec<u8>;
}

pub trait RLPDecode: Sized {
    fn decode(rlp: &[u8]) -> Result<Self, RLPDecodeError>;
    fn decode_unfinished(rlp: &[u8]) -> Result<(Self, &[u8]), RLPDecodeError>;
}
}

ethrex-metrics

Purpose: Prometheus metrics collection.

Metric Categories:

• Block metrics (height, gas, execution time)
• Transaction metrics (types, counts, errors)
• P2P metrics (peers, messages, sync progress)
• RPC metrics (requests, latency)

ethrex-crypto

Purpose: Cryptographic primitives.

Features:

• Keccak-256 hashing
• ECDSA signature recovery
• BLS signatures (for beacon chain)

L2-Specific Crates

ethrex-l2

Purpose: L2 sequencer and prover integration.

Components:

• Sequencer logic
• State diff computation
• Prover interface
• L1 interaction (deposits, withdrawals)

ethrex-prover

Purpose: Zero-knowledge proof generation.

Supported Provers:

• SP1 (Succinct)
• RISC0
• TDX (Trusted Execution)

Test and Development Crates

ef-tests

Purpose: Ethereum Foundation test runner.

Runs official Ethereum tests to verify protocol compliance.

ethrex-dev

Purpose: Development mode utilities.

Features:

• Local development network
• Block import from files
• Test fixtures

Crate Features

Many crates support feature flags:

Adding New Functionality

When adding new features, consider:

1. Where does it belong?

   • Pure data types → ethrex-common
   • Database operations → ethrex-storage
   • EVM changes → ethrex-vm
   • Chain logic → ethrex-blockchain
   • API endpoints → ethrex-networking/rpc
   • P2P messages → ethrex-networking/p2p

2. Dependency direction

   • Lower-level crates should not depend on higher-level ones
   • Common types flow down, behaviors flow up

3. Testing

   • Unit tests in the crate
   • Integration tests in tests/ directory
   • EF tests for protocol compliance

Related Documentation

• System Overview - How crates work together
• Block Execution - Execution flow across crates
• Sync State Machine - Sync implementation details

Fundamentals

This section covers the core concepts and technical details behind ethrex as an Ethereum execution client. Here you'll find explanations about sync modes, networking, databases, security, and more.

Databases

Ethrex uses a versioning system to ensure we don't run on invalid data if we restart the node after a breaking change to the DB structure. This system consists of a STORE_SCHEMA_VERSION constant, defined in crates/storage/lib.rs that must be increased after any breaking change and that is checked every time we start the node.

Networking

The network crate handles the ethereum networking protocols. This involves:

• Discovery protocol: built on top of udp and it is how we discover new nodes.
• devP2P: sits on top of tcp and is where the actual blockchain information exchange happens.

Implementation follows the official spec which can be found here. Also, we've been inspired by some geth code.

Discovery protocol

In the next section, we'll be looking at the discovery protocol (discv4 to be more specific) and the way we have it set up. There are many points for improvement and here we discuss some possible solutions to them.

At startup, the discovery server launches three concurrent tokio tasks:

• The listen loop for incoming requests.
• A revalidation loop to ensure peers remain responsive.
• A recursive lookup loop to request new peers and keep our table filled.

Before starting these tasks, we run a startup process to connect to an array of initial nodes.

Before diving into what each task does, first, we need to understand how we are storing our nodes. Nodes are stored in an in-memory matrix which we call a Kademlia table, though it isn't really a Kademlia table as we don't thoroughly follow the spec but we take it as a reference, you can read more here. This table holds:

• Our node_id: The node's unique identifier computed by obtaining the keccak hash of the 64 bytes starting from index 1 of the encoded pub key.
• A vector of 256 buckets which holds:
  • peers: a vector of 16 elements of type PeersData where we save the node record and other related data that we'll see later.
  • replacements: a vector of 16 elements of PeersData that are not connected to us, but we consider them as potential replacements for those nodes that have disconnected from us.

Peers are not assigned to any bucket but they are assigned based on its $0\le \text{distance}\le 255$ to our node_id. Distance is defined by:

#![allow(unused)]
fn main() {
pub fn distance(node_id_1: H512, node_id_2: H512) -> usize {
    let xor = node_id_1 ^ node_id_2;
    let distance = U256::from_big_endian(xor.as_bytes());
    distance.bits().saturating_sub(1)
}
}

Startup

Before starting the server, we do a startup where we connect to an array of seeders or bootnodes. This involves:

• Receiving bootnodes via CLI params
• Inserting them into our table
• Pinging them to notify our presence, so they acknowledge us.

This startup is far from being completed. The current state allows us to do basic tests and connections. Later, we want to do a real startup by first trying to connect to those nodes we were previously connected. For that, we'd need to store nodes on the database. If those nodes aren't enough to fill our table, then we also ping some bootnodes, which could be hardcoded or received through the cli. Current issues are opened regarding startup and nodes db.

Listen loop

The listen loop handles messages sent to our socket. The spec defines 6 types of messages:

• Ping: Responds with a pong message. If the peer is not in our table we add it, if the corresponding bucket is already filled then we add it as a replacement for that bucket. If it was inserted we send a ping from our end to get an endpoint proof.
• Pong: Verifies that the pong corresponds to a previously sent ping, if so we mark the peer as proven.
• FindNodes: Responds with a neighbors message that contains as many as the 16 closest nodes from the given target. A target is a pubkey provided by the peer in the message. The response can't be sent in one packet as it might exceed the discv4 max packet size. So we split it into different packets.
• Neighbors: First we verify that we have sent the corresponding find_node message. If so, we receive the peers, store them, and ping them. Also, every find_node request may have a tokio Sender attached, if that is the case, we forward the nodes from the message through the channel. This becomes useful when waiting for a find_node response, something we do in the lookups.
• ENRRequest: currently not implemented see here.
• ENRResponse: same as above.

Re-validations

Re-validations are tasks that are implemented as intervals, that is: they run an action every x whichever unit of time (currently configured to run every 30 seconds). The current flow of re-validation is as follows

1. Every 30 seconds (by default) we ping the three least recently pinged peers: this may be fine now to keep simplicity, but we might prefer to choose three random peers instead to avoid the search which might become expensive as our buckets start to fill with more peers.
2. In the next iteration we check if they have answered
   • if they have: we increment the liveness field by one.
   • otherwise: we decrement the liveness by a third of its value.
3. If the liveness field is 0, we delete it and insert a new one from the replacements table.

Liveness checks are not part of the spec but are taken from geth, see here. This field is useful because it provides us with good criteria of which nodes are connected and we "trust" more. This trustiness is useful when deciding if we want to store this node in the database to use it as a future seeder or when establishing a connection in p2p.

Re-validations are another point of potential improvement. While it may be fine for now to keep simplicity at max, pinging the last recently pinged peers becomes quite expensive as the number of peers in the table increases. And it also isn't very "just" in selecting nodes so that they get their liveness increased so we trust them more and we might consider them as a seeder. A possible improvement could be:

• Keep two lists: one for nodes that have already been pinged, and another one for nodes that have not yet been revalidated. Let's call the former "a" and the second "b".
• In the beginning, all nodes would belong to "a" and whenever we insert a new node, they would be pushed to "a".
• We would have two intervals: one for pinging "a" and another for pinging to nodes in "b". The "b" would be quicker, as no initial validation has been done.
• When picking a node to ping, we would do it randomly, which is the best form of justice for a node to become trusted by us.
• When a node from b responds successfully, we move it to a, and when one from a does not respond, we move it to b.

This improvement follows somewhat what geth does, see here.

Recursive Lookups

Recursive lookups are as with re-validations implemented as intervals. Their current flow is as follows:

1. Every 30min we spawn three concurrent lookups: one closest to our pubkey and two others closest to randomly generated pubkeys.
2. Every lookup starts with the closest nodes from our table. Each lookup keeps track of:
   • Peers that have already been asked for nodes
   • Peers that have been already seen
   • Potential peers to query for nodes: a vector of up to 16 entries holding the closest peers to the pubkey. This vector is initially filled with nodes from our table.
3. We send a find_node to the closest 3 nodes (that we have not yet asked) from the pubkey.
4. We wait for the neighbors' response and push or replace those who are closer to the potential peers.
5. We select three other nodes from the potential peers vector and do the same until one lookup has no node to ask.

The way to do lookups isn't part of the spec. Our implementation aligns with geth approach, see here.

An example of how you might build a network

Finally, here is an example of how you could build a network and see how they connect to each other:

We'll have three nodes: a, b, and c, we'll start a, then b setting a as a bootnode, and finally we'll start c with b as bootnode we should see that c connects to both a and b and so all the network should be connected.

node a:

cargo run --release -- --network ./fixtures/genesis/kurtosis.json

We get the enode by querying the node_info and using jq:

curl -s http://localhost:8545 \
-X POST \
-H "Content-Type: application/json" \
--data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
| jq '.result.enode'

node b:

We start a new server passing the enode from node a as an argument. Also changing the database dir and the ports is needed to avoid conflicts.

cargo run --release -- --network ./fixtures/genesis/kurtosis.json --bootnodes=`NODE_A_ENODE` \
--datadir=ethrex_b --authrpc.port=8552 --http.port=8546 --p2p.port=30305 --discovery.port=30306

node c Finally, with node_c we connect to node_b. When the lookup runs, node_c should end up connecting to node_a:

cargo run --release -- --network ./fixtures/genesis/kurtosis.json --bootnodes=`NODE_B_ENODE` \
--datadir=ethrex_c --authrpc.port=8553 --http.port=8547 --p2p.port=30308 --discovery.port=30310

We get the enode by querying the node_info and using jq:

curl -s http://localhost:8546 \
-X POST \
-H "Content-Type: application/json" \
--data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
| jq '.result.enode'

You could also spawn nodes from other clients and it should work as well.

Sync Modes

Full sync

Full syncing works by downloading and executing every block from genesis. This means that full syncing will only work for networks that started after The Merge, as ethrex only supports post merge execution.

Snap sync

For snap sync, you can view the main document here.

Snap Sync

API reference: https://github.com/ethereum/devp2p/blob/master/caps/snap.md

Terminology:

• Peers: Other Ethereum execution clients we are connected to, and which can respond to snap requests.
• Pivot: The block we have chosen to snap-sync to. The pivot block changes continually as it becomes too old, because nodes don't serve old data (i.e. more than 128 blocks in the past). Read below for more details on this.

Concept

What

Executing all blocks to rebuild the state is slow. It is also not possible on ethrex, because we don’t support pre-merge execution. Therefore, we need to download the current state from our peers. The largest challenge in snap sync is downloading the state (the account state trie and storage state tries). Secondary concerns are downloading headers and bytecodes.

First solution: Fast-Sync

Fast-sync is the original method used in Ethereum to download a Patricia Merkle trie. The idea is to download the trie top-down, starting from the root, then recursively downloading child nodes until the entire trie is obtained.

There are two problems with this:

• Peers stop responding to node requests at some point. When requesting a trie node, you specify the state root for which you want the node. If the root is 128 or more blocks old, peers will not serve the request.
• Scanning the entire trie to find missing nodes is slow.

For the first problem: once peers stop serving nodes for a given root¹, we stop fast-syncing, update the pivot, and restart the process. The naïve approach would be to download the new root and recursively fetch all its children, checking each time whether they already exist in the DB.

Example of a possible state after stopping fast sync due to staleness.

In the example, even if we find that node { hash: 0x317f, path: 0 } is correct, we still need to check all its children in the DB (in this case none are present).

To solve the second problem, we introduce an optimization, which is called the "Membatch"². This allows us to maintain a new invariant:

If a node is present in the DB, then that node and all its children must be present.

This removes the need to explore entire subtrees: when walking down the trie, if a node is in the DB, the whole subtree can be skipped.

To maintain this invariant, we do the following:

• When we get a new node, we don't immediately store it in the database. We keep track of the amount of every node's children that are not yet in the database. As long as it's not zero, we keep it in a separate in-memory structure "Membatch" instead of on the db.
• When a node has all of its children in the db, we commit it and recursively go up the tree to see if its parent needs to be committed, etc.
• When the nodes are written to the database, all its parents are deleted from the db, which preserves the subtree invariant. Because lower down nodes are always written first, we never delete a valid node.

Example of a possible state after stopping fast sync due to staleness with membatch.

Speeding up: Snap-Sync

Fast-sync is slow (ethrex fast-sync of a fresh “hoodi” state takes ~45 minutes—4.5× slower than snap-sync). To accelerate it, we use a key property of fast-sync:

Fast-sync can “heal” any partial trie that obeys the invariant, even if that trie is not consistent with any real on-chain state.

Snap sync exploits this by:

• downloading only the leaves (the accounts and storage slots) from any recent state
• assembling a trie from these leaves
• running fast-sync (“healing”) to repair it into a consistent trie. In our code, we call the fast-sync step "healing".

Example run:

- We download the 4 accounts in the state trie from two different blocks

- We rebuild the trie

- We run the healing algorithm and get a correct tree at the end of it

This method alone provides up to a 4-5 times boost in performance, as computing the trie is way faster than downloading it.

Implementation

Generalized Flowchart of snapsync

Flags

When developing snap sync there are flags to take into account that are used only for testing:

• If the SKIP_START_SNAP_SYNC environment variable is set and isn't empty, it will skip the step of downloading the leaves and will immediately begin healing. This simulates the behaviour of fast-sync.

• If debug assertions are on, the program will validate that the entire state and storage tries are valid by traversing the entire trie and recomputing the roots. If any is found to be wrong, it will print an error and exit the program. This is used for debugging purposes; a validation error here means that there is 100% a bug in snap sync.

• --syncmode [full, default:snap] which defines what kind of sync we use. Full is executing each block, and isn't possible on a fresh sync for mainnet and sepolia.

File Structure

The sync module is a component of the ethrex-p2p crate, found in crates/networking/p2p folder. The main sync functions are found in:

• crates/networking/p2p/sync.rs
• crates/networking/p2p/peer_handler.rs
• crates/networking/p2p/sync/state_healing.rs
• crates/networking/p2p/sync/storage_healing.rs
• crates/networking/p2p/sync/code_collector.rs

Syncer and Sync Modes

The struct that handles the needed handles for syncing is the Syncer, and it has a variable to indicate if the snap mode is enabled. The Sync Modes are defined in sync.rs as follows.

#![allow(unused)]

fn main() {
/// Manager in charge the sync process
#[derive(Debug)]
pub struct Syncer {
    /// This is also held by the SyncManager allowing it to track the latest syncmode, without modifying it
    /// No outside process should modify this value, only being modified by the sync cycle
    snap_enabled: Arc<AtomicBool>,
    peers: PeerHandler,
    // Used for cancelling long-living tasks upon shutdown
    cancel_token: CancellationToken,
    blockchain: Arc<Blockchain>,
    /// This string indicates a folder where the snap algorithm will store temporary files that are
    /// used during the syncing process
    datadir: PathBuf,
}

pub enum SyncMode {
    #[default]
    Full,
    Snap,
}
}

The flow of the program in default mode is to start by doing snap sync, then we switch to fullsync at the end and continue catching up by executing blocks.

Downloading Headers

The first step is downloading all the headers, through the request_block_headers function. This function does the following steps:

• Request from peers the number of the sync_head that we received from the consensus client
• Divide the headers into discrete "chunks" to ask our peers
  • Currently, the headers are divided into 800 chunks³
• Queue those chunks as tasks into a channel
  • These tasks ask the peers for their data, and respond through a channel
• Read from the channel to get a task
• Finds the best free peers
• Spawn a new async job to ask the peer for the task
• If the channel for new is empty, check if everything is downloaded
• Read from the channel of responses
• Store the read result

Downloading Account Values

API

When downloading the account values, we use the snap function GetAccountRange. This request receives:

• rootHash: state_root of the block we're trying to download
• startingHash: Account hash⁴ of the first to retrieve
• limitHash: Account hash after which to stop serving data
• responseBytes: Soft limit at which to stop returning data

This method returns the following

• accounts: List of consecutive accounts from the trie
  • accHash: Hash of the account address (trie path)
  • accBody: Account body in slim format
• proof: List of trie nodes proving the account range

The proof is a merkle proof of the accounts provided, and the root of that merkle must equal to the rootHash. In ethrex this is checked by the verify_range function.

#![allow(unused)]
fn main() {
/// Verifies that the key value range belongs to the trie with the given root given the edge proofs for the range
/// Also returns true if there is more state to be fetched (aka if there are more keys to the right of the given range)
pub fn verify_range(
    root: H256,
    left_bound: &H256,
    keys: &[H256],
    values: &[ValueRLP],
    proof: &[Vec<u8>],
) -> Result<bool, TrieError>
}

We know we have finished a range if the last of the accounts downloaded is to the right of the bound we have set to the request, or if the verify_range returns true.

Dump to file

To avoid having all of the accounts in memory, when their size in memory exceeds 64MiB we dump them to a new file. These files are a subfolder of the datadir folder called "account_state_snapshots". For an optimization for faster insertion, these are stored ordered in the RocksDB sst file format.

Flowchart

Insertion of Accounts

The sst files in the "account_state_snapshots" subfolder are ingested into a RocksDB database. This provides an ordered array that is used for insertion.

More detailed documentation found in sorted_trie_insert.md.

Downloading Storage Slots

The download of the storage slots is conceptually similar to the download of accounts, but very different in implementation. The method uses the snap function GetStorageRanges. This request has the following parameters:

• rootHash: state_root of the block we're trying to download
• accountHashes: List of all the account address hashes of the storage tries to serve
• startingHash: Storage slot hash of the first to retrieve
• limitHash: Storage slot hash after which to stop serving
• responseBytes: Soft limit at which to stop returning data

The parameters startingHash and limitHash are only read when accountHashes is a single account.

The return is similar to the one from GetAccountRange, but with multiple results, one for each account provided, with the following parameters:

• slots: List of list of consecutive slots from the trie (one list per account)
  • slotHash: Hash of the storage slot key (trie path)
  • slotData: Data content of the slot
• proof: List of trie nodes proving the slot range

From these parameters, there is a couple of difficulties that pop up.

• We need to know which accounts have storage that needs to be downloaded
• We need to know what storage root each account has to be able to verify it

To solve these issues we take two actions:

• Before we download the storage slots we ensure that the state trie is in a consistent complete state. This is accomplished by doing the insertion of accounts step first and then healing the trie. If during the storage slot download the pivot becomes stale, we heal the trie again with the new pivot, to keep the trie up to date.
• When inserting the accounts, we grab a list of all the accounts with their storage root. If the account is healed, we marked the storage root as None, to indicate we should check in the DB what is the state of the storage root.

The time traveling problem

During snap sync development, we kept running into a problematic edge case: what we call "time traveling". This is when a certain account is in state A at a certain pivot, then changes to B in the next pivot, then goes back to A on a subsequent pivot change. This was a huge source of problems because it broke an invariant we assumed to be true, namely:

• If an account changes its state on pivot change, we will encounter it during state healing. This is NOT true if the trie is hash based.

The reason for this is the time traveling scenario. When an account goes back to a state we already have on our database, we do not heal it (because we already have it), even though its state has changed. This means the code won't realize the account has changed its storage root and won't update it.

This should not be a problem in a path based trie (which we currently have), but it's important to keep it in mind and make sure that whatever code we write keeps this time traveling edge case in mind and solves it.

Repeated Storage Roots

A large amount of the accounts with storage have exactly the same storage as other accounts.⁵ As such, when we are creating tasks for download, it's important to group the tasks by storage root and not download them twice.

Big Accounts

Storage trie sizes have a very uneven distribution. Around 70% of all ethereum mainnet contracts have only 1 or 2 storage slots. However, a few contracts have more storage slots than all account leaves in the entire state trie. As such, the code needs to take this pareto distribution into account to download storage tries fast.

At the beginning of the algorithm, we divide the accounts into chunks of 300 storage roots and their corresponding accounts. We start downloading the storage slots, until we find an account whose storage doesn't fit into a single request. This will be indicated by the proof field having the data indicating that there are still more nodes to download in that account.

When we reach that situation, we chunk the big account based on the "density"⁶ of storage slots we downloaded, following this code to get chunks of 10,000 slots⁷. We create the tasks to download those intervals, and store all of the intervals in a struct to check when everything for that account was properly downloaded.

#![allow(unused)]
fn main() {
    // start_hash_u256 is the hash of the address of the last slot
    // slot_count is the amount of slots we have downloaded
    // The division gives us the density (maximum possible slots/actual slots downloaded) 
    // we want chunks of 10.000 slots, so we multiply those two numbers
    let storage_density = start_hash_u256 / slot_count;
    let slots_per_chunk = U256::from(10000);
    let chunk_size = storage_density
        .checked_mul(slots_per_chunk)
        .unwrap_or(U256::MAX);
}

Tasks API

#![allow(unused)]
fn main() {
struct StorageTask {
    // Index of the first storage account we want to download
    start_index: usize,
    // Index of the last storage account we want to download (not inclusive)
    end_index: usize,
    // startingHash, used when the task is downloading a single account
    start_hash: H256,
    // end_hash is Some if the task is to download a big task
    end_hash: Option<H256>,
}

struct StorageTaskResult {
    // Index of the first storage account we want to download
    start_index: usize,
    // Slots we have successfully downloaded with the hash of the slot + value
    account_storages: Vec<Vec<(H256, U256)>>,
    // Which peer answered the task, used for scoring
    peer_id: H256,
    // Index of the first storage account we still need to download
    remaining_start: usize,
    // Index of the last storage account we still need to download
    remaining_end: usize,
    // remaining_hash_range[0] is the hash of the last slot we downloaded (so we need to download starting from there)
    // remaining_hash_range[1] is the end_hash from the original StorageTask
    remaining_hash_range: (H256, Option<H256>),
}
}

Big Accounts Flow

Retry Limit

Currently, if ethrex has been downloading storages for more than 2 pivots, the node will stop trying to download storage, and fallback to heal (fast sync) all the storage accounts that were still missing downloads. This prevents snap sync from hanging due to an edge case we do not currently handle if an account time travels. See the "snap sync concerns" document for details on what this edge case is.

Downloading Bytecodes

Whenever an account is downloaded or healed we check if the code is not empty. If it isn't, we store it for future download. This is added to a list, and when the list grows beyond a certain size it is written to disk. After the healing is done and we have a complete state and storage tree, we start with the download of bytecodes, chunking them to avoid memory overflow.

Forkchoice update

Once the entire state trie, all storage tries and contract bytecodes are downloaded, we switch the sync mode from snap to full, and we do an apply_forkchoice to mark the last pivot as the last block.

1. We update pivots based only on the timestamp, not on the peer response. According to the spec, stale roots return empty responses, but Byzantine peers may return empty responses at arbitrary times. Therefore, we rely on the rule that nodes must keep at least 128 blocks. Once time > timestamp + 128 * 12 we mark the pivot as stale. ↩

2. The membatch is an idea taken from geth, and the name comes from their code. The name should be updated to "pendingNodes" as it reflects its current use. ↩

3. This currently isn't a named constant, we should change that ↩

4. All accounts and storages are sent and found through the hash of their address. Example: the account with address 0xf003 would be found through the 0x26c2...38c1 hash, and would be found before the account with address 0x0001 whose hash would be 0x49d0...49d5 ↩

5. This may be for a variety of reasons, but the most likely is ERC20 tokens that were deployed and never used. ↩

6. actually specific volume (maximum possible slots/actual slots downloaded) ↩

7. 10_000 slots is a number chosen without hard data, we should review that number. ↩

Can you delete accounts in Ethereum? Yes

How it happens

Ethereum accounts are broadly divided into two categories:

• Externally Owned Accounts (EOA): accounts for general users to transfer eth and call contracts.
• Contracts: which execute code and store data.

Creating EOA is done through sending ETH into a new address, at which point the account is created and added into the state trie.

Creating a contract can be done through the CREATE and CREATE2 opcode. Notably, those opcodes check that the account is created at an address where the code is empty and the nonce is zero, but it doesn't check balance. As such, a contract can be created through taking over an existing account.

During the creation of a contract, the init_code is run which can include the self destruct opcode that deletes the contract in the same transaction it was created. Normally, this deletes an account that was created in the same transaction (because contracts are usually created over empty accounts) but in this case the account already existed because it already had some balance. This is the only edge case in which an account can go from existing to non-existing from one block to another after the Cancun fork.

How we found it

Snap-sync is broadly divided into two stages:

• Downloading the leaves of the state (account states) and storage tries (storage slots)
• Healing (reconciling the state).

Healing is needed because the leaves can be downloaded from disparate blocks, and to "fix" only the nodes of the trie that changed between blocks. In depth explanation.

We were working under the assumption that accounts were never deleted, so we adopted some specific optimizations. During the state healing stage every account that was "healed" was added into a list of accounts that needed to be checked for storage healing. When healing the storage of those accounts the algorithm requested their account states and expected them to be there to see if they had any storage that needed healing. This led to the storage healing threads panicking when they failed to find the account that was deleted.

During the test of snapsync mainnet, we started seeing that storage healing was panicking, so we added some logs to see what account hashes were being accessed and when were they healed vs accessed. Exploring the database we saw that the offending account was present in a previous state and missing in the next one, with the corresponding merkle proof matching the block state root. Originally we suspected a reorg, but searching the blocks we saw they were finalized in the chain.

The account state present indicated an account with nonce 0, no code and no storage but with balance. We didn't have access to the account address, as the state trie only stores the hash of the account address so we turned to another strategy to find it. Using etherscan's API allowing to search internal transactions from a block range, we explored the range where we knew the account existed in the state trie. Hashing all of the to and from of the transactions we found the transaction that deleted the account with a self destruct. Despite the account becoming a contract just during that transaction, we saw that 900 blocks before it was created with a transfer. The result of the self destruct was the transfer of 0.044 ETH from one account to another.

The specific transaction that created the contract: https://etherscan.io/tx/0xf23b2c233410141cda0c6d24f21f0074c494565bfd54ce008c5ce1b30b23b0da

Snap Sync Concerns

Code Improvement opportunities

Storage downloads

When downloading storages, there's a possibility that storage leaves never finish downloading. This can happen if an account time travels (i.e. if it goes from state A to state B on a pivot change, then back to state A on a subsequent pivot). The reason for this is that we have an in-memory cache where we keep track of the storage root for every account that needs to be downloaded. On every state healing phase on a pivot change, we update the storage root of every account we encounter. However, if an account time travels, we will not encounter it during healing because we already had it, and thus our storage root for it will be wrong.

NOTE: this should no longer be a problem now that our trie is path based, because on every state healing phase we erase the account's previous state. However, it's important to keep this scenario in mind when developing snap sync; any change in how we handle our trie or our assumptions may run into a problem it does not properly handle time traveling accounts.

Handling the pivot and reorgs

We are currently asking the pivot from our peers. We should have a system for handling the pivot from our consensus client. We should also be able to understand if the new pivot received is a reorg. In that case, we can't fullsync, but we can fast-sync between those pivots relatively easily.

Potential Bytecode Nonresponse

We are currently asking for all the bytecodes that we have seen, never checking if those bytecodes are currently in the tree. This isn't a problem for most codes that are immutable, but EOA may change their code to be a delegated, and the old code may be deleted from other peers in that scenario. We should consider pruning the bytecode requests if one is downloaded during healing.

Performance

For performance, having an efficient cache of accounts that need to have their storage downloaded in memory is key, and we should avoid going to the db as much as possible. In particular in storage healing we start by trying to get all of the storage healing roots, and this can be sped up considerably if we avoid going to the db.

Improving debug

The functions validate_state_root and validate_storage_roots are very slow, as they rebuild the entire state trie in memory.

Code Quality

In general, snap sync lacks explanation comments that detail the functioning of the algorithm. Variables and structs should be renamed to make it properly readable.

Storage downloads

Request storages is a very hard function to read, as the data structures were constantly modified to introduce speed optimizations. As such, this function is critical to restructure and manage it taking into account memory concerns.

This function also has a lot of numeric constants inserted in the code directly, and should be handled better by having defined consts with explanations.

Healing

There are two healing challenges. We should have a single main algorithm for healing, not have the code duplicated across two files. On top of that, the Membatch structure should be replaced with "PendingNodes" as it's a far more descriptive name.

Memory Concerns

Storage accounts

Currently, we use a struct accounts_by_root_hash that has an unbounded size in memory. When rewriting this algorithm we should make sure that we never use more than a certain amount of memory for it; the rest should be on disk.

Sorted Trie Insertion

This document describes the algorithm implemented in crates/common/trie/trie_sorted.rs which is used to speed up the insertion time in snap sync. During that step we are inserting all of the accounts state and storage slots downloaded into the Ethereum world state Merkle Patricia Trie. To know how that trie works, it's recommended to read this primer first.

Concept

Naive algorithm: we insert keys in arbitrary (unsorted) order. This version requires O(n*log(n)) reads and writes to disk. This is because each insertion creates a new leaf, which modifies the hash of its parent branch recursively. We could avoid reads to disk by having the trie in memory, but this is unviable for large amounts of state data.

Example of the Naive implementation:

If the input data is sorted, the computation can be optimized to be O(n). In the example, just by reading 0x0EBB and 0x172E, we know that there is a branch node as root (because they start with different nibbles), and that the leaf will have a partial path of 0xEBB (because no node exists between 0x0EBB and 0x172E if it's sorted). The root branch node we know exists and will be modified, so we don't write until we have read all input.

Implementation

The implementation maintains three pointers:

1. The current element being processed.
2. The next input value.
3. The parent of the current element.

All parents that can still be modified are stored in a "parent stack". Based on these, the algorithm can determine the next write operation to perform.

Scenarios

Depending on the state of the three current pointers, one of 3 scenarios can happen:

Scenario 1: Current and next value are siblings with the current parent being the parent of both values. This happens when the parent and both values share the same number of nibbles at the beginning of their paths. In our example, all node paths start with 0x1 and then diverge.

In this scenario, we can compute the leaf for the current value, write it, update the parent to include a pointer to that leaf, and then continue.

Scenario 2: Current and next values are siblings of a new current parent. This happens when the parent shares less nibbles from their paths than what the siblings share. In our example, the current and next value share 0x17, while the parent only shares 0x1.

In this scenario, we know the leaf we need to compute from the current value so we write that. Furthermore, we know that we need a new branch at 0x17, so we create it and insert the leaf we just computed and insert it into the branch. The current parent is stored in the "parent stack", and the new branch becomes the current parent.

Scenario 3: The current parent is not the parent of the next value. This happens when the parent doesn't have all of the nibbles of its path.

In this scenario, we know the leaf we need to compute from the current value, so we write that. We change the current value to be the current parent, and the new current parent is popped from the "parent stack".

These three scenarios keep repeating themselves until the trie is complete, at which point the algorithm returns a hash to the root node branch.

Inserting with extensions

In general, each write to disk is prepared to properly handle extensions as the write function knows what it's writing and what was its parent and full path. As such, it can check if the insertion is a branch and if an extension is needed.

A specific edge case is the root node, which is assumed to always be a branch node, but the code has a special case check to see if the root node has a single child, in which case it changes to an extension or leaf as needed, while modifying the other nodes in the trie.

Concurrency

The slowest step in this process is flushing nodes to disk. To avoid stalling during writes, the algorithm uses an internal buffer that holds a fixed number of nodes. Once the buffer is filled, it creates a new task that writes it to disk in the background.

We want to limit the amount of buffers we can have, so we allocate a fixed number of buffers at the beginning and we use a channel for the algorithm to receive empty buffers and the writing task clears the buffer and sends it back through the channels.

These tasks are executed using a custom thread pool defined in /crates/concurrency/concurrency.rs

Healing Algorithm Explanation and Documentation (Before Path Based)

Healing is the last step of Snap Sync. Snap begins the downloading of the state and storage tries by downloading the leaves (account states and storage slots), and from those leaves we reconstruct the intermediate nodes (branches and extension). Afterwards we may be left with a malformed trie, as that step will resume the download of leaves with a new state root if the old one times out.

The purpose of the healing algorithm is to "heal" that trie so that it ends up in a consistent state.

Healing Conceptually

The malformed trie is going to have large sections of the trie which are in a correct state, as we had all of the leaves in those sections and those accounts haven’t been modified in the blocks that happened concurrently to the snapsync algorithm.

Example of a trie where 3 leaves were downloaded in block 1 and 1 was downloaded in block 2. The trie root is different from the state root of block 2, as one of the leaf nodes was modified in block 2.

The algorithm attempts to rebuild the trie through downloading the missing nodes, starting from the top. If the node is present in the database that means that we have that and all of their child nodes present in the database. If not, we download the node and check if the children of the root are present, applying the algorithm recursively.

Iteration 1 of algorithm

Iteration 2 of algorithm

Iteration 3 of algorithm

Final state of trie after healing

Implementation

The algorithm is implemented in ethrex currently in crates/networking/p2p/sync/state_healings.rs and crates/networking/p2p/sync/storage_healing.rs. All of our code examples are from the account state trie.

API

The API used is the ethereum capability snap/1, documented at https://github.com/ethereum/devp2p/blob/master/caps/snap.md and for healing the only method used is GetTrieNodes. This method allows us to ask our peers for nodes in a trie. We ask the nodes by path to the node, not by hash.

#![allow(unused)]
fn main() {
pub struct GetTrieNodes {    
    pub id: u64,    
    pub root_hash: H256,    
    // [[acc_path, slot_path_1, slot_path_2,...]...]    
    // The paths can be either full paths (hash) or 
    // only the partial path (compact-encoded nibbles)    
    pub paths: Vec<Vec<Bytes>>,    
    pub bytes: u64,
}
}

Staleness

The spec allows the nodes to stop responding if the request is older than 128 blocks. In that case, the response to the GetTrieNodes will be empty. As such, our algorithm checks periodically if the block is stale, and stops executing. In that scenario, we must be sure that we leave the storage in a consistent state at any given time and don't break our invariants.

#![allow(unused)]
fn main() {
// Current Staleness logic code
// We check with a clock if we are stale        
if !is_stale && current_unix_time() > staleness_timestamp {
    info!("state healing is stale");            
    is_stale = true;       
}
// We make sure that we have stored everything that we need to the database
if is_stale && nodes_to_heal.is_empty() && inflight_tasks == 0 {
  info!("Finished inflight tasks");            
  db_joinset.join_all().await;            
  break;
}
}

Membatch

Currently, our algorithm has an invariant, which is that if we have a node in storage we have its and all of its children are present. Therefore, when we download for a node if some of its children are missing we can’t immediately store it on disk. Our implementation currently stores the nodes in temporary structure called membatch, which stores the node and how many of its children are missing. When a child gets stored, we reduce the counter of missing children of the parent. If that numbers reaches 0, we write the parent to the database.

In code, the membatch is currently a HashMap<Nibbles, MembatchEntryValue> with the value being the following struct

#![allow(unused)]
fn main() {
pub struct MembatchEntryValue {
    /// The node to be flushed into storage
    node: Node,
    /// How many of the nodes that are child of this are not in storage
    children_not_in_storage_count: u64,
    /// Which is the parent of this node
    parent_path: Nibbles,
}
}

Known Optimization Issues

• Membatch gets cleared between iterations, while it could be preserved and the hash checked.
• When checking if a child is present in storage, we can also check if it’s in the membatch. If it is, we can skip that download and act like we have immediately downloaded that node.
• Membatch is currently a HashMap, a BTreeMap or other structures may be faster in real use.
• Storage healing receives as a parameter a list of accounts that need to be healed and it has to get their state before it can run. Doing those reads could be more efficient.

Introduction

Layer 2 (L2) solutions are protocols built on top of Ethereum to increase scalability and reduce transaction costs. L2s process transactions off-chain and, in the case of Rollups, they periodically post data or proofs back to Ethereum Layer 1, inheriting its security.

Ethrex is a framework that lets you launch your own L2 rollup or blockchain. With ethrex, you can deploy, run, and experiment with custom L2 networks, taking advantage of Ethereum's security while enabling high throughput and low fees.

Get started with your L2

Check out the Quickstart L2 guide to start your rollup in just a command, or jump right into the Deploy an L2 for more detailed instructions.

Deploy an L2

This section provides step-by-step guides for deploying different types of ethrex L2 chains, including vanilla, validium, and based configurations, as well as a shared bridge enabled L2 and migrations between versions. Each guide outlines the necessary commands and parameters to successfully deploy and start an L2 node.

Use this section to choose the deployment method that best fits your needs and follow the instructions accordingly.

• Overview
• Deploying a vanilla ethrex L2
• Deploying a validium ethrex L2
• Deploying a based ethrex L2
• Ethrex <> Aligned
• Deploying a shared bridge enabled L2
• Deploying a fee token
• Upgrades

Deploying an ethrex L2

As outlined in the introduction, ethrex L2 offers a wide range of features to its users. The most common is a classic centralized L2 managed by an operator, which can be an individual or a DAO. ethrex L2 can also function as a Validium, which is similarly centralized and operator-managed, with the key difference that network data is not posted to L1 during settlement.

In addition to these classic functionalities, ethrex L2 provides a novel and continually evolving feature in the industry: ethrex L2 as a based rollup. Unlike the previous options, this is a permissionless and decentralized L2 sequencer—anyone can run a node and participate in the network.

In this section, we will cover how to deploy any of these options.

Before proceeding, note that this guide assumes you have ethrex installed. If you haven't installed it yet, follow one of the methods in the Installation Guide. If you're looking to build from source, don't skip this section—we'll cover that method here, as it is independent of the deployment approach you choose later.

Building from source (skip if ethrex is already installed)

Prerequisites

Ensure you have the following installed on your system:

• Rust and Cargo (install via rustup)
• Solidity compiler v0.8.31 (refer to Solidity documentation)
• SP1 Toolchain (if you plan to use SP1 proving, refer to SP1 documentation)
• RISC0 Toolchain (if you plan to use RISC0 proving, refer to RISC0 documentation)
• CUDA Toolkit 12.9 (if you plan to use GPU acceleration for SP1 or RISC0 proving)

1. Clone the official ethrex repository:

   git clone https://github.com/lambdaclass/ethrex
   cd ethrex
   

2. Install the binary to your $PATH:

   # For dummy proving
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql
   
   # For SP1 CPU proving (very slow, not recommended)
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1
   
   # For RISC0 CPU proving (very slow, not recommended)
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,risc0
   
   # For SP1 and RISC0 CPU proving (very slow, not recommended)
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,risc0
   
   # For SP1 GPU proving
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,gpu
   
   # For RISC0 GPU proving
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,risc0,gpu
   
   # For SP1 and RISC0 GPU proving
   COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,risc0,gpu
   

   By default cargo install places the binary at ~/.cargo/bin/ethrex (make sure that directory is on your $PATH). Add --force to the commands above if you need to overwrite a previous installation.

PROVER_REPRODUCIBLE_BUILD=true COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,risc0,gpu

Deploying a vanilla ethrex L2

In this section, we'll cover how to deploy a vanilla ethrex L2 on a public network such as Holesky, Sepolia, or Mainnet.

Prerequisites

This guide assumes that you have ethrex installed and available in your PATH. If you haven't installed it yet, follow one of the methods in the Installation Guide. If you want to build the binary from source, refer to the Building from source section and select the appropriate build option.

1. Deploy the Contracts

First, deploy and initialize the contracts on L1 using the ethrex l2 deploy command (for more details on the ethrex CLI, see the ethrex CLI Reference section):

ethrex l2 deploy \
  --eth-rpc-url <L1_RPC_URL> \
  --private-key <PRIVATE_KEY> \
  --genesis-l2-path <PATH_TO_L2_GENESIS_FILE> \
  --bridge-owner <COMMON_BRIDGE_OWNER_ADDRESS> \
  --on-chain-proposer-owner <ON_CHAIN_PROPOSER_OWNER_ADDRESS> \
  --committer.l1-address <L1_COMMITTER_ADDRESS> \
  --proof-sender.l1-address <L1_PROOF_SENDER_ADDRESS> \
  --env-file-path <PATH_TO_ENV_FILE> \
  --randomize-contract-deployment

2. Start the L2 node

Once the contracts are deployed, start the L2 node:

ethrex l2 \
  --l1.bridge-address <COMMON_BRIDGE_ADDRESS> \
  --l1.on-chain-proposer-address <ON_CHAIN_PROPOSER_ADDRESS> \
  --block-producer.coinbase-address <L2_COINBASE_ADDRESS> \
  --proof-coordinator.l1-private-key <L1_PROOF_SENDER_PRIVATE_KEY> \
  --committer.l1-private-key <L1_COMMITTER_PRIVATE_KEY> \
  --eth.rpc-url <L1_RPC_URL> \
  --network <PATH_TO_L2_GENESIS_FILE> \
  --no-monitor

That's it! You now have a vanilla ethrex L2 up and running. However, one key component is still missing: state proving. The L2 state is considered final only after a batch execution ZK proof is successfully verified on-chain. Generating these proofs requires running a dedicated prover, which is covered in the Run an ethrex L2 Prover section.

Deploying a validium ethrex L2

In this section, we'll cover how to deploy a validium ethrex L2 on a public network such as Holesky, Sepolia, or Mainnet.

Prerequisites

This guide assumes that you have ethrex installed and available in your PATH. If you haven't installed it yet, follow one of the methods in the Installation Guide. If you want to build the binary from source, refer to the Building from source section and select the appropriate build option.

1. Deploy the Contracts

First, deploy and initialize the contracts on L1 using the ethrex l2 deploy command (for more details on the ethrex CLI, see the ethrex CLI Reference section):

ethrex l2 deploy \
  --validium true \
  --eth-rpc-url <L1_RPC_URL> \
  --private-key <PRIVATE_KEY> \
  --genesis-l2-path <PATH_TO_L2_GENESIS_FILE> \
  --bridge-owner <COMMON_BRIDGE_OWNER_ADDRESS> \
  --on-chain-proposer-owner <ON_CHAIN_PROPOSER_OWNER_ADDRESS> \
  --committer.l1-address <L1_COMMITTER_ADDRESS> \
  --proof-sender.l1-address <L1_PROOF_SENDER_ADDRESS> \
  --env-file-path <PATH_TO_ENV_FILE> \
  --randomize-contract-deployment

2. Start the L2 node

Once the contracts are deployed, start the L2 node:

ethrex l2 \
  --validium \
  --l1.bridge-address <COMMON_BRIDGE_ADDRESS> \
  --l1.on-chain-proposer-address <ON_CHAIN_PROPOSER_ADDRESS> \
  --block-producer.coinbase-address <L2_COINBASE_ADDRESS> \
  --proof-coordinator.l1-private-key <L1_PROOF_SENDER_PRIVATE_KEY> \
  --committer.l1-private-key <L1_COMMITTER_PRIVATE_KEY> \
  --eth.rpc-url <L1_RPC_URL> \
  --network <PATH_TO_L2_GENESIS_FILE> \
  --no-monitor

That's it! You now have a validium ethrex L2 up and running. However, one key component is still missing: state proving. The L2 state is considered final only after a batch execution ZK proof is successfully verified on-chain. Generating these proofs requires running a dedicated prover, which is covered in the Run an ethrex L2 Prover section.

Deploying a based ethrex L2

TBD

Running Ethrex in Aligned Mode

This guide extends the Deploy an L2 overview and shows how to run an ethrex L2 with Aligned mode enabled.

For a comprehensive technical deep-dive into the Aligned integration architecture, see Aligned Layer Integration.

It assumes:

• You already installed the ethrex binary to your $PATH (for example from the repo root with cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1 --force).
• You have the ethrex repository checked out locally for the make targets referenced below.

Important: Aligned mode only supports SP1 proofs. The sp1 feature must be enabled when building with Aligned mode.

• Check How to Run (local devnet) for development or testing.
• Check How to Run (testnet) for a prod-like environment.

How to run (testnet)

1. Generate the prover ELF/VK

From the ethrex repository root run:

make -C crates/l2 build-prover-sp1 # optional: GPU=true

This will generate the SP1 ELF program and verification key under:

• crates/l2/prover/src/guest_program/src/sp1/out/riscv32im-succinct-zkvm-elf
• crates/l2/prover/src/guest_program/src/sp1/out/riscv32im-succinct-zkvm-vk-u32

2. Deploying L1 Contracts

Run the deployer with the Aligned settings:

COMPILE_CONTRACTS=true \
ETHREX_L2_ALIGNED=true \
ETHREX_DEPLOYER_ALIGNED_AGGREGATOR_ADDRESS=<ALIGNED_AGGREGATOR_ADDRESS> \
ETHREX_L2_SP1=true \
ETHREX_DEPLOYER_RANDOMIZE_CONTRACT_DEPLOYMENT=true \
ethrex l2 deploy \
  --eth-rpc-url <ETH_RPC_URL> \
  --private-key <YOUR_PRIVATE_KEY> \
  --on-chain-proposer-owner <ON_CHAIN_PROPOSER_OWNER>  \
  --bridge-owner <BRIDGE_OWNER_ADDRESS>  \
  --genesis-l2-path fixtures/genesis/l2.json \
  --proof-sender.l1-address <PROOF_SENDER_L1_ADDRESS>

3. Deposit funds to the AggregationModePaymentService contract from the proof sender

Aligned uses a quota-based payment model. You need to deposit ETH to obtain quota for proof submissions using the Aligned CLI.

First, clone the Aligned repository and build the CLI:

git clone https://github.com/yetanotherco/aligned_layer.git
cd aligned_layer
git checkout 54ca2471624700536561b6bd369ed9f4d327991e

Then run the deposit command:

cd aggregation_mode/cli

cargo run --release -- deposit \
  --private-key <PROOF_SENDER_PRIVATE_KEY> \
  --network <NETWORK> \
  --rpc-url <RPC_URL>

Where <NETWORK> is one of: devnet, hoodi, or mainnet.

Example for Hoodi testnet:

cargo run --release -- deposit \
  --private-key 0x... \
  --network hoodi \
  --rpc-url https://ethereum-hoodi-rpc.publicnode.com

Note: The deposit command sends a fixed amount of ETH (currently 0.0035 ETH) to the payment service contract. The contract addresses are automatically resolved based on the network parameter.

Monitoring Quota Balance

To check your remaining quota, you can query the AggregationModePaymentService contract directly:

# Get the payment service contract address for your network from Aligned docs
# Then query the quota balance for your proof sender address
cast call <PAYMENT_SERVICE_ADDRESS> "getQuota(address)(uint256)" <PROOF_SENDER_ADDRESS> --rpc-url <RPC_URL>

Monitor your quota balance regularly. When the L1ProofSender runs out of quota, you'll see AlignedSubmitProofError with an insufficient quota message in the logs. Deposit more funds before this happens to avoid proof submission failures.

4. Running a node

Run the sequencer using the installed ethrex binary:

ethrex l2 \
  --watcher.block-delay 0 \
  --network fixtures/genesis/l2.json \
  --l1.bridge-address <BRIDGE_ADDRESS> \
  --l1.timelock-address <TIMELOCK_ADDRESS> \
  --l1.on-chain-proposer-address <ON_CHAIN_PROPOSER_ADDRESS> \
  --eth.rpc-url <ETH_RPC_URL> \
  --aligned \
  --aligned-network <ALIGNED_NETWORK>  \
  --block-producer.coinbase-address <COINBASE_ADDRESS>  \
  --committer.l1-private-key <COMMITTER_PRIVATE_KEY>  \
  --proof-coordinator.l1-private-key <PROOF_COORDINATOR_PRIVATE_KEY>  \
  --aligned.beacon-url <ALIGNED_BEACON_URL> \
  --datadir ethrex_l2 \
  --no-monitor

Both committer and proof coordinator should have funds.

Aligned params explanation:

• --aligned: Enables aligned mode, enforcing all required parameters.
• --aligned.beacon-url: URL of the beacon client used by the Aligned SDK to verify proof aggregations, it has to support /eth/v1/beacon/blobs
• --aligned-network: Parameter used by the Aligned SDK. Available networks: devnet, hoodi, mainnet.
• --aligned.from-block: (Optional) Starting L1 block number for proof aggregation search. Helps avoid scanning old blocks from before proofs were being sent. If not set, the search starts from the beginning.

If you can't find a beacon client URL which supports that endpoint, you can run your own with lighthouse and ethrex:

Create secrets directory and jwt secret

mkdir -p ethereum/secrets/
openssl rand -hex 32 | tr -d "\n" | tee ./ethereum/secrets/jwt.hex

lighthouse bn --network <NETWORK> --execution-endpoint http://localhost:8551 --execution-jwt ./ethereum/secrets/jwt.hex --checkpoint-sync-url <CHECKPOINT_URL> --http --purge-db-force --supernode

ethrex --authrpc.jwtsecret ./ethereum/secrets/jwt.hex --network <NETWORK>

5. Running the Prover

In another terminal start the prover:

make -C crates/l2 init-prover-sp1 GPU=true # The GPU parameter is optional

Then you should wait until Aligned aggregates your proof. Note that proofs are typically aggregated every 24 hours.

How to run (local devnet)

Set Up the Aligned Environment

1. Clone the Aligned repository and checkout the tested revision:

   git clone git@github.com:yetanotherco/aligned_layer.git
   cd aligned_layer
   git checkout 54ca2471624700536561b6bd369ed9f4d327991e
   

2. Edit the aligned_layer/network_params.rs file to send some funds to the committer and integration_test addresses:

   prefunded_accounts: '{
       "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266": { "balance": "100000000000000ETH" },
       "0x70997970C51812dc3A010C7d01b50e0d17dc79C8": { "balance": "100000000000000ETH" },
   
       ...
       "0xa0Ee7A142d267C1f36714E4a8F75612F20a79720": { "balance": "100000000000000ETH" },
   +   "0x4417092B70a3E5f10Dc504d0947DD256B965fc62": { "balance": "100000000000000ETH" },
   +   "0x3d1e15a1a55578f7c920884a9943b3b35d0d885b": { "balance": "100000000000000ETH" },
        }'
   

   You can also decrease the seconds per slot in aligned_layer/network_params.rs:

   # Number of seconds per slot on the Beacon chain
     seconds_per_slot: 4
   

   Change ethereum-genesis-generator to 5.2.3

   ethereum_genesis_generator_params:
     # The image to use for ethereum genesis generator
     image: ethpandaops/ethereum-genesis-generator:5.2.3
   

3. Make sure you have the latest version of kurtosis installed and start the ethereum-package:

   cd aligned_layer
   make ethereum_package_start
   

   If you need to stop it run make ethereum_package_rm

4. Start the payments poller (in a new terminal):

   cd aligned_layer
   make agg_mode_payments_poller_start_ethereum_package
   

   This starts PostgreSQL, runs migrations, and starts the payments poller.

5. Start the Aligned gateway (in a new terminal):

   cd aligned_layer
   make agg_mode_gateway_start_ethereum_package
   

   The gateway will listen on http://127.0.0.1:8089.

6. Build and start the proof aggregator in dev mode (in a new terminal):

   cd aligned_layer
   # Build the dev aggregator binary (uses mock proofs, no actual proving)
   AGGREGATOR=sp1 cargo build --manifest-path ./aggregation_mode/Cargo.toml --release --bin proof_aggregator_dev
   
   # Start the aggregator
   make proof_aggregator_start_dev_ethereum_package AGGREGATOR=sp1
   

   Note: The dev mode aggregator uses mock proofs for faster iteration. For production-like testing, use make proof_aggregator_start_ethereum_package SP1_PROVER=cuda AGGREGATOR=sp1 instead (requires more resources and a CUDA-capable GPU).

Initialize L2 node

1. Deploy the L1 contracts, specifying the AlignedProofAggregatorService contract address:

   COMPILE_CONTRACTS=true \
   ETHREX_L2_ALIGNED=true \
   ETHREX_DEPLOYER_ALIGNED_AGGREGATOR_ADDRESS=0xcbEAF3BDe82155F56486Fb5a1072cb8baAf547cc \
   ETHREX_L2_SP1=true \
   make -C crates/l2 deploy-l1
   

   [!NOTE] This command requires the COMPILE_CONTRACTS env variable to be set, as the deployer needs the SDK to embed the proxy bytecode.

   You will see that some deposits fail with the following error:

   2025-10-13T19:44:51.600047Z ERROR ethrex::l2::deployer: Failed to deposit address=0x0002869e27c6faee08cca6b765a726e7a076ee0f value_to_deposit=0
   2025-10-13T19:44:51.600114Z  WARN ethrex::l2::deployer: Failed to make deposits: Deployer EthClient error: eth_sendRawTransaction request error: insufficient funds for gas * price + value: have 0 want 249957710190063
   

   This is because not all the accounts are pre-funded from the genesis.

2. Deposit funds to the AggregationModePaymentService contract from the proof sender using the Aligned CLI:

   # From the aligned_layer repository root
   cd aggregation_mode/cli
   
   cargo run --release -- deposit \
     --private-key 0x39725efee3fb28614de3bacaffe4cc4bd8c436257e2c8bb887c4b5c4be45e76d \
     --network devnet \
     --rpc-url http://localhost:8545
   

3. Start the L2 node:

   ETHREX_ALIGNED_MODE=true \
   ETHREX_ALIGNED_BEACON_URL=http://127.0.0.1:58801 \
   ETHREX_ALIGNED_NETWORK=devnet \
   ETHREX_PROOF_COORDINATOR_DEV_MODE=false \
   SP1=true \
   make -C crates/l2 init-l2
   

   Suggestion: When running the integration test, consider increasing the --committer.commit-time to 2 minutes. This helps avoid having to aggregate the proofs twice. You can do this by adding the following flag to the init-l2-no-metrics target:

   --committer.commit-time 120000
   

4. Start the SP1 prover in a different terminal:

   make -C crates/l2 init-prover-sp1 GPU=true # The GPU flag is optional
   

Aggregate proofs:

After some time, you will see that the l1_proof_verifier is waiting for Aligned to aggregate the proofs. In production, proofs are typically aggregated every 24 hours. For local testing, the proof aggregator started in step 8 will process proofs automatically.

If the aggregator is not running or you need to trigger a new aggregation cycle, run:

cd aligned_layer
make proof_aggregator_start_dev_ethereum_package AGGREGATOR=sp1

This will reset the last aggregated block counter and start processing queued proofs.

If successful, the l1_proof_verifier will print the following logs:

INFO ethrex_l2::sequencer::l1_proof_verifier: Proof for batch 1 aggregated by Aligned with commitment 0xa9a0da5a70098b00f97d96cee43867c7aa8f5812ca5388da7378454580af2fb7 and Merkle root 0xa9a0da5a70098b00f97d96cee43867c7aa8f5812ca5388da7378454580af2fb7
INFO ethrex_l2::sequencer::l1_proof_verifier: Batches verified in OnChainProposer, with transaction hash 0x731d27d81b2e0f1bfc0f124fb2dd3f1a67110b7b69473cacb6a61dea95e63321

Behavioral Differences in Aligned Mode

Prover

• Generates Compressed proofs instead of Groth16 (used in standard mode).
• Required because Aligned accepts compressed SP1 proofs.
• Only SP1 proofs are supported for Aligned mode.

Note: RISC0 support is not currently available in Aligned's aggregation mode. The codebase retains RISC0 code paths (verifier IDs, merkle proof handling, contract logic) for future compatibility when Aligned re-enables RISC0 support.

Proof Sender

• Sends proofs to the Aligned Gateway instead of directly to the OnChainProposer contract.
• Uses a quota-based payment model (requires depositing to the AggregationModePaymentService contract).
• Tracks the last proof sent using the rollup store.

Proof Verifier

• Spawned only in Aligned mode (not used in standard mode).
• Monitors whether the next proof has been aggregated by Aligned using the ProofAggregationServiceProvider.
• Once verified, collects all already aggregated proofs and triggers the advancement of the OnChainProposer contract by sending a single transaction.

OnChainProposer

• Uses verifyBatchesAligned() instead of verifyBatch() (used in standard mode).
• Receives an array of proofs to verify.
• Delegates proof verification to the AlignedProofAggregatorService contract.

Supported Networks

The Aligned SDK supports the following networks:

Failure Recovery

For guidance on handling Aligned Layer failures and outages, see the Aligned Failure Recovery Guide.

Aligned Layer Failure Recovery Guide

This guide provides operators with procedures for handling various Aligned Layer failure scenarios when running ethrex L2 in Aligned mode.

SDK Version: This documentation is based on Aligned Aggregation Mode SDK revision 54ca2471624700536561b6bd369ed9f4d327991e.

WARNING: This document is intended to be iterated and improved with use and with ethrex and Aligned upgrades. If you encounter a scenario not covered here or find that a procedure needs adjustment, please contribute improvements.

Table of Contents

1. Understanding the Aligned Integration
2. Scenario 1: Aligned Stops Then Recovers
3. Scenario 2: Aligned Loses a Proof Before Verification
4. Scenario 3: Aligned Permanent Shutdown
5. Scenario 4: Insufficient Quota Balance
6. Scenario 5: Proof Marked as Invalid by Aligned
7. Monitoring and Detection

Understanding the Aligned Integration

Before handling failures, understand the proof lifecycle in Aligned mode:

1. Prover generates compressed SP1 proof
2. L1ProofSender submits proof to Aligned Gateway (HTTP)
3. Aligned Gateway queues proof for aggregation
4. Aligned Aggregator aggregates multiple proofs (typically every 24 hours)
5. Aggregated proof is posted to L1 (AlignedProofAggregationService)
6. L1ProofVerifier polls for aggregation status
7. Once aggregated, L1ProofVerifier calls verifyBatchesAligned() on OnChainProposer

Key Components:

• L1ProofSender: Submits SP1 proofs to Aligned Gateway via HTTP
• L1ProofVerifier: Polls Aligned for aggregation status and triggers on-chain verification
• Aligned Gateway: Receives and queues proofs via HTTP REST API
• Aligned Aggregator: Aggregates proofs and posts to L1

Note: Aligned mode only supports SP1 proofs.

Scenario 1: Aligned Stops Then Recovers

Symptoms

• L1ProofSender logs show connection errors to Aligned Gateway
• Proofs are generated but not being sent
• AlignedGetNonceError in logs (failed to get nonce from gateway)
• AlignedSubmitProofError in logs (HTTP request to gateway failed)

Note: AlignedFeeEstimateError indicates your Ethereum RPC endpoints are failing, not Aligned. Fee estimation uses your configured --eth.rpc-url to query L1 gas prices.

Impact

• Proofs queue locally in the rollup store
• Batch verification on L1 stalls
• No data loss - proofs remain in local storage

Recovery Steps

No manual intervention required. The system handles this automatically:

1. L1ProofSender continuously retries sending proofs at the configured interval (--proof-coordinator.send-interval, default 30000ms)
2. Once Aligned recovers, proofs will be submitted in order
3. L1ProofVerifier will resume polling and verification

What to Monitor

Check L1ProofSender status

curl -X GET http://localhost:5555/health | jq '.proof_sender'

Watch logs for recovery

# Look for: "Submitting proof to Aligned" followed by "Submitted proof to Aligned"

Configuration Tips

• Consider increasing --proof-coordinator.send-interval during known Aligned maintenance windows to reduce log noise

Scenario 2: Aligned Loses a Proof Before Verification

Symptoms

• Proof was successfully submitted (logs show "Submitted proof to Aligned")
• L1ProofVerifier shows "Proof has not been aggregated by Aligned" for extended periods
• No aggregation event for the proof on Aligned's side

Impact

• Chain verification is blocked at the lost proof's batch number
• Proofs for subsequent batches continue to generate and queue

Recovery Steps

Step 1: Confirm the Proof is Lost

Check what batch the system thinks it sent:

-- Check the latest sent batch pointer (SQL storage)
SELECT batch FROM latest_sent WHERE _id = 0;

Check if the proof exists locally:

-- Check if proof exists in the rollup store (SQL storage)
-- prover_type: 1 = RISC0, 2 = SP1
SELECT batch, prover_type, length(proof) as proof_size
FROM batch_proofs
WHERE batch = <BATCH_NUMBER>;

Find the nonce used for the batch:

When the L1ProofSender submits a proof to Aligned, it logs the batch number and the nonce used:

INFO ethrex_l2::sequencer::l1_proof_sender: Submitted proof to Aligned batch_number=5 nonce=42 task_id=...

Important: Pay attention to these logs and note down the batch_number → nonce mapping. The nonce is needed to verify if the gateway received the proof.

Check if Aligned has aggregated the proof on-chain:

Use the Aligned CLI's verify-on-chain command:

cd aligned_layer/aggregation_mode/cli

cargo run --release -- verify-on-chain \
  --network <NETWORK> \
  --rpc-url <RPC_URL> \
  --beacon-url <BEACON_URL> \
  --proving-system sp1 \
  --vk-hash <VK_HASH_FILE> \
  --public-inputs <PUBLIC_INPUTS_FILE>

The L1ProofVerifier also continuously checks this - if it keeps logging "has not yet been aggregated" for an extended period, the proof likely wasn't received by Aligned.

Check if the gateway received the proof:

The SDK provides get_receipts_for(address, nonce) to check if a proof is in the gateway's database:

#![allow(unused)]
fn main() {
use aligned_sdk::gateway::AggregationModeGatewayProvider;

let gateway = AggregationModeGatewayProvider::new(network);
let receipts = gateway.get_receipts_for(proof_sender_address, Some(nonce)).await?;

for receipt in receipts {
    println!("Nonce: {}, Status: {}", receipt.nonce, receipt.status);
}
}

If the proof exists locally, latest_sent shows the batch was sent, but neither the gateway has a receipt nor Aligned has aggregated it after an extended period, the proof was likely lost.

Step 2: Reset the Latest Sent Batch Pointer

The proof still exists in the database - the system just thinks it was already sent. Reset the latest_sent value to make the L1ProofSender resend it:

-- Reset to the batch before the lost one (SQL storage)
-- This will cause L1ProofSender to resend batch N on the next iteration
UPDATE latest_sent SET batch = <BATCH_NUMBER - 1> WHERE _id = 0;

For example, if batch 5 was lost:

UPDATE latest_sent SET batch = 4 WHERE _id = 0;

Note: It's safe to resend a proof even if Aligned didn't actually lose it. Aligned treats each submission with a different nonce as a separate entry, so the SDK will return Ok and queue the proof again. If the original proof was already aggregated, the L1ProofVerifier will find it when checking the commitment (which is deterministic based on vk + public inputs). The only downside is paying an extra aggregation fee for the duplicate submission.

Step 3: Wait for Automatic Resend

Once the pointer is reset:

1. L1ProofSender will detect that batch N needs to be sent
2. The existing proof will be retrieved from the database
3. The proof will be resubmitted to Aligned

No proof regeneration is needed since the proof data is still stored locally.

Prevention

• Monitor proof submission success rates
• Set up alerts for proofs stuck in "not aggregated" state for >N minutes
• Keep the quota balance funded (see Scenario 4)

Scenario 3: Aligned Permanent Shutdown

Symptoms

• Sustained inability to connect to Aligned Gateway
• Aligned team confirms permanent shutdown or migration

Impact

• Critical: Batch verification on L1 is completely blocked
• Users cannot withdraw funds (withdrawals require batch verification)
• L2 can continue producing blocks but they won't be finalized

Recovery Steps

This requires switching from Aligned mode to Standard mode.

Step 1: Stop the L2 Node

• Gracefully stop the sequencer
• This prevents new proofs from being generated in the wrong format

Step 2: Upgrade OnChainProposer Contract

The OnChainProposer contract needs to be reconfigured through a timelock upgrade. See the Upgrades documentation for the upgrade procedure.

Configuration changes:

• Set ALIGNED_MODE = false
• Enable the direct verifiers (REQUIRE_SP1_PROOF, REQUIRE_RISC0_PROOF)
• Set verifier contract addresses (SP1_VERIFIER_ADDRESS, RISC0_VERIFIER_ADDRESS)

Step 3: Clear Incompatible Proofs

Proofs generated in Compressed format (for Aligned) are incompatible with Standard mode (Groth16). Delete all unverified proofs:

-- Get the last verified batch from L1 (check OnChainProposer.lastVerifiedBatch())
-- Then delete all proofs for batches after that
DELETE FROM batch_proofs WHERE batch > <LAST_VERIFIED_BATCH>;

Step 4: Restart Node in Standard Mode

Update your node configuration to disable Aligned mode:

# Remove Aligned-specific flags
ethrex l2 \
  # ... other flags ...
  # DO NOT include: --aligned
  # DO NOT include: --aligned-network
  # DO NOT include: --aligned.beacon-url

Step 5: Regenerate Proofs in Groth16 Format

1. Restart the prover(s) - they will automatically generate Groth16 proofs (since --aligned is not set)
2. ProofCoordinator will request proofs starting from lastVerifiedBatch + 1
3. L1ProofSender will submit directly to OnChainProposer.verifyBatch()

Scenario 4: Insufficient Quota Balance

Symptoms

• Proof submission fails with insufficient balance/quota errors
• L1ProofSender logs show: AlignedSubmitProofError with insufficient quota message

The error from the Aligned SDK looks like:

Submit error: Insufficient balance, address: 0x<YOUR_PROOF_SENDER_ADDRESS>

Impact

• New proofs cannot be submitted to Aligned
• Verification stalls for new batches

Recovery Steps

Step 1: Deposit More Funds

Using the Aligned CLI from the aligned_layer repository:

cd aligned_layer/aggregation_mode/cli

cargo run --release -- deposit \
  --private-key <PROOF_SENDER_PRIVATE_KEY> \
  --network <NETWORK> \
  --rpc-url <RPC_URL>

Where <NETWORK> is one of: devnet, hoodi, or mainnet.

Prevention

• Monitor the AggregationModePaymentService contract for your address's quota balance
• Track proof submission frequency to estimate quota consumption
• Consider depositing a larger buffer to reduce maintenance frequency

Scenario 5: Proof Marked as Invalid by Aligned

Symptoms

• Logs show: "Proof is invalid, will be deleted"
• Aligned returns InvalidProof error during submission

Impact

• Invalid proof is automatically deleted from local storage
• Proof regeneration is triggered automatically

Recovery Steps

Automatic recovery - the system handles this:

1. L1ProofSender detects InvalidProof error
2. Proof is deleted from rollup store
3. ProofCoordinator detects missing proof
4. New proof is requested from prover
5. Fresh proof is submitted

Investigation

If proofs are repeatedly marked invalid:

1. Check prover version compatibility: Ensure prover ELF/VK matches the deployed contract
2. Verify public inputs: Mismatched batch data can cause invalid proofs
3. Check Aligned network: Ensure you're using the correct network (devnet/testnet/mainnet)

Monitoring and Detection

Key Log Messages

Health Check Endpoint

curl -X GET http://localhost:5555/health | jq

The response includes:

• proof_sender: L1ProofSender status and configuration
• network: Aligned network being used
• fee_estimate: Fee estimation type (instant/default)

Contract Error Codes

Summary

References

• Aligned Layer Integration
• Running Ethrex in Aligned Mode
• Aligned Layer Documentation
• Upgrades Guide

Synchronous Composability (PoC)

Status

Development branch: sync_comp_poc

Commands

Prerequisites

• A fresh-cloned ethrex repository.
• rex installed and available in your PATH. If you haven't installed it yet, follow one of the methods in the rex repository.

Run a supernode

The following command will:

1. Remove both L1 and L2 dev databases (to start from scratch).
2. Start an ethrex supernode, i.e. an L1 execution client embedded with an L2 sequencer node.

rm -rf dev_ethrex_l*; RUSTFLAGS="-Awarnings" COMPILE_CONTRACTS=true RUST_LOG=off cargo run -r -F l2,l2-sql -- l2 --supernode --block-producer.coinbase-address $(rex a -z) --committer.l1-private-key 0x850643a0224065ecce3882673c21f56bcf6eef86274cc21cadff15930b59fc8c --proof-coordinator.l1-private-key 0xf296c7802555da2a5a662be70e078cbd38b44f96f8615ae529da41122ce8db05 --eth.rpc-url http://localhost:8545 --validium --no-monitor --datadir dev_ethrex_l2 --network ./fixtures/genesis/l2.json --http.port 1729 --committer.commit-time 86400000

# Same but enabling logs

rm -rf dev_ethrex_l*; RUSTFLAGS="-Awarnings" COMPILE_CONTRACTS=true RUST_LOG=info,ethrex_p2p=error,ethrex_l2::sequencer::l1_committer=debug cargo run -r -F l2,l2-sql -- l2 --supernode --block-producer.coinbase-address $(rex a -z) --committer.l1-private-key 0x850643a0224065ecce3882673c21f56bcf6eef86274cc21cadff15930b59fc8c --proof-coordinator.l1-private-key 0xf296c7802555da2a5a662be70e078cbd38b44f96f8615ae529da41122ce8db05 --eth.rpc-url http://localhost:8545 --validium --no-monitor --datadir dev_ethrex_l2 --network ./fixtures/genesis/l2.json --http.port 1729 --committer.commit-time 86400000

Testing L1 -> L2 synchronous composability

Synchronous Deposits

rex transfer 999999999999999999 0x67cad0d689b799f385d2ebcf3a626254a9074e12 0x41443995d9eb6c6d6df51e55db2b188b12fe0f80d32817e57e11c64acff1feb8

L1 contract calling into an L2 contract

# Deploy a Counter.sol contract in the L1

rex deploy --contract-path crates/l2/contracts/src/example/Counter.sol 0 0x41443995d9eb6c6d6df51e55db2b188b12fe0f80d32817e57e11c64acff1feb8 --remappings ""

# Update that contract state by statically calling a contract in the L2

rex send 0x3fe21258005ca065695d205aac21168259e58155 "update(address)" 0x67cad0d689b799f385d2ebcf3a626254a9074e12 --private-key 0x41443995d9eb6c6d6df51e55db2b188b12fe0f80d32817e57e11c64acff1feb8

Testing L2 -> L1 synchronous composability

Synchronous Withdrawals

# Deposit
rex transfer 999999999999999999 0x67cad0d689b799f385d2ebcf3a626254a9074e12 0x41443995d9eb6c6d6df51e55db2b188b12fe0f80d32817e57e11c64acff1feb8

# Withdrawal
rex l2 withdraw 111111111111111111 0x41443995d9eb6c6d6df51e55db2b188b12fe0f80d32817e57e11c64acff1feb8

Introduction

L1 -> L2 Synchronous Composability

Synchronous Deposits

Deposits are the process by which L1 users can enter L2 in some form. This process begins and ends on L1 through a series of steps:

1. Initiate the deposit on L1:
   • A user sends a transaction to L1, either via an ETH transfer to the CommonBridge contract or by calling the deposit function on the same contract. Both actions execute the same logic, which, upon successful execution, emits a log containing the necessary information for the sequencer of the corresponding L2 to process it.
   • This transaction must be included in a block, and that block must be finalized for the sequencer on the corresponding L2 to detect the log on L1.
2. Process the deposit on L2:
   • When the sequencer processes this log, it includes a transaction in its mempool that mints the corresponding ETH to the recipient's address, thereby ensuring the recipient has funds on L2.
3. Commit the deposit process from L2 to L1:
   • Eventually, the L2 batch that includes this mint transaction is sealed and committed to L1. This commit transaction must be included in an L1 block and finalized.
   • The same batch is sent to a prover to generate a ZK proof validating the previously committed batch.
4. Verify the deposit process from L2 to L1 (deposit finalization):
   • Eventually, the batch execution proof is generated and returned to the sequencer, which submits it for verification on L1 via a verify transaction.
   • The verify transaction, assuming it is valid, must be included in an L1 block and finalized.

This 4-step process requires, by definition, that it occur across different L1 slots. The number of slots needed can vary based on L1's configuration, but even assuming a sufficiently fast commit time, real-time proving to generate the proof quickly, and a sufficiently fast proof submission time, this process would still require at least 2 slots: the first is always mandatory to emit the log that the sequencer listens for, and with significant luck, finalization could occur in the next slot.

Synchronous Composability enables this entire process to happen within the same L1 slot. In other words, the transaction that initiates the deposit, the deposit processing on L2, the commit transaction for the batch that included the mint, the generation of the execution proof for that batch, and the verify transaction for the same batch all occur in the same L1 slot.

L1 Contract Calling into an L2 Contract

Another capability enabled by synchronous composability is the ability to call L2 contracts from L1.

A simple example of this would be updating the state of a counter on L1 with the current state of another counter that resides on L2.

Unlike deposits, which do not require synchronous composability to function normally, calling an L2 contract from L1 and using the result as part of the L1 execution is not possible without this feature.

L2 -> L1 Synchronous Composability

TBD

Rollup Requirements for SC and How We Addressed Them in the PoC

To achieve synchronous composability, our rollup needed to fulfill the following requirements:

1. Reorg with L1: The rollup consumes unconfirmed L1 data and therefore must reorganize (reorg) with L1.
2. Instant Settlement: The rollup must be able to settle within one L1 slot, requiring real-time proving.
3. Coordinated Sequencing: The L2 proposer is the L1 proposer or works closely together (e.g., issues L1 inclusion preconfs).

We addressed these requirements in the following manner:

1. For this PoC, we removed reorgs from the equation.
2. Our L2 block builder would force a commit batch transaction after building a block that includes a scoped call. Assuming real-time proving by skipping verification, the commit transaction now serves as a settlement transaction.
3. We extended the ethrex functionality with a supernode mode that operates essentially as an L1 and L2 node sharing both states. This allows the L1 to insert transactions into the L2 mempool and simulate the L2 state in real time, while the L2 can insert transactions into the L1 mempool and simulate the L1 state in real time.

Future work

TBD

Deploying an ethrex L2 with shared bridge enabled

In this section, we'll cover how to deploy two ethrex L2 with shared bridge enabled on a devnet.

Prerequisites

This guide assumes that you have the ethrex repository cloned.

Steps

Change directory

Every command should be run under crates/l2

cd crates/l2

Start an L1

make init-l1

Deploy the first L2

On another terminal

ETHREX_SHARED_BRIDGE_DEPLOY_ROUTER=true make deploy-l1

Start the first L2

Replace L1_BRIDGE_ADDRESS, L1_ON_CHAIN_PROPOSER_ADDRESS and ROUTER_ADDRESS with the outputs of the previous command, you can also check it under cmd/.env.

../../target/release/ethrex \
	l2 \
	--watcher.block-delay 0 \
	--network ../../fixtures/genesis/l2.json \
	--http.port 1729 \
	--http.addr 0.0.0.0 \
	--metrics \
	--metrics.port 3702 \
	--datadir dev_ethrex_l2 \
	--l1.bridge-address <L1_BRIDGE_ADDRESS> \
	--l1.on-chain-proposer-address <L1_ON_CHAIN_PROPOSER_ADDRESS> \
	--eth.rpc-url http://localhost:8545 \
	--osaka-activation-time 1761677592 \
	--block-producer.coinbase-address 0x0007a881CD95B1484fca47615B64803dad620C8d \
	--block-producer.base-fee-vault-address 0x000c0d6b7c4516a5b274c51ea331a9410fe69127 \
	--block-producer.operator-fee-vault-address 0xd5d2a85751b6F158e5b9B8cD509206A865672362 \
	--block-producer.operator-fee-per-gas 1000000000 \
	--committer.l1-private-key 0x385c546456b6a603a1cfcaa9ec9494ba4832da08dd6bcf4de9a71e4a01b74924 \
	--proof-coordinator.l1-private-key 0x39725efee3fb28614de3bacaffe4cc4bd8c436257e2c8bb887c4b5c4be45e76d \
	--proof-coordinator.addr 127.0.0.1 \
    --l1.router-address <ROUTER_ADDRESS> \
    --watcher.l2-rpcs http://localhost:1730 \
    --watcher.l2-chain-ids 1730

Deploy the second L2

On another terminal

Copy the ../../fixtures/genesis/l2.json file to ../../fixtures/genesis/l2_2.json and modify chain id to 1730

Replace ROUTER_ADDRESS with the outputs of the first deploy

../../target/release/ethrex l2 deploy \
	--eth-rpc-url http://localhost:8545 \
	--private-key 0x385c546456b6a603a1cfcaa9ec9494ba4832da08dd6bcf4de9a71e4a01b74924 \
	--on-chain-proposer-owner 0x4417092b70a3e5f10dc504d0947dd256b965fc62 \
	--bridge-owner 0x4417092b70a3e5f10dc504d0947dd256b965fc62 \
	--deposit-rich \
	--private-keys-file-path ../../fixtures/keys/private_keys_l1.txt \
	--genesis-l1-path ../../fixtures/genesis/l1.json \
	--genesis-l2-path ../../fixtures/genesis/l2_2.json \
    --randomize-contract-deployment \
    --router.address <ROUTER_ADDRESS>

Start the second L2

Replace L1_BRIDGE_ADDRESS and L1_ON_CHAIN_PROPOSER_ADDRESS with the outputs of the previous command, you can also check it under cmd/.env. And ROUTER_ADDRESS with the outputs of the first deploy

../../target/release/ethrex \
	l2 \
	--watcher.block-delay 0 \
	--network ../../fixtures/genesis/l2_2.json \
	--http.port 1730 \
	--http.addr 0.0.0.0 \
	--metrics \
	--metrics.port 3703 \
	--datadir dev_ethrex_l2_2 \
	--l1.bridge-address <L1_BRIDGE_ADDRESS> \
	--l1.on-chain-proposer-address <L1_ON_CHAIN_PROPOSER_ADDRESS> \
	--eth.rpc-url http://localhost:8545 \
	--osaka-activation-time 1761677592 \
	--block-producer.coinbase-address 0x0007a881CD95B1484fca47615B64803dad620C8d \
	--block-producer.base-fee-vault-address 0x000c0d6b7c4516a5b274c51ea331a9410fe69127 \
	--block-producer.operator-fee-vault-address 0xd5d2a85751b6F158e5b9B8cD509206A865672362 \
	--block-producer.operator-fee-per-gas 1000000000 \
	--committer.l1-private-key 0x385c546456b6a603a1cfcaa9ec9494ba4832da08dd6bcf4de9a71e4a01b74924 \
	--proof-coordinator.l1-private-key 0x39725efee3fb28614de3bacaffe4cc4bd8c436257e2c8bb887c4b5c4be45e76d \
	--proof-coordinator.addr 127.0.0.1 \
    --proof-coordinator.port 3901 \
    --l1.router-address <ROUTER_ADDRESS> \
    --watcher.l2-rpcs http://localhost:1729 \
    --watcher.l2-chain-ids 65536999

Start the prover

On another terminal

../../target/release/ethrex \
	l2 prover \
	--proof-coordinators tcp://127.0.0.1:3900 tcp://127.0.0.1:3901 \
	--backend exec

Deploying a fee token

Upgrades

From v7 to v8

Database migration (local node)

This migration applies to the L2 node database only (SQL-backed store). It does not change any on-chain contracts.

The messages table was renamed to l1_messages. Copy the data and then remove the old table:

INSERT INTO l1_messages
SELECT *
FROM messages;

Then delete the messages table.

From v8 to v9

Timelock upgrade (L1 contracts)

From ethrex v9 onwards, the Timelock contract manages access to the OnChainProposer (OCP). The OCP owner becomes the Timelock, and the deprecated authorizedSequencerAddresses mapping is replaced by Timelock roles.

What changes

• Sequencer permissions move to Timelock roles (SEQUENCER).
• Commit and verify transactions must target the Timelock, not the OCP.
• Governance and Security Council accounts control upgrades and emergency actions via the Timelock.

1) Deploy Timelock (proxy + implementation)

Deploy a Timelock proxy and implementation using your standard UUPS deployment flow. Record the proxy address; that is the address you will initialize and use later.

2) Initialize Timelock

Call the Timelock initializer on the proxy:

initialize(uint256 minDelay,address[] sequencers,address governance,address securityCouncil,address onChainProposer)

• sequencers should include the L1 committer and proof sender addresses (and any other accounts that should commit or verify).
• securityCouncil is typically the current OCP owner (ideally a multisig).
• governance is the account that will schedule and execute timelocked upgrades.

3) Transfer OCP ownership to Timelock

From the current OCP owner, transfer ownership with transferOwnership(address).

Then accept ownership from the Timelock:

• Normal path: schedule and execute acceptOwnership() through the Timelock (respects minDelay).
• Emergency path: the Security Council can call emergencyExecute on the Timelock with calldata for acceptOwnership() to accept immediately.

Note: acceptOwnership() must be executed by the Timelock (the pending owner), so it cannot be called directly from an EOA.

4) Configure the L2 node to use Timelock

This is required because the sequencer can no longer call the OCP directly once the Timelock is the owner. Set the Timelock address so commit and verify calls target the Timelock:

• CLI flag: --l1.timelock-address <TIMELOCK_PROXY_ADDRESS>
• Env var: ETHREX_TIMELOCK_ADDRESS=<TIMELOCK_PROXY_ADDRESS>

The committer requires this address for non-based deployments, and the proof sender/verifier will use it when present.

Do this before restarting the sequencer after the ownership transfer. If the node keeps targeting the OCP after the transfer, commit/verify calls will revert (onlyOwner). If you point to the Timelock before the transfer, the Timelock will forward but the OCP will still reject it because the Timelock is not the owner yet.

5) Verify the migration

• OCP owner() returns the Timelock address.
• Sequencer addresses return true for hasRole(SEQUENCER, <addr>) on the Timelock.
• The L2 node logs show commit/verify txs sent to the Timelock.

Database migration (local node)

This migration applies to the L2 node database only (SQL-backed store). It does not change any on-chain contracts.

The balance_diffs table added a new value_per_token column of type BLOB:

ALTER TABLE balance_diffs
ADD COLUMN value_per_token BLOB;

Run a prover

This section provides step-by-step guides for running an ethrex L2 prover, which is responsible for generating ZK proofs for L2 blocks. These proofs are then submitted to L1 for verification, finalizing the state of your L2.

Use this section to choose which prover setup best fits your already deployed ethrex L2 and follow the instructions accordingly.

• Overview
• Running an ethrex L2 SP1 prover
• Running an ethrex L2 RISC0 prover
• Running an ethrex L2 TDX prover
• Running multiple provers

Run an ethrex prover

Deploying the ethrex L2 contracts on L1 and starting the node isn't everything when it comes to setting up your full ethrex L2 stack.

If you've been following the deployment guide, you should already have an ethrex L2 node running and connected to L1. If that's not the case, I recommend reviewing that guide before proceeding.

The next step is to run the prover—the component responsible for generating ZK proofs for the L2 blocks. These proofs will then be sent to L1 for verification, finalizing the state of your L2.

In this section, we'll cover how to run one or more ethrex L2 provers.

Before proceeding, note that this guide assumes you have ethrex installed. If you haven't installed it yet, follow one of the methods in the Installation Guide. If you're looking to build from source, don't skip this section—we'll cover that method here, as it is independent of the deployment approach you choose later.

Building from source (skip if ethrex is already installed)

Prerequisites

Ensure you have the following installed on your system:

• Rust and Cargo (install via rustup)
• Solidity compiler v0.8.31 (refer to Solidity documentation)
• SP1 Toolchain (if you plan to use SP1 proving, refer to SP1 documentation)
• RISC0 Toolchain (if you plan to use RISC0 proving, refer to RISC0 documentation)
• CUDA Toolkit 12.9 (if you plan to use GPU acceleration for SP1 or RISC0 proving)

1. Clone the official ethrex repository:

   git clone https://github.com/lambdaclass/ethrex
   cd ethrex
   

2. Install the binary to your $PATH:

   # For SP1 CPU proving (very slow, not recommended)
   cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1
   
   # For RISC0 CPU proving (very slow, not recommended)
   cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,risc0
   
   # For SP1 and RISC0 CPU proving (very slow, not recommended)
   cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,risc0
   
   # For SP1 GPU proving
   cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,gpu
   
   # For RISC0 GPU proving
   cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,risc0,gpu
   
   # For SP1 and RISC0 GPU proving
   cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,risc0,gpu
   

   cargo install places the binary at ~/.cargo/bin/ethrex; ensure that directory is on your $PATH. Add --force if you need to reinstall.

PROVER_REPRODUCIBLE_BUILD=true COMPILE_CONTRACTS=true cargo install --locked --path cmd/ethrex --bin ethrex --features l2,l2-sql,sp1,risc0,gpu

Run an ethrex L2 SP1 prover

In this section, we'll guide you through the steps to run an ethrex L2 prover that utilizes SP1 for generating ZK proofs. These proofs are essential for validating batch execution and state settlement on your ethrex L2.

Prerequisites

• This guide assumes that you have ethrex installed with the SP1 feature and available in your PATH. If you haven't installed it yet, follow one of the methods in the Installation Guide. If you want to build the binary from source, refer to the Building from source section and select the appropriate build option.
• This guide also assumes that you have already deployed an ethrex L2 with SP1 enabled. If you haven't done so yet, please refer to one of the Deploying an ethrex L2 guides.

Start an ethrex L2 SP1 prover

Once you have your ethrex L2 deployed with SP1 enabled, you can start the SP1 prover using the following command:

ethrex l2 prover \
--backend sp1 \
--proof-coordinators http://localhost:3900

Troubleshooting

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

If you encounter the following error when starting the SP1 prover with GPU support:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

This error indicates that Docker is unable to find a suitable GPU driver for running containers with GPU support. To resolve this issue, follow these steps:

1. Install NVIDIA Container Toolkit: Ensure that you have the NVIDIA Container Toolkit installed on your system. This toolkit allows Docker to utilize NVIDIA GPUs. You can follow the installation instructions from the official NVIDIA documentation.
2. Configure Docker to use the NVIDIA runtime: After installing the NVIDIA Container Toolkit, you need to configure Docker to use the NVIDIA runtime by default. You can do this by following the instructions in the Configuring Docker documentation.

Run an ethrex L2 RISC0 prover

In this section, we'll guide you through the steps to run an ethrex L2 prover that utilizes RISC0 for generating ZK proofs. These proofs are essential for validating batch execution and state settlement on your ethrex L2.

Prerequisites

• This guide assumes that you have ethrex installed with the RISC0 feature and available in your PATH. If you haven't installed it yet, follow one of the methods in the Installation Guide. If you want to build the binary from source, refer to the Building from source section and select the appropriate build option.
• This guide also assumes that you have already deployed an ethrex L2 with RISC0 enabled. If you haven't done so yet, please refer to one of the Deploying an ethrex L2 guides.

Start an ethrex L2 RISC0 prover

Once you have your ethrex L2 deployed with RISC0 enabled, you can start the RISC0 prover using the following command:

ethrex l2 prover \
--backend risc0 \
--proof-coordinators http://localhost:3900

Run an ethrex TDX prover

In this section, we'll guide you through the steps to run an ethrex L2 TDX prover for generating TEE proofs. These proofs are essential for validating batch execution and state settlement on your ethrex L2.

Prerequisites

• This guide assumes that you have already deployed an ethrex L2 with TDX enabled. If you haven't done so yet, please refer to one of the Deploying an ethrex L2 guides.
• A machine with TDX support with the required setup.

Start an ethrex L2 TDX prover

There's no official release of our ethrex L2 TDX prover yet, so you need to build ethrex from source. To do this, clone the ethrex repository and run:

git clone https://github.com/lambdaclass/ethrex.git

cd ethrex/crates/l2/tee/quote-gen

make run

Run multiple provers

In this section, we'll guide you through the steps to run multiple ethrex L2 provers for generating ZK proofs using different backends. These proofs are essential for validating batch execution and state settlement on your ethrex L2.

Prerequisites

• This guide assumes that you have already deployed an ethrex L2 with TDX enabled. If you haven't done so yet, please refer to one of the Deploying an ethrex L2 guides.

Start multiple ethrex L2 provers

Once you have your ethrex L2 deployed with multiple proving backends enabled (SP1, RISC0, TDX), refer to the following guides to start each prover:

• Run an ethrex L2 SP1 prover
• Run an ethrex L2 RISC0 prover
• Run an ethrex TDX prover

Each prover should be started in different machines to ensure they operate independently and efficiently. Make sure to configure each prover with the appropriate backend flag and proof coordinator URLs as specified in their respective guides.

Monitoring and Metrics

Ethrex exposes metrics in Prometheus format on port 9090 by default. The easiest way to monitor your node is to use the provided Docker Compose stack, which includes Prometheus and Grafana preconfigured.

Quickstart: Monitoring Stack with Docker Compose

1. Clone the repository:

   git clone https://github.com/lambdaclass/ethrex.git
   cd ethrex/metrics
   

2. Start the monitoring stack:

   docker compose -f docker-compose-metrics.yaml -f docker-compose-metrics-l2.overrides.yaml up -d
   

This will launch Prometheus and Grafana, already set up to scrape ethrex metrics.

Accessing Metrics and Dashboards

• Prometheus: http://localhost:9091
• Grafana: http://localhost:3001
  • Default login: admin / admin
  • Prometheus is preconfigured as a data source
  • Example dashboards are included in the repo

Metrics from ethrex will be available at http://localhost:9090/metrics in Prometheus format.

Custom Configuration

Your ethrex setup may differ from the default configuration. Check your endpoints at provisioning/prometheus/prometheus_l2.yaml.

For manual setup or more details, see the Prometheus documentation and Grafana documentation.

Admin API

This API exposes endpoints to manage the Sequencer.

Base URL

By default the server is listening on 127.0.0.1:5555 but can be configured with --admin-server.addr <address> --admin-server.port <port>

Endpoints

Health

Sequencer Health

Description

Performs a healthcheck on all the components of the sequencer returning a json with the status

Endpoint

GET /health

Example

curl -X GET http://localhost:5555/health

Admin server health

Description

Performs a healthcheck on the http admin server

Endpoint

GET /admin/health

Example

curl -X GET http://localhost:5555/admin/health

L1 Committer

Start Committer immediately

Description

Starts the committer immediately (with a delay of 0).

Endpoint

GET /committer/start

Example

curl -X GET http://localhost:5555/committer/start

Start Committer (with delay)

Description

Starts the committer with a configurable delay.

Endpoint

GET /committer/start/{delay}

Example

curl -X GET http://localhost:5555/committer/start/60000

Parameters

Stop Committer

Description

Stops the committer.

Endpoint

GET /committer/stop

Example

curl -X GET http://localhost:5555/committer/stop

Rollup Stages and ethrex

This document explains how the L2Beat rollup stage definitions map to the current ethrex L2 stack.

Important Distinctions

Stages are properties of a deployed L2, whereas ethrex is a framework that different projects may configure and govern in their own way. In what follows we make two simplifying assumptions:

• If ethrex provides the functionality required to deploy a Stage X rollup, we consider ethrex capable of achieving Stage X, even if a particular deployment chooses not to enable some features.
• When we talk about ethrex L2 we are referring to ethrex in rollup mode, not Validium. In rollup mode, Ethereum L1 is the data availability layer; in Validium mode it is not.

The L2Beat framework evaluates decentralization specifically, not security from bugs. A Stage 2 rollup could still have vulnerabilities if the proof system is experimental or unaudited.

Stage 0

Stage 0 ("Full Training Wheels") represents the basic operational requirements for a rollup.

Summary

Detailed Analysis

Does the project call itself a rollup?

Yes. As stated in the introduction:

Ethrex is a framework that lets you launch your own L2 rollup or blockchain.

Are L2 state roots posted on L1?

Yes. Every time a batch is committed to the OnChainProposer on L1, the new L2 state root is sent and stored in the batchCommitments mapping as newStateRoot for that batch.

Does the project provide data availability on L1?

Yes. When committing a batch in rollup mode (non-validium), the transaction must include a non-zero blob hash, so a blob MUST be sent to the OnChainProposer on L1.

• The architecture docs state that the blob contains the RLP-encoded L2 blocks and fee configuration
• The blob commitment (blobKZGVersionedHash) is included in the batch commitment and re-checked during proof verification

This means all data needed to reconstruct the L2 (transactions and state) is published on L1 as blobs.

Is software capable of reconstructing the rollup's state available?

Yes.

• The L2 node provides the ethrex l2 reconstruct subcommand to follow L1 commitments and reconstruct the state from blobs
• The state reconstruction blobs doc explains how to generate and use blobs for replaying state
• The data availability and prover docs describe how published data is used to reconstruct and verify state

Does the project use a proper proof system?

Yes, assuming proofs are enabled.

ethrex supports multiple proving mechanisms:

• zkVM validity proofs: SP1 and RISC0
• TDX attestations: TEE-based verification
• Aligned Layer: Optional proof aggregation for cost efficiency

The OnChainProposer contract can be configured to require many combinations of these mechanisms. A batch is only verified on L1 if all configured proofs pass and their public inputs match the committed data (state roots, withdrawals, blobs, etc.).

Are there at least 5 external actors that can submit a fraud proof?

Not applicable. ethrex uses validity proofs, not fraud proofs. There is no on-chain "challenge game" where watchers submit alternate traces to invalidate a state root.

Stage 0 Assessment

ethrex L2 meets all Stage 0 requirements.

Stage 1

Stage 1 ("Limited Training Wheels") requires that users have trustless exit guarantees, with a Security Council retaining only limited emergency powers.

Core Principle

"The only way (other than bugs) for a rollup to indefinitely block an L2→L1 message or push an invalid L2→L1 message is by compromising ≥75% of the Security Council."

Summary

Detailed Analysis

Can L2→L1 messages be censored?

Yes, this is the main Stage 1 gap.

The sequencer can indefinitely block/censor an L2→L1 message (e.g., a withdrawal) by simply not including the withdrawal transaction in an L2 block. This does not require compromising the owner/Security Council.

What's missing: A forced-inclusion mechanism where users can submit their withdrawal directly on L1, and the sequencer must include it in a subsequent batch within a bounded time window or lose sequencing rights.

Can the sequencer push invalid L2→L1 messages?

No. The sequencer cannot unilaterally make L1 accept an invalid L2→L1 message. This would require:

• Changing contract code
• Updating the verifying key in OnChainProposer

Only the Security Council (owner) can perform those upgrades.

What about upgrades from entities outside the Security Council?

In ethrex L2 contracts, there are no entities other than the owner that can perform upgrades. Therefore:

• Upgrades initiated by entities outside the Security Council are not possible
• If such an upgrade path were introduced, it would need to provide the required 7-day exit window

Security Council Configuration

Both OnChainProposer and CommonBridge are upgradeable contracts controlled by a single owner address. ethrex itself does not hard-code a Security Council, but a deployment can introduce one by making the owner a multisig.

According to L2Beat requirements, the Security Council should have:

• At least 8 members
• ≥75% threshold for critical actions
• Diverse signers from different organizations/jurisdictions

Stage 1 Assessment

ethrex L2 does not meet Stage 1 requirements today.

The main gap is censorship-resistant L2→L1 messages. The sequencer can ignore withdrawal transactions indefinitely, and there is no forced-inclusion mechanism (unlike the existing forced-inclusion mechanism for deposits).

Path to Stage 1

To achieve Stage 1, ethrex would need:

1. Forced withdrawal inclusion: Implement an L1 mechanism where users can submit withdrawals directly, with sequencer penalties for non-inclusion
2. Security Council multisig: Deploy owner as an 8+ member multisig with ≥75% threshold
3. Exit window enforcement: ethrex has Timelock functionality that gates the OnChainProposer, but deployment configuration must enforce ≥7 day delays for non-emergency upgrades

Stage 2

Stage 2 ("No Training Wheels") requires fully permissionless proving and tightly constrained emergency upgrade powers.

Summary

Detailed Analysis

Is the validity proof system permissionless?

No. In the standard OnChainProposer implementation (crates/l2/contracts/src/l1/OnChainProposer.sol), committing and verifying batches are restricted to authorized sequencer addresses only. Submitting proofs is not permissionless.

Do users have at least 30 days to exit before unwanted upgrades?

No. There is no protocol-level exit window tied to contract upgrades. UUPS upgrades can be executed by the owner without a mandatory delay.

Is the Security Council restricted to act only due to on-chain errors?

No. There is no built-in restriction that limits the owner to responding only to detected on-chain bugs. The owner can pause or upgrade contracts for any reason.

Stage 2 Assessment

ethrex L2 does not meet Stage 2 requirements.

Path to Stage 2

To achieve Stage 2, ethrex would need (in addition to Stage 1 requirements):

1. Permissionless proving: Allow anyone to submit validity proofs for batches
2. 30-day exit window: Implement mandatory delay for all contract upgrades
3. Restricted SC powers: Limit Security Council actions to adjudicable on-chain bugs only
4. Mature proof system: Battle-tested ZK provers with comprehensive security audits

Comparison with Other Rollups

Based Rollups

Based rollups delegate sequencing to Ethereum L1 validators rather than using a centralized sequencer. This is particularly relevant for ethrex as it implements based sequencing (currently in development).

Taiko Alethia is the first based rollup on mainnet. It requires two proofs per block: SGX (Geth) is mandatory, plus one of SGX (Reth), SP1, or RISC0. Critically, blocks can be proven with TEE only (no ZK) if both SGX verifiers are used. As of early 2025, only ~30% of blocks use ZK proofs. L2BEAT warns that "funds can be stolen if a malicious block is proven by compromised SGX instances." Taiko plans to require 100% ZK coverage with the Shasta fork in Q4 2025.

*L2BEAT currently classifies Taiko as "not even Stage 0" because "the proof system is still under development." However, Taiko has been a multi-prover based rollup since the Pacaya fork and the system is architecturally prepared for Stage 0. This appears to be a classification nuance rather than a fundamental gap.

Surge is a based rollup template by Nethermind, built on the Taiko stack and designed to target Stage 2 from inception. It removes centralized sequencing entirely, letting Ethereum validators handle transaction ordering. Not yet deployed as a production rollup.

ZK Rollups

Scroll became the first ZK rollup to achieve Stage 1 (April 2025) through the Euclid upgrade, which introduced permissionless sequencing fallback and a 12-member Security Council with 75% threshold.

zkSync Era is currently experiencing a proof system pause due to a vulnerability, causing partial liveness failure. Previously, a critical bug in zk-circuits was discovered that could have led to $1.9B in potential losses if exploited.

*L2BEAT states they "haven't finished evaluation" of zkSync Era's Stage 1 elements - not that zkSync fails requirements. The main pending item is a forced inclusion mechanism. With 75% of proving already delegated to external provers and decentralized sequencing (ChonkyBFT) underway, zkSync appears architecturally Stage 1-ready.

Starknet reached Stage 1 but shares its SHARP verifier with other StarkEx rollups. The verifier can be changed by a 2/4 multisig with 8-day delay. The Security Council (9/12) retains instant upgrade capability. This shared verifier creates concentration risk across multiple chains.

Optimistic Rollups

Arbitrum One uses BoLD (Bounded Liquidity Delay) for permissionless fraud proofs - anyone can challenge state assertions. However, Arbitrum remains Stage 1 because the Security Council retains broad override powers. Stage 2 requires restricting SC to "provable bugs only" and extending exit windows to 30 days. The ~6.4 day withdrawal delay is inherent to the optimistic model.

Optimism has permissionless fault proofs but L2BEAT notes: "There is no exit window for users to exit in case of unwanted regular upgrades as they are initiated by the Security Council with instant upgrade power." Both Arbitrum and Optimism are technically ready for Stage 2 but held back by intentional governance constraints, not technical limitations.

L2BEAT Risk Summary

Key Observations

1. No rollup has achieved Stage 2 yet - All production rollups remain at Stage 0 or Stage 1
2. Classification vs architecture gaps - Some rollups (Taiko, zkSync Era) are classified lower than their architecture supports due to L2BEAT evaluation timing or minor gaps
3. Governance is the bottleneck - Arbitrum and Optimism have permissionless proofs but are held at Stage 1 by intentional Security Council powers, not technical limitations
4. Based rollups are newer - Taiko and ethrex are pioneering based sequencing, both at Stage 0
5. Multi-proof is emerging - ethrex, Taiko, and Scroll are all exploring multi-proof systems for enhanced security

Recommendations

For Stage 1 Compliance

1. Implement forced withdrawal inclusion

   • Users can submit withdrawal requests directly to L1
   • Sequencer must include within N blocks or face penalties
   • Fallback mechanism if sequencer fails to include

2. Deploy Security Council as multisig

   • 8+ diverse signers
   • 75%+ threshold (e.g., 6/8)
   • Document emergency procedures

3. Add upgrade timelock

   • Minimum 7-day delay for non-emergency upgrades
   • Emergency path requires SC threshold

For Future Stage 2 Transition

1. Open proof submission

   • Remove sequencer-only restriction on verifyBatch()
   • Anyone can submit valid proofs

2. Extend exit window to 30+ days

   • Mandatory delay on all upgrade paths
   • Clear user notification mechanism

3. Formalize SC restrictions

   • On-chain governance limiting SC powers
   • Transparent criteria for emergency actions

4. Proof system maturity

   • Comprehensive security audits
   • Multiple independent prover implementations
   • Operational track record

Conclusion

ethrex L2 currently satisfies all Stage 0 requirements and provides a solid foundation for rollup deployments.

The path to Stage 1 is clear but requires implementing censorship-resistant withdrawals through a forced-inclusion mechanism. This is the primary gap preventing Stage 1 compliance.

Stage 2 requires additional work on permissionless proving, extended exit windows, and formal restrictions on Security Council powers.

References

L2Beat Resources

• L2Beat Stages Framework
• L2Beat Forum: Stages Update
• L2Beat: Introducing Stages

Rollup Comparisons

• Taiko Alethia on L2Beat
• Scroll on L2Beat
• zkSync Era on L2Beat
• Starknet on L2Beat
• Arbitrum One on L2Beat
• Optimism on L2Beat
• Surge: A Based Rollup Template

ethrex Documentation

• ethrex Data Availability
• ethrex Withdrawals
• ethrex Based Sequencing

Architecture

This section provides an overview of the architecture of an L2 rollup built with ethrex. Here you'll find:

• High-level diagrams and explanations of the main components
• Details on how the sequencer, prover, and other modules interact
• Information about aligned mode, the prover, the sequencer, and more

Use this section to understand how the different parts of an ethrex L2 fit together. The overview is a good place to start.

General overview of the ethrex L2 stack

This document aims to explain how the Lambda ethrex L2 and all its moving parts work.

Intro

At a high level, the way an L2 works is as follows:

• There is a contract in L1 that tracks the current state of the L2. Anyone who wants to know the current state of the chain need only consult this contract.
• Every once in a while, someone (usually the sequencer, but could be a decentralized network, or even anyone at all in the case of a based contestable rollup) builds a batch of new L2 blocks and publishes it to L1. We will call this the commit L1 transaction.
• For L2 batches to be considered finalized, a zero-knowledge proof attesting to the validity of the batch needs to be sent to L1, and its verification needs to pass. If it does, everyone is assured that all blocks in the batch were valid and thus the new state is. We call this the verification L1 transaction.

We omitted a lot of details in this high level explanation. Some questions that arise are:

• What does it mean for the L1 contract to track the state of L2? Is the entire L2 state kept on it? Isn't it really expensive to store a bunch of state on an Ethereum smart contract?
• What does the ZK proof prove exactly?
• How do we make sure that the sequencer can't do anything malicious if it's the one proposing blocks and running every transaction?
• How does someone go in and out of the L2, i.e., how do you deposit money from L1 into L2 and then withdraw it? How do you ensure this can't be tampered with? Bridges are by far the most vulnerable part of blockchains today and going in and out of the L2 totally sounds like a bridge.

Below some answers to these questions, along with an overview of all the moving parts of the system.

How do you prove state?

Now that general purpose zkVMs exist, most people have little trouble with the idea that you can prove execution. Just take the usual EVM code you wrote in Rust, compile to some zkVM target instead and you're mostly done. You can now prove it.

What's usually less clear is how you prove state. Let's say we want to prove a new L2 batch of blocks that were just built. Running the ethrex execute_block function on a Rust zkVM for all the blocks in the batch does the trick, but that only proves that you ran the VM correctly on some previous state/batch. How do you know it was the actual previous state of the L2 and not some other, modified one?

In other words, how do you ensure that:

• Every time the EVM reads from some storage slot (think an account balance, some contract's bytecode), the value returned matches the actual value present on the previous state of the chain.

For this, the VM needs to take as a public input the previous state of the L2, so the prover can show that every storage slot it reads is consistent with it, and the verifier contract on L1 can check that the given public input is the actual previous state it had stored. However, we can't send the entire previous state as public input because it would be too big; this input needs to be sent on the verification transaction, and the entire L2 state does not fit on it.

To solve this, we do what we always do: instead of having the actual previous state be the public input, we build a Merkle Tree of the state and use its root as the input. Now the state is compressed into a single 32-byte value, an unforgeable representation of it; if you try to change a single bit, the root will change. This means we now have, for every L2 batch, a single hash that we use to represent it, which we call the batch commitment (we call it "commitment" and not simply "state root" because, as we'll see later, this won't just be the state root, but rather the hash of a few different values including the state root).

The flow for the prover is then roughly as follows:

• Take as public input the previous batch commitment and the next (output) batch commitment.
• Execute all blocks in the batch to prove its execution is valid. Here "execution" means more than just transaction execution; there's also header validation, transaction validation, etc. (essentially all the logic ethrex needs to follow when executing and adding a new block to the chain).
• For every storage slot read, present and verify a merkle path from it to the previous state root (i.e. previous batch commitment).
• For every storage slot written, present and verify a merkle path from it to the next state root (i.e. next batch commitment).

As a final note, to keep the public input a 32 byte value, instead of passing the previous and next batch commitments separately, we hash the two of them and pass that. The L1 contract will then have an extra step of first taking both commitments and hashing them together to form the public input.

These two ideas will be used extensively throughout the rest of the documentation:

• Whenever we need to add some state as input, we build a merkle tree and use its root instead. Whenever we use some part of that state in some way, the prover provides merkle paths to the values involved. Sometimes, if we don't care about efficient inclusion proofs of parts of the state, we just hash the data altogether and use that instead.
• To keep the batch commitment (i.e. the value attesting to the entire state of the chain) a 32 byte value, we hash the different public inputs into one. The L1 contract is given all the public inputs on commit, checks their validity and then squashes them into one through hashing.

Reconstructing state/Data Availability

While using a merkle root as a public input for the proof works well, there is still a need to have the state on L1. If the only thing that's published to it is the state root, then the sequencer could withhold data on the state of the chain. Because it is the one proposing and executing blocks, if it refuses to deliver certain data (like a merkle path to prove a withdrawal on L1), people may not have any place to get it from and get locked out of the chain or some of their funds.

This is called the Data Availability problem. As discussed before, sending the entire state of the chain on every new L2 batch is impossible; state is too big. As a first next step, what we could do is:

• For every new L2 batch, send as part of the commit transaction the list of transactions in the batch. Anyone who needs to access the state of the L2 at any point in time can track all commit transactions, start executing them from the beginning and reconstruct the state.

This is now feasible; if we take 200 bytes as a rough estimate for the size of a single transfer between two users (see this post for the calculation on legacy transactions) and 128 KB as a reasonable transaction size limit we get around ~650 transactions at maximum per commit transaction (we are assuming we use calldata here, blobs can increase this limit as each one is 128 KB and we could use multiple per transaction).

Going a bit further, instead of posting the entire transaction, we could just post which accounts have been modified and their new values (this includes deployed contracts and their bytecode of course). This can reduce the size a lot for most cases; in the case of a regular transfer as above, we only need to record balance updates of two accounts, which requires sending just two (address, balance) pairs, so (20 + 32) * 2 = 104 bytes, or around half as before. Some other clever techniques and compression algorithms can push down the publishing cost of this and other transactions much further.

This is called state diffs. Instead of publishing entire transactions for data availability, we only publish whatever state they modified. This is enough for anyone to reconstruct the entire state of the chain.

Detailed documentation on the state diffs spec.

How do we prevent the sequencer from publishing the wrong state diffs?

Once again, state diffs have to be part of the public input. With them, the prover can show that they are equal to the ones returned by the VM after executing all blocks in the batch. As always, the actual state diffs are not part of the public input, but their hash is, so the size is a fixed 32 bytes. This hash is then part of the batch commitment. The prover then assures us that the given state diff hash is correct (i.e. it exactly corresponds to the changes in state of the executed blocks).

There's still a problem however: the L1 contract needs to have the actual state diff for data availability, not just the hash. This is sent as part of calldata of the commit transaction (actually later as a blob, we'll get to that), so the sequencer could in theory send the wrong state diff. To make sure this can't happen, the L1 contract hashes it to make sure that it matches the actual state diff hash that is included as part of the public input.

With that, we can be sure that state diffs are published and that they are correct. The sequencer cannot mess with them at all; either it publishes the correct state diffs or the L1 contract will reject its batch.

Compression

Because state diffs are compressed to save space on L1, this compression needs to be proven as well. Otherwise, once again, the sequencer could send the wrong (compressed) state diffs. This is easy though, we just make the prover run the compression and we're done.

EIP 4844 (a.k.a. Blobs)

While we could send state diffs through calldata, there is a (hopefully) cheaper way to do it: blobs. The Ethereum Cancun upgrade introduced a new type of transaction where users can submit a list of opaque blobs of data, each one of size at most 128 KB. The main purpose of this new type of transaction is precisely to be used by rollups for data availability; they are priced separately through a blob_gas market instead of the regular gas one and for all intents and purposes should be much cheaper than calldata.

Using EIP 4844, our state diffs would now be sent through blobs. While this is cheaper, there's a new problem to address with it. The whole point of blobs is that they're cheaper because they are only kept around for approximately two weeks and ONLY in the beacon chain, i.e. the consensus side. The execution side (and thus the EVM when running contracts) does not have access to the contents of a blob. Instead, the only thing it has access to is a KZG commitment of it.

This is important. If you recall, the way the L1 ensured that the state diff published by the sequencer was correct was by hashing its contents and ensuring that the hash matched the given state diff hash. With the contents of the state diff now no longer accessible by the contract, we can't do that anymore, so we need another way to ensure the correct contents of the state diff (i.e. the blob).

The solution is through a proof of equivalence between polynomial commitment schemes. The idea is as follows: proofs of equivalence allow you to show that two (polynomial) commitments point to the same underlying data. In our case, we have two commitments:

• The state diff commitment calculated by the sequencer/prover.
• The KZG commitment of the blob sent on the commit transaction (recall that the blob should just be the state diff).

If we turn the first one into a polynomial commitment, we can take a random evaluation point through Fiat Shamir and prove that it evaluates to the same value as the KZG blob commitment at that point. The commit transaction then sends the blob commitment and, through the point evaluation precompile, verifies that the given blob evaluates to that same value. If it does, the underlying blob is indeed the correct state diff.

Our proof of equivalence implementation follows Method 1 here. What we do is the following:

Prover side

• Take the state diff being committed to as 4096 32-byte chunks (these will be interpreted as field elements later on, but for now we don't care). Call these chunks $d_i$, with i ranging from 0 to 4095.

• Build a merkle tree with the $d_i$ as leaves. Note that we can think of the merkle root as a polynomial commitment, where the i-th leaf is the evaluation of the polynomial on the i-th power of $\omega$, the 4096-th root of unity on $F_q$, the field modulus of the BLS12-381 curve. Call this polynomial $f$. This is the same polynomial that the L1 KZG blob commits to (by definition). Call the L1 blob KZG commitment $C_1$ and the merkle root we just computed $C_2$.

• Choose x as keccak($C_1$, $C_2$) and calculate the evaluation $f(x)$; call it y. To do this calculation, because we only have the $d_i$, the easiest way to do it is through the barycentric formula. IMPORTANT: we are taking the $d_i$, x, y, and $\omega$ as elements of $F_q$, NOT the native field used by our prover. The evaluation thus is:

  $$y=f(x)=\frac{x^{4096}-1}{4096}\sum _{i=0}^{4095}{d}_i\frac{{\omega }^i}{x-{\omega }^i}$$

• Set x and y as public inputs. All the above shows the verifier on L1 that we made a polynomial commitment to the state diff, that its evaluation on x is y, and that x was chosen through Fiat-Shamir by hashing the two commitments.

Verifier side

• When committing to the data on L1 send, as part of the calldata, a kzg blob commitment along with an opening proving that it evaluates to y on x. The contract, through the point evaluation precompile, checks that both:
  • The commitment's hash is equal to the versioned hash for that blob.
  • The evaluation is correct.

Transition to RLP-encoded Blocks

The state diff approach has been deprecated. While it provided a more compact representation, it only guaranteed the availability of the modified state, not the original transactions themselves. To ensure that transactions are also publicly available, Ethrex now publishes RLP-encoded blocks, together with their corresponding fee configurations, directly in blobs (see Transaction fees).

This new approach guarantees both transaction and state availability, at the cost of higher data size. According to our internal measurements (block_vs_state_diff_measurements.md), sending block lists in blobs instead of state diffs decreases the number of transactions that can fit in a single blob by approximately 2× for ETH transfers and 3× for ERC20 transfers.

L1<->L2 communication

To communicate between L1 and L2, we use two mechanisms called Privileged transactions, and L1 messages. In this section we talk a bit about them, first going through the more specific use cases for Deposits and Withdrawals.

Deposits

The mechanism for depositing funds to L2 from L1 is explained in detail in "Deposits".

Withdrawals

The mechanism for withdrawing funds from L2 back to L1 is explained in detail in "Withdrawals".

Recap

Batch Commitment

An L2 batch commitment contains:

• The new L2 state root.
• The latest block's hash
• The KZG versioned hash of the blobs published by the L2
• The rolling hash of the processed privileged transactions
• The Merkle root of the withdrawal logs

These are committed as public inputs of the zk proof that validates a new L2 state.

L1 contract checks

Commit transaction

For the commit transaction, the L1 verifier contract receives the batch commitment, as defined previously, for the new batch.

The contract will then:

• Check that the batch number is the immediate successor of the last committed batch.
• Check that the batch has not been committed already.
• Check that the lastBlockHash is not zero.
• If privileged transactions were processed, it checks the submitted hash against the one in the CommonBridge contract.
• If withdrawals were processed, it publishes them to the CommonBridge contract.
• It checks that a blob was published if the L2 is running as a rollup, or that no blob was published if it's running as a validium.
• Calculate the new batch commitment and store it.

Verify transaction

On a verification transaction, the L1 contract receives the following:

• The batch number.
• The RISC-V Zero-Knowledge proof of the batch execution (if enabled).
• The SP1 Zero-Knowledge proof of the batch execution (if enabled).
• The TDX Zero-Knowledge proof of the batch execution (if enabled).

The contract will then:

• Check that the batch number is the immediate successor of the last verified batch.
• Check that the batch has been committed.
• It removes the pending transaction hashes from the CommonBridge contract.
• It verifies the public data of the proof, checking that the data committed in the commitBatch call matches the data in the public inputs of the proof.
• Pass the proof and public inputs to the verifier and assert the proof passes.
• If the proof passes, finalize the L2 state, setting the latest batch as the given one and allowing any withdrawals for that batch to occur.

What the sequencer cannot do

• Forge Transactions: Invalid transactions (e.g. sending money from someone who did not authorize it) are not possible, since part of transaction execution requires signature verification. Every transaction has to come along with a signature from the sender. That signature needs to be verified; the L1 verifier will reject any block containing a transaction whose signature is not valid.
• Withhold State: Every L1 commit transaction needs to send the corresponding state diffs for it and the contract, along with the proof, make sure that they indeed correspond to the given batch. TODO: Expand with docs on how this works.
• Mint money for itself or others: The only valid protocol transaction that can mint money for a user is an L1 deposit. Every one of these mint transactions is linked to exactly one deposit transaction on L1. TODO: Expand with some docs on the exact details of how this works.

What the sequencer can do

The main thing the sequencer can do is CENSOR transactions. Any transaction sent to the sequencer could be arbitrarily dropped and not included in blocks. This is not completely enforceable by the protocol, but there is a big mitigation in the form of an escape hatch.

TODO: Explain this in detail.

Ethrex L2 sequencer

Components

The L2 Proposer is composed of the following components:

Block Producer

Creates Blocks with a connection to the auth.rpc port.

L1 Watcher

This component monitors the L1 for new deposits made by users. For that, it queries the CommonBridge contract on L1 at regular intervals (defined by the config file) for new DepositInitiated() events. Once a new deposit event is detected, it creates the corresponding deposit transaction on the L2. It also periodically fetches the BlobBaseFee from L1 (at a configured interval), which is used to compute the L1 fees.

L1 Transaction Sender (a.k.a. L1 Committer)

As the name suggests, this component sends transactions to the L1. But not any transaction, only commit and verify transactions.

Commit transactions are sent when the Proposer wants to commit to a new batch of blocks. These transactions contain the batch data to be committed in the L1.

Verify transactions are sent by the Proposer after the prover has successfully generated a proof of block execution to verify it. These transactions contain the new state root of the L2, the hash of the state diffs produced in the block, the root of the withdrawals logs merkle tree and the hash of the processed deposits.

Proof Coordinator

The Proof Coordinator is a simple TCP server that manages communication with a component called the Prover. The Prover acts as a simple TCP client that makes requests to prove a block to the Coordinator. It responds with the proof input data required to generate the proof. Then, the Prover executes a zkVM, generates the Groth16 proof, and sends it back to the Coordinator.

The Proof Coordinator centralizes the responsibility of determining which block needs to be proven next and how to retrieve the necessary data for proving. This design simplifies the system by reducing the complexity of the Prover, it only makes requests and proves blocks.

For more information about the Proof Coordinator, the Prover, and the proving process itself, see the Prover Docs.

L1 Proof Sender

The L1 Proof Sender is responsible for interacting with Ethereum L1 to manage proof verification. Its key functionalities include:

• Connecting to Ethereum L1 to send proofs for verification.
• Dynamically determine required proof types based on active verifier contracts (REQUIRE_<prover>_PROOF).
• Ensure blocks are verified in the correct order by invoking the verify(..) function in the OnChainProposer contract. Upon successful verification, an event is emitted to confirm the block's verification status.
• Operating on a configured interval defined by proof_send_interval_ms.

Configuration

Configuration is done either by CLI flags or through environment variables. Run cargo run --release --bin ethrex -- l2 --help in the repository's root directory to see the available CLI flags and envs.

ethrex-prover for L2

Intro

The prover consists of two main components: handling incoming proving data from the L2 sequencer, specifically from the ProofCoordinator component, and the actual zkVM running and generating proofs of execution.

In summary, the prover manages the inputs from the ProofCoordinator and then calls the zkVM to perform the proving process and generate the zero-knowledge proof (groth16 for on-chain verification, or a compressed STARK for verification via Aligned Layer).

Workflow

The ProofCoordinator monitors requests for new jobs from the Prover, which are sent when the prover is available. Upon receiving a new job, the Prover generates the proof, after which the Prover sends the proof back to the ProofCoordinator.

sequenceDiagram
    participant zkVM
    participant Prover
    participant ProofCoordinator
    Prover->>+ProofCoordinator: ProofData::Request
    ProofCoordinator-->>-Prover: ProofData::Response(inputs)
    Prover->>+zkVM: Prove(inputs)
    zkVM-->>-Prover: generates zk proof
    Prover->>+ProofCoordinator: ProofData::Submit(batch number, proof)
    ProofCoordinator-->>-Prover: ProofData::SubmitAck(batch number)

For running the prover, see Deploy an L2. For developer-focused setup and run instructions, see Running the Prover. For comprehensive details on the internals of the prover, see ethrex-prover.

TDX execution module

This document has documentation related to proving ethrex blocks using TDX.

Usage

On a machine with TDX support with the required setup go to quote-gen and run

make run

What is TDX?

TDX is an Intel technology implementing a Trusted Execution Environment. Such an environment allows verifying certain code was executed without being tampered with or observed.

These verifications (attestations) are known as "quotes" and contain signatures verifying the attestation was generated by a genuine processor, the measurements at the time, and a user-provided piece of data binding the proof.

The measurements are saved to four Run Time Measurement Registers (RTMR), with each RTMR representing a boot stage. This is analogous to how PCRs work.

Usage considerations

Do not hardcode quote verification parameters as they might change.

It's easy to silently overlook non-verified areas such as accidentally leaving login enabled, or not verifying the integrity of the state.

Boot sequence

• Firmware (OVMF here) is loaded (and hashed into RTMR[0])
• UKI is loaded (and hashed into a RTMR)
• kernel and initrd are extracted from the UKI and executed
• root partition is verified using the roothash= value provided on the kernel cmdline and the hash partition with the dm-verity merkle tree
• root partition is mounted read-only
• (WIP) systemd executes the payload

Image build components

For reproducibility of images and hypervisor runtime we use Nix.

hypervisor.nix

This builds the modified (with patches for TDX support) qemu, and TDX-specific VBIOS (OVMF) and exports a script to run a given image (the parameters, specifically added devices, affect the measurements).

service.nix

This contains the quote-gen service. Its hash changes every time a non-gitignored file changes.

image.nix

Exports an image that uses UKI and dm-verity to generate an image where changing any component changes the hash of the bootloader (the UKI image), which is measured by the BIOS.

Running

You can enable the prover by setting ETHREX_L2_TDX=true.

For development purposes, you can use the flag ETHREX_TDX_DEV_MODE=true to disable quote verification. This allows you to run the quote generator even without having TDX-capable hardware.

Ensure the proof coordinator is reachable at 172.17.0.1. You can bring up the network by first starting the L2 components:

// cd crates/l2
make init ETHREX_L2_TDX=true PROOF_COORDINATOR_ADDRESS=0.0.0.0

And in another terminal, running the VM:

// cd crates/l2
make -C tee/quote-gen run

Troubleshooting

unshare: write failed /proc/self/uid_map: Operation not permitted

If you get this error when building the image, it's probably because your OS has unprivileged userns restricted by default. You can undo this by running the following commands as root, or running the build as root while disabling sandboxing.

sysctl kernel.unprivileged_userns_apparmor_policy=0
sysctl kernel.apparmor_restrict_unprivileged_userns=0

RTMR/MRTD mismatch

If any code or dependencies changed, the measurements will change.

To obtain the new measurements, first you obtain the quote by running the prover (you don't need to have the l2 running). Its output will contain Sending quote <very long hex string>.

This usually causes a RTMR1 mismatch. The easiest way to obtain the new RTMR values is by looking at the printed quote for the next 96 bytes after the RTMR0, corresponding to RTMR1||RTMR2 (48 bytes each).

More generally, you can generate a report with DCAP.verifyAndAttestOnChain(quote) which validates and extracts the report.

Look at bytes 341..485 of the output for RTMRs and bytes 149..197 for the MRTD.

For example, the file quote.example contains a quote, which can be turned into the following report:

00048100000000b0c06f000000060103000000000000000000000000005b38e33a6487958b72c3c12a938eaa5e3fd4510c51aeeab58c7d5ecee41d7c436489d6c8e4f92f160b7cad34207b00c100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000e702060000000000

91eb2b44d141d4ece09f0c75c2c53d247a3c68edd7fafe8a3520c942a604a407de03ae6dc5f87f27428b2538873118b7 # MRTD

000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

4f3d617a1c89bd9a89ea146c15b04383b7db7318f41a851802bba8eace5a6cf71050e65f65fd50176e4f006764a42643 # RTMR0
53827a034d1e4c7f13fd2a12aee4497e7097f15a04794553e12fe73e2ffb8bd57585e771951115a13ec4d7e6bc193038 # RTMR1
2ca1a728ff13c36195ad95e8f725bf00d7f9c5d6ed730fb8f50cccad692ab81aefc83d594819375649be934022573528 # RTMR2
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 # RTMR3

39618efd10b14136ab416d6acfff8e36b23533a90000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Interacting with the L2

This section explains how to interact with your L2 rollup built with ethrex. Here you'll find guides for:

• Depositing and withdrawing assets
• Connecting wallets like MetaMask
• Deploying smart contracts
• Maintaining and operating the sequencer
• Interacting with shared bridge

Use these guides to perform common actions and manage your L2 network.

Deposit assets into the L2

To transfer ETH from Ethereum L1 to your L2 account, you need to use the CommonBridge as explained in this section.

Prerequisites for L1 deposit

• An L1 account with sufficient ETH balance, for developing purposes you can use:
  • Address: 0x8943545177806ed17b9f23f0a21ee5948ecaa776
  • Private Key: 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31
• The address of the deployed CommonBridge contract.
• An Ethereum utility tool like Rex

Making a deposit

Making a deposit in the Bridge, using Rex, is as simple as:

# Format: rex l2 deposit <AMOUNT> <PRIVATE_KEY> <BRIDGE_ADDRESS> [L1_RPC_URL]
rex l2 deposit 50000000 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31 0x65dd6dc5df74b7e08e92c910122f91d7b2d5184f

Verifying the updated L2 balance

Once the deposit is made you can verify the balance has increased with:

# Format: rex l2 balance <ADDRESS> [RPC_URL]
rex l2 balance 0x8943545177806ed17b9f23f0a21ee5948ecaa776

For more information on what you can do with the CommonBridge see Ethrex L2 contracts.

Withdraw assets from the L2

This section explains how to withdraw funds from the L2 through the native bridge.

Prerequisites for L2 withdrawal

• An L2 account with sufficient ETH balance, for developing purposes you can use:
  • Address: 0x8943545177806ed17b9f23f0a21ee5948ecaa776
  • Private Key: 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31
• The address of the deployed CommonBridge L2 contract (note here that we are calling the L2 contract instead of the L1 as in the deposit case). If not specified, You can use:
  • CommonBridge L2: 0x000000000000000000000000000000000000ffff
• An Ethereum utility tool like Rex.

Making a withdrawal

Using Rex, we simply run the rex l2 withdraw command, which uses the default CommonBridge address.

# Format: rex l2 withdraw <AMOUNT> <PRIVATE_KEY> [RPC_URL]
rex l2 withdraw 5000 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31

If the withdrawal is successful, the hash will be printed like this:

Withdrawal sent: <L2_WITHDRAWAL_TX_HASH>
...

Claiming the withdrawal

After making a withdrawal, it has to be claimed in the L1, through the L1 CommonBridge contract. For that, we can use the Rex command rex l2 claim-withdraw, with the tx hash obtained in the previous step. But first, it is necessary to wait for the block that includes the withdrawal to be verified.

# Format: rex l2 claim-withdraw <L2_WITHDRAWAL_TX_HASH> <PRIVATE_KEY> <BRIDGE_ADDRESS> [L1_RPC_URL] [RPC_URL]
rex l2 claim-withdraw <L2_WITHDRAWAL_TX_HASH> 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31 0x65dd6dc5df74b7e08e92c910122f91d7b2d5184f

Verifying the withdrawal

Once the withdrawal is made you can verify the balance has decreased in the L2 with:

rex l2 balance 0x8943545177806ed17b9f23f0a21ee5948ecaa776

And also increased in the L1:

rex balance 0x8943545177806ed17b9f23f0a21ee5948ecaa776

Connect a Wallet

You can connect your L2 network to MetaMask to interact with your rollup using a familiar wallet interface.

Add Your L2 Network to MetaMask

1. Open MetaMask and click the network dropdown.
2. Select "Add custom network".
3. Enter your L2 network details:
   • Network Name: (choose any name, e.g. "My L2 Rollup")
   • RPC URL: http://localhost:1729 (or your L2 node's RPC endpoint)
   • Chain ID: (use the chain ID from your L2 genesis config)
   • Currency Symbol: (e.g. ETH)
   • Block Explorer URL: (optional, can be left blank)
4. Save the network.

You can now use MetaMask to send transactions and interact with contracts on your L2.

Tip: If you are running the L2 node on a remote server, replace localhost with the server's IP or domain.

Deploy a Contract to L2

You can deploy smart contracts to your L2 using rex, a simple CLI tool for interacting with Ethereum-compatible networks.

1. Generate the Contract Bytecode

First, compile your Solidity contract to get the deployment bytecode. You can use solc (v0.8.31) for this:

solc --bin MyContract.sol -o out/

The bytecode will be in out/MyContract.bin

2. Deploy with rex

Use the following command to deploy your contract:

rex deploy --rpc-url http://localhost:1729 <BYTECODE> 0 <PRIVATE_KEY>

• Replace <BYTECODE> with the hex string from your compiled contract (e.g., contents of MyContract.bin)
• Replace <PRIVATE_KEY> with your wallet's private key. It must have funds in L2
• Adjust the --rpc-url if your L2 node is running elsewhere

For more details and advanced usage, see the rex repository.

Blockscout for ethrex L2

TBD

L2 Hub

TBD

Interacting with the shared bridge

This document details different scenarios for interacting with shared bridge enabled L2s.

Prerequisites

This guide assumes you already have two L2s running with the shared bridge enabled. Refer to Deploy a shared bridge enabled L2

ETH Transfer

Check balances

Check the balances before sending the transfer

rex balance 0xe25583099ba105d9ec0a67f5ae86d90e50036425 http://localhost:1729 # Receiver balance on first L2
rex balance 0x8943545177806ed17b9f23f0a21ee5948ecaa776 http://localhost:1730 # Sender balance on second L2

Send the transfer

rex send --rpc-url http://localhost:1730 --private-key 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31 --value 10000000000000001 0x000000000000000000000000000000000000FFFF 'sendToL2(uint256,address,uint256,bytes)' 65536999 0xe25583099ba105d9ec0a67f5ae86d90e50036425 100000 "" --gas-price 3946771033

Check balances

After some time the balances should change (about 1-2 minutes)

rex balance 0xe25583099ba105d9ec0a67f5ae86d90e50036425 http://localhost:1729 # Receiver balance on first L2
rex balance 0x8943545177806ed17b9f23f0a21ee5948ecaa776 http://localhost:1730 # Sender balance on second L2

Contract Call

Add the contract

Create a Counter.sol file with the following content

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.20;

contract Counter {
    uint256 public count;

    function increment() external {
        count += 1;
    }

    function get() external view returns (uint256) {
        return count;
    }
}

Deploy the contract

rex deploy --rpc-url http://localhost:1729 --remappings 0 --contract-path ./Counter.sol 0 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31

Save the contract address for the next steps:

export COUNTER_ADDRESS=<COUNTER_ADDRESS> 

Check counter value

rex call $COUNTER_ADDRESS "get()" --rpc-url http://localhost:1729

Increase the counter from the other L2

rex send --rpc-url http://localhost:1730 --private-key 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31 0x000000000000000000000000000000000000FFFF 'sendToL2(uint256,address,uint256,bytes)' 65536999 $COUNTER_ADDRESS 100000 d09de08a --gas-price 3946771033

Check counter value

rex call $COUNTER_ADDRESS "get()" --rpc-url http://localhost:1729

Contract Call and ETH Transfer

Add the contract

Create a Counter.sol file with the following content (The increment function is now payable)

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.20;

contract Counter {
    uint256 public count;

    function increment() external payable {
        count += 1;
    }

    function get() external view returns (uint256) {
        return count;
    }
}

Deploy the contract

rex deploy --rpc-url http://localhost:1729 --remappings 0 --contract-path ./Counter.sol 0 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31

Save the contract address for the next steps:

export COUNTER_ADDRESS=<COUNTER_ADDRESS> 

Check counter value

rex call $COUNTER_ADDRESS "get()" --rpc-url http://localhost:1729

Check counter balance

rex balance $COUNTER_ADDRESS http://localhost:1729

Increase the counter from the other L2

rex send --rpc-url http://localhost:1730 --private-key 0xbcdf20249abf0ed6d944c0288fad489e33f66b3960d9e6229c1cd214ed3bbe31 --value 1000 0x000000000000000000000000000000000000FFFF 'sendToL2(uint256,address,uint256,bytes)' 65536999 $COUNTER_ADDRESS 100000 d09de08a --gas-price 3946771033

Check counter value

rex call $COUNTER_ADDRESS "get()" --rpc-url http://localhost:1729

Check counter balance

rex balance $COUNTER_ADDRESS http://localhost:1729

Troubleshooting

If you can't deploy the counter contract, either because of Transaction intrinsic gas overflow or because the transaction is never included in a block. Retry the deploy command adding --priority-gas-price and --gas-price with the same value, increment it by 10 until it deploys correctly.

Fundamentals

In L2 mode, the ethrex code is repurposed to run a rollup that settles on Ethereum as the L1.

The main differences between this mode and regular ethrex are:

• In regular rollup mode, there is no consensus; the node is turned into a sequencer that proposes blocks for the chain. In based rollup mode, consensus is achieved by a mechanism that rotates sequencers, enforced by the L1.
• Block execution is proven using a RISC-V zkVM (or attested to using TDX, a Trusted Execution Environment) and its proofs (or signatures/attestations) are sent to L1 for verification.
• A set of Solidity contracts to be deployed to the L1 are included as part of chain initialization.
• Two new types of transactions are included: deposits (native token mints) and withdrawals.

At a high level, the following new parts are added to the node:

• A proposer component, in charge of continually creating new blocks from the mempool transactions. This replaces the regular flow that an Ethereum L1 node has, where new blocks come from the consensus layer through the forkChoiceUpdate -> getPayload -> NewPayload Engine API flow in communication with the consensus layer.
• A prover subsystem, which itself consists of two parts:
  • A proverClient that takes new blocks from the node, proves them, then sends the proof back to the node to send to the L1. This is a separate binary running outside the node, as proving has very different (and higher) hardware requirements than the sequencer.
  • A proverServer component inside the node that communicates with the prover, sending witness data for proving and receiving proofs for settlement on L1.
• L1 contracts with functions to commit to new state and then verify the state transition function, only advancing the state of the L2 if the proof verifies. It also has functionality to process deposits and withdrawals to/from the L2.
• The EVM is lightly modified with new features to process deposits and withdrawals accordingly.

Ethrex L2 documentation

For general documentation, see:

• Architecture overview for a high-level view of the ethrex L2 stack.
• Smart contracts has information on L1 and L2 smart contracts including steps to upgrade and transfer ownership.
• Based sequencing contains ethrex's roadmap for becoming based.
• State diffs explains the mechanism needed to provide data availability.
• How asset deposits and withdrawals work.
• Fee token
• Exit window and Timelock for upgrade safety mechanisms.
• Aligned Layer Integration details how ethrex L2 integrates with Aligned Layer for proof aggregation and verification.

State diffs

This architecture was inspired by MatterLabs' ZKsync pubdata architecture.

To provide data availability for our blockchain, we need to publish enough information on every commit transaction to be able to reconstruct the entire state of the L2 from the beginning by querying the L1.

The data needed is:

• The nonce and balance of every EOA.
• The nonce, balance, and storage of every contract account. Note that storage here is a mapping (U256 → U256), so there are a lot of values inside it.
• The bytecode of every contract deployed on the chain.
• All withdrawal Logs.

After executing a batch of L2 blocks, the EVM will return the following data:

• A list of every storage slot modified in the batch, with their previous and next values. A storage slot is a mapping (address, slot) -> value. Note that, in a batch, there could be repeated writes to the same slot. In that case, we keep only the latest write; all the others are discarded since they are not needed for state reconstruction.
• The bytecode of every newly deployed contract. Every contract deployed is then a pair (address, bytecode).
• A list of withdrawal logs (as explained in milestone 1 we already collect these and publish a merkle root of their values as calldata, but we still need to send them as the state diff).
• A list of triples (address, nonce_increase, balance) for every modified account. The nonce_increase is a value that says by how much the nonce of the account was increased in the batch (this could be more than one as there can be multiple transactions for the account in the batch). The balance is just the new balance value for the account.

The full state diff sent for each batch will then be a sequence of bytes encoded as follows. We use the notation un for a sequence of n bits, so u16 is a 16-bit sequence and u96 a 96-bit one, we don't really care about signedness here; if we don't specify it, the value is of variable length and a field before it specifies it.

• The first byte is a u8: the version header. For now it should always be one, but we reserve it for future changes to the encoding/compression format.
• Next come the block header info of the last block in the batch:
  • The tx_root, receipts_root and parent_hash are u256 values.
  • The gas_limit, gas_used, timestamp, block_number and base_fee_per_gas are u64 values.
• Next the ModifiedAccounts list. The first two bytes (u16) are the amount of element it has, followed by its entries. Each entry correspond to an altered address and has the form:
  • The first byte is the type of the modification. The value is a u8, constrained to the range [1; 23], computed by adding the following values:
    • 1 if the balance of the EOA/contract was modified.
    • 2 if the nonce of the EOA/contract was modified.
    • 4 if the storage of the contract was modified.
    • 8 if the contract was created and the bytecode is previously unknown.
    • 16 if the contract was created and the bytecode is previously known.
  • The next 20 bytes, a u160, is the address of the modified account.
  • If the balance was modified (i.e. type & 0x01 == 1), the next 32 bytes, a u256, is the new balance of the account.
  • If the nonce was modified (i.e. type & 0x02 == 2), the next 2 bytes, a u16, is the increase in the nonce.
  • If the storage was modified (i.e. type & 0x04 == 4), the next 2 bytes, a u16, is the number of storage slots modified. Then come the sequence of (key_u256, new_value_u256) key value pairs with the modified slots.
  • If the contract was created and the bytecode is previously unknown (i.e. type & 0x08 == 8), the next 2 bytes, a u16, is the length of the bytecode in bytes. Then come the bytecode itself.
  • If the contract was created and the bytecode is previously known (i.e. type & 0x10 == 16), the next 32 bytes, a u256, is the hash of the bytecode of the contract.
  • Note that values 8 and 16 are mutually exclusive, and if type is greater or equal to 4, then the address is a contract. Each address can only appear once in the list.
• Next the WithdrawalLogs field:
  • First two bytes are the number of entries, then come the tuples (to_u160, amount_u256, tx_hash_u256).
• Next the PrivilegedTransactionLogs field:
  • First two bytes are the number of entries, then come the tuples (to_u160, value_u256).
• In case of the only changes on an account are produced by withdrawals, the ModifiedAccounts for that address field must be omitted. In this case, the state diff can be computed by incrementing the nonce in one unit and subtracting the amount from the balance.

To recap, using || for byte concatenation and [] for optional parameters, the full encoding for state diffs is:

version_header_u8 ||
// Last Block Header info
tx_root_u256 || receipts_root_u256 || parent_hash_u256 ||
gas_limit_u64 || gas_used_u64 || timestamp_u64 ||
block_number_u64 || base_fee_per_gas_u64
// Modified Accounts
number_of_modified_accounts_u16 ||
(
  type_u8 || address_u160 || [balance_u256] || [nonce_increase_u16] ||
  [number_of_modified_storage_slots_u16 || (key_u256 || value_u256)... ] ||
  [bytecode_len_u16 || bytecode ...] ||
  [code_hash_u256]
)...
// Withdraw Logs
number_of_withdraw_logs_u16 ||
(to_u160 || amount_u256 || tx_hash_u256) ...
// Privileged Transactions Logs
number_of_privileged_transaction_logs_u16 ||
(to_u160 || value_u256) ...

The sequencer will then make a commitment to this encoded state diff (explained in the EIP 4844 section how this is done) and send on the commit transaction:

• Through calldata, the state diff commitment (which is part of the public input to the proof).
• Through the blob, the encoded state diff.

Comparative Analysis: Transaction Volume in Blobs Using State Diffs and Transaction Lists

The following are results from measurements conducted to understand the efficiency of blob utilization in an ethrex L2 network through the simulation of different scenarios with varying transaction complexities (e.g., ETH transfers, ERC20 transfers, and other complex smart contract interactions) and data encoding strategies, with the final goal of estimating the approximate number of transactions that can be packed into a single blob using state diffs versus full transaction lists, thereby optimizing calldata costs and achieving greater scalability.

Measurements (Amount of transactions per batch)

ETH Transfers

ERC20 Transfers

Summary

Conclusion

Sending block lists in blobs instead of state diffs decreases the number of transactions that can fit in a single blob by approximately 2x for ETH transfers and 3x for ERC20 transfers.

How these measurements were done

Prerequisites

• Fresh cloned ethrex repository
• The spammer and measurer code provided in the appendix set up for running (you can create a new cargo project and copy the code there)

Steps

1. Run an L2 ethrex:

For running the measurements, we need to run an ethrex L2 node. For doing that, change your current directory to ethrex/crates/l2 in your fresh-cloned ethrex and run the following in a terminal:

ETHREX_COMMITTER_COMMIT_TIME=120000 MEMPOOL_MAX_SIZE=1000000 make init-l2-dev

This will set up and run an ethrex L2 node in dev mode with a mempool size big-enough to be able to handle the spammer transactions. And after this you should see the ethrex L2 monitor running.

2. Run the desired transactions spammer

In another terminal, change your current directory to the spammer code you want to run (either ETH or ERC20) and run:

cargo run

It's ok not to see any logs or prints as output, since the spammer code doesn't print anything.

If you go back to the terminal where the L2 node is running, you should start seeing the following:

1. The mempool table growing in size as transactions are being sent to the L2 node.
2. In the L2 Blocks table, new blocks with #Txs greater than 0 being created as the spammer transactions are included in blocks.
3. Every 2 minutes (or the time you set in ETHREX_COMMITTER_COMMIT_TIME), new batches being created in the L2 Batches table.

3. Run the measurer

In another terminal, change your current directory to the measurer code and run:

cargo run

This will start printing the total number of transactions included in each batch until the last committed one.

Appendix

• ETH Transactions Spammer
  • main.rs
  • Cargo.toml
• Measurer
  • main.rs
  • Cargo.toml
• ERC20 Transactions Spammer
  • main.rs
  • Cargo.toml

ETH Transactions Spammer

main.rs

use ethrex_common::{
    Address, U256,
    types::{EIP1559Transaction, Transaction, TxKind},
};
use ethrex_l2_rpc::signer::{LocalSigner, Signable, Signer};
use ethrex_l2_sdk::send_generic_transaction;
use ethrex_rpc::EthClient;
use tokio::time::sleep;
use url::Url;

#[tokio::main]
async fn main() {
    let chain_id = 65536999;
    let senders = vec![
        "7a738a3a8ee9cdbb5ee8dfc1fc5d97847eaba4d31fd94f89e57880f8901fa029",
        "8cfe380955165dd01f4e33a3c68f4e08881f238fbbea71a2ab407f4a3759705b",
        "5bb463c0e64039550de4f95b873397b36d76b2f1af62454bb02cf6024d1ea703",
        "3c0924743b33b5f06b056bed8170924ca12b0d52671fb85de1bb391201709aaf",
        "6aeeda1e7eda6d618de89496fce01fb6ec685c38f1c5fccaa129ec339d33ff87",
    ]
    .iter()
    .map(|s| Signer::Local(LocalSigner::new(s.parse().expect("invalid private key"))))
    .collect::<Vec<Signer>>();
    let eth_client: EthClient =
        EthClient::new(Url::parse("http://localhost:1729").expect("Invalid URL"))
            .expect("Failed to create EthClient");
    let mut nonce = 0;
    loop {
        for sender in senders.clone() {
            let signed_tx = generate_signed_transaction(nonce, chain_id, &sender).await;
            send_generic_transaction(&eth_client, signed_tx.into(), &sender)
                .await
                .expect("Failed to send transaction");
            sleep(std::time::Duration::from_millis(10)).await;
        }
        nonce += 1;
    }
}

async fn generate_signed_transaction(nonce: u64, chain_id: u64, signer: &Signer) -> Transaction {
    Transaction::EIP1559Transaction(EIP1559Transaction {
        nonce,
        value: U256::one(),
        gas_limit: 250000,
        max_fee_per_gas: u64::MAX,
        max_priority_fee_per_gas: 10,
        chain_id,
        to: TxKind::Call(Address::random()),
        ..Default::default()
    })
    .sign(&signer)
    .await
    .expect("failed to sign transaction")
}

Cargo.toml

[package]
name = "tx_spammer"
version = "0.1.0"
edition = "2024"

[dependencies]
ethrex-sdk = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-common = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-l2-rpc = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-rpc = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }

tokio = { version = "1", features = ["full"] }
url = "2"
hex = "0.4"

Measurer

A simple program that queries the L2 node for batches and blocks, counting the number of transactions in each block, and summing them up per batch.

main.rs

use reqwest::Client;
use serde_json::{Value, json};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut batch = 1;

    loop {
        let (first, last) = fetch_batch(batch).await;
        let mut txs = 0u64;
        for number in first as u64..=last as u64 {
            txs += fetch_block(number).await;
        }
        println!("Total transactions in batch {}: {}", batch, txs);

        batch += 1;
    }
}

async fn fetch_batch(number: u64) -> (i64, i64) {
    // Create the JSON body equivalent to the --data in curl
    let body = json!({
        "method": "ethrex_getBatchByNumber",
        "params": [format!("0x{:x}", number), false],
        "id": 1,
        "jsonrpc": "2.0"
    });

    // Create a blocking HTTP client
    let client = Client::new();

    // Send the POST request
    let response = client
        .post("http://localhost:1729")
        .header("Content-Type", "application/json")
        .json(&body)
        .send()
        .await
        .expect("Failed to send request")
        .json::<Value>()
        .await
        .unwrap();

    let result = &response["result"];
    let first_block = &result["first_block"].as_i64().unwrap();
    let last_block = &result["last_block"].as_i64().unwrap();
    (*first_block, *last_block)
}

async fn fetch_block(number: u64) -> u64 {
    // Create the JSON body equivalent to the --data in curl
    let body = json!({
        "method": "eth_getBlockByNumber",
        "params": [format!("0x{:x}", number), false],
        "id": 1,
        "jsonrpc": "2.0"
    });

    // Create a blocking HTTP client
    let client = Client::new();

    // Send the POST request
    let response = client
        .post("http://localhost:1729")
        .header("Content-Type", "application/json")
        .json(&body)
        .send()
        .await
        .expect("Failed to send request")
        .json::<Value>()
        .await
        .unwrap();

    let result = &response["result"];
    let transactions = &result["transactions"];
    transactions.as_array().unwrap().len() as u64
}

Cargo.toml

[package]
name = "measurer"
version = "0.1.0"
edition = "2024"

[dependencies]
reqwest = { version = "0.11", features = ["json"] }
serde_json = "1.0"
tokio = { version = "1", features = ["full"] }

ERC20 Transactions Spammer

main.rs

use ethrex_blockchain::constants::TX_GAS_COST;
use ethrex_common::{
    Address, U256,
    types::{EIP1559Transaction, GenericTransaction, Transaction, TxKind, TxType},
};
use ethrex_l2_rpc::signer::{LocalSigner, Signable, Signer};
use ethrex_l2_sdk::{
    build_generic_tx, calldata::encode_calldata, create_deploy, send_generic_transaction,
    wait_for_transaction_receipt,
};
use ethrex_rpc::{EthClient, clients::Overrides};
use tokio::time::sleep;
use url::Url;

// ERC20 compiled artifact generated from this tutorial:
// https://medium.com/@kaishinaw/erc20-using-hardhat-a-comprehensive-guide-3211efba98d4
// If you want to modify the behaviour of the contract, edit the ERC20.sol file,
// and compile it with solc.
const ERC20: &str = include_str!("./TestToken.bin").trim_ascii();

#[tokio::main]
async fn main() {
    let chain_id = 65536999;
    let signer = Signer::Local(LocalSigner::new(
        "39725efee3fb28614de3bacaffe4cc4bd8c436257e2c8bb887c4b5c4be45e76d"
            .parse()
            .expect("invalid private key"),
    ));
    let eth_client: EthClient =
        EthClient::new(Url::parse("http://localhost:1729").expect("Invalid URL"))
            .expect("Failed to create EthClient");
    let contract_address = erc20_deploy(eth_client.clone(), &signer)
        .await
        .expect("Failed to deploy ERC20 contract");

    let senders = vec![
        "7a738a3a8ee9cdbb5ee8dfc1fc5d97847eaba4d31fd94f89e57880f8901fa029",
        "8cfe380955165dd01f4e33a3c68f4e08881f238fbbea71a2ab407f4a3759705b",
        "5bb463c0e64039550de4f95b873397b36d76b2f1af62454bb02cf6024d1ea703",
        "3c0924743b33b5f06b056bed8170924ca12b0d52671fb85de1bb391201709aaf",
        "6aeeda1e7eda6d618de89496fce01fb6ec685c38f1c5fccaa129ec339d33ff87",
    ]
    .iter()
    .map(|s| Signer::Local(LocalSigner::new(s.parse().expect("invalid private key"))))
    .collect::<Vec<Signer>>();
    claim_erc20_balances(contract_address, eth_client.clone(), senders.clone())
        .await
        .expect("Failed to claim ERC20 balances");
    let mut nonce = 1;
    loop {
        for sender in senders.clone() {
            let signed_tx =
                generate_erc20_transaction(nonce, chain_id, &sender, &eth_client, contract_address)
                    .await;
            send_generic_transaction(&eth_client, signed_tx.into(), &sender)
                .await
                .expect("Failed to send transaction");
            println!(
                "Sent transaction with nonce {} for address {}",
                nonce,
                sender.address()
            );
            sleep(std::time::Duration::from_millis(10)).await;
        }
        nonce += 1;
    }
}

// Given an account vector and the erc20 contract address, claim balance for all accounts.
async fn claim_erc20_balances(
    contract_address: Address,
    client: EthClient,
    accounts: Vec<Signer>,
) -> eyre::Result<()> {
    for account in accounts {
        let claim_balance_calldata = encode_calldata("freeMint()", &[]).unwrap();

        let claim_tx = build_generic_tx(
            &client,
            TxType::EIP1559,
            contract_address,
            account.address(),
            claim_balance_calldata.into(),
            Default::default(),
        )
        .await
        .unwrap();
        let tx_hash = send_generic_transaction(&client, claim_tx, &account)
            .await
            .unwrap();
        wait_for_transaction_receipt(tx_hash, &client, 1000)
            .await
            .unwrap();
    }

    Ok(())
}

async fn deploy_contract(
    client: EthClient,
    deployer: &Signer,
    contract: Vec<u8>,
) -> eyre::Result<Address> {
    let (_, contract_address) =
        create_deploy(&client, deployer, contract.into(), Overrides::default()).await?;

    eyre::Ok(contract_address)
}

async fn erc20_deploy(client: EthClient, deployer: &Signer) -> eyre::Result<Address> {
    let erc20_bytecode = hex::decode(ERC20).expect("Failed to decode ERC20 bytecode");
    deploy_contract(client, deployer, erc20_bytecode).await
}

async fn generate_erc20_transaction(
    nonce: u64,
    chain_id: u64,
    signer: &Signer,
    client: &EthClient,
    contract_address: Address,
) -> GenericTransaction {
    let send_calldata = encode_calldata(
        "transfer(address,uint256)",
        &[
            ethrex_l2_common::calldata::Value::Address(Address::random()),
            ethrex_l2_common::calldata::Value::Uint(U256::one()),
        ],
    )
    .unwrap();

    let tx = build_generic_tx(
        client,
        TxType::EIP1559,
        contract_address,
        signer.address(),
        send_calldata.into(),
        Overrides {
            chain_id: Some(chain_id),
            value: None,
            nonce: Some(nonce),
            max_fee_per_gas: Some(i64::MAX as u64),
            max_priority_fee_per_gas: Some(10_u64),
            gas_limit: Some(TX_GAS_COST * 100),
            ..Default::default()
        },
    )
    .await
    .unwrap();

    tx
}

Cargo.toml

[package]
name = "tx_spammer"
version = "0.1.0"
edition = "2024"

[dependencies]
ethrex-sdk = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-common = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-l2-rpc = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-rpc = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
tokio = { version = "1", features = ["full"] }
ethrex-l2-common = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
ethrex-blockchain = { git = "https://github.com/lambdaclass/ethrex.git", tag = "v6.0.0" }
url = "2"
hex = "0.4"
eyre = "0.6"

TestToken.bin

608060405234801561000f575f5ffd5b506040518060400160405280600881526020017f46756e546f6b656e0000000000000000000000000000000000000000000000008152506040518060400160405280600381526020017f46554e0000000000000000000000000000000000000000000000000000000000815250816003908161008b9190610598565b50806004908161009b9190610598565b5050506100b83369d3c21bcecceda10000006100bd60201b60201c565b61077c565b5f73ffffffffffffffffffffffffffffffffffffffff168273ffffffffffffffffffffffffffffffffffffffff160361012d575f6040517fec442f0500000000000000000000000000000000000000000000000000000000815260040161012491906106a6565b60405180910390fd5b61013e5f838361014260201b60201c565b5050565b5f73ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff1603610192578060025f82825461018691906106ec565b92505081905550610260565b5f5f5f8573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f205490508181101561021b578381836040517fe450d38c0000000000000000000000000000000000000000000000000000000081526004016102129392919061072e565b60405180910390fd5b8181035f5f8673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f2081905550505b5f73ffffffffffffffffffffffffffffffffffffffff168273ffffffffffffffffffffffffffffffffffffffff16036102a7578060025f82825403925050819055506102f1565b805f5f8473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f205f82825401925050819055505b8173ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff167fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef8360405161034e9190610763565b60405180910390a3505050565b5f81519050919050565b7f4e487b71000000000000000000000000000000000000000000000000000000005f52604160045260245ffd5b7f4e487b71000000000000000000000000000000000000000000000000000000005f52602260045260245ffd5b5f60028204905060018216806103d657607f821691505b6020821081036103e9576103e8610392565b5b50919050565b5f819050815f5260205f209050919050565b5f6020601f8301049050919050565b5f82821b905092915050565b5f6008830261044b7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff82610410565b6104558683610410565b95508019841693508086168417925050509392505050565b5f819050919050565b5f819050919050565b5f61049961049461048f8461046d565b610476565b61046d565b9050919050565b5f819050919050565b6104b28361047f565b6104c66104be826104a0565b84845461041c565b825550505050565b5f5f905090565b6104dd6104ce565b6104e88184846104a9565b505050565b5b8181101561050b576105005f826104d5565b6001810190506104ee565b5050565b601f82111561055057610521816103ef565b61052a84610401565b81016020851015610539578190505b61054d61054585610401565b8301826104ed565b50505b505050565b5f82821c905092915050565b5f6105705f1984600802610555565b1980831691505092915050565b5f6105888383610561565b9150826002028217905092915050565b6105a18261035b565b67ffffffffffffffff8111156105ba576105b9610365565b5b6105c482546103bf565b6105cf82828561050f565b5f60209050601f831160018114610600575f84156105ee578287015190505b6105f8858261057d565b86555061065f565b601f19841661060e866103ef565b5f5b8281101561063557848901518255600182019150602085019450602081019050610610565b86831015610652578489015161064e601f891682610561565b8355505b6001600288020188555050505b505050505050565b5f73ffffffffffffffffffffffffffffffffffffffff82169050919050565b5f61069082610667565b9050919050565b6106a081610686565b82525050565b5f6020820190506106b95f830184610697565b92915050565b7f4e487b71000000000000000000000000000000000000000000000000000000005f52601160045260245ffd5b5f6106f68261046d565b91506107018361046d565b9250828201905080821115610719576107186106bf565b5b92915050565b6107288161046d565b82525050565b5f6060820190506107415f830186610697565b61074e602083018561071f565b61075b604083018461071f565b949350505050565b5f6020820190506107765f83018461071f565b92915050565b610e8c806107895f395ff3fe608060405234801561000f575f5ffd5b506004361061009c575f3560e01c80635b70ea9f116100645780635b70ea9f1461015a57806370a082311461016457806395d89b4114610194578063a9059cbb146101b2578063dd62ed3e146101e25761009c565b806306fdde03146100a0578063095ea7b3146100be57806318160ddd146100ee57806323b872dd1461010c578063313ce5671461013c575b5f5ffd5b6100a8610212565b6040516100b59190610b05565b60405180910390f35b6100d860048036038101906100d39190610bb6565b6102a2565b6040516100e59190610c0e565b60405180910390f35b6100f66102c4565b6040516101039190610c36565b60405180910390f35b61012660048036038101906101219190610c4f565b6102cd565b6040516101339190610c0e565b60405180910390f35b6101446102fb565b6040516101519190610cba565b60405180910390f35b610162610303565b005b61017e60048036038101906101799190610cd3565b610319565b60405161018b9190610c36565b60405180910390f35b61019c61035e565b6040516101a99190610b05565b60405180910390f35b6101cc60048036038101906101c79190610bb6565b6103ee565b6040516101d99190610c0e565b60405180910390f35b6101fc60048036038101906101f79190610cfe565b610410565b6040516102099190610c36565b60405180910390f35b60606003805461022190610d69565b80601f016020809104026020016040519081016040528092919081815260200182805461024d90610d69565b80156102985780601f1061026f57610100808354040283529160200191610298565b820191905f5260205f20905b81548152906001019060200180831161027b57829003601f168201915b5050505050905090565b5f5f6102ac610492565b90506102b9818585610499565b600191505092915050565b5f600254905090565b5f5f6102d7610492565b90506102e48582856104ab565b6102ef85858561053e565b60019150509392505050565b5f6012905090565b6103173369d3c21bcecceda100000061062e565b565b5f5f5f8373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f20549050919050565b60606004805461036d90610d69565b80601f016020809104026020016040519081016040528092919081815260200182805461039990610d69565b80156103e45780601f106103bb576101008083540402835291602001916103e4565b820191905f5260205f20905b8154815290600101906020018083116103c757829003601f168201915b5050505050905090565b5f5f6103f8610492565b905061040581858561053e565b600191505092915050565b5f60015f8473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f205f8373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f2054905092915050565b5f33905090565b6104a683838360016106ad565b505050565b5f6104b68484610410565b90507fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff8110156105385781811015610529578281836040517ffb8f41b200000000000000000000000000000000000000000000000000000000815260040161052093929190610da8565b60405180910390fd5b61053784848484035f6106ad565b5b50505050565b5f73ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff16036105ae575f6040517f96c6fd1e0000000000000000000000000000000000000000000000000000000081526004016105a59190610ddd565b60405180910390fd5b5f73ffffffffffffffffffffffffffffffffffffffff168273ffffffffffffffffffffffffffffffffffffffff160361061e575f6040517fec442f050000000000000000000000000000000000000000000000000000000081526004016106159190610ddd565b60405180910390fd5b61062983838361087c565b505050565b5f73ffffffffffffffffffffffffffffffffffffffff168273ffffffffffffffffffffffffffffffffffffffff160361069e575f6040517fec442f050000000000000000000000000000000000000000000000000000000081526004016106959190610ddd565b60405180910390fd5b6106a95f838361087c565b5050565b5f73ffffffffffffffffffffffffffffffffffffffff168473ffffffffffffffffffffffffffffffffffffffff160361071d575f6040517fe602df050000000000000000000000000000000000000000000000000000000081526004016107149190610ddd565b60405180910390fd5b5f73ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff160361078d575f6040517f94280d620000000000000000000000000000000000000000000000000000000081526004016107849190610ddd565b60405180910390fd5b8160015f8673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f205f8573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f20819055508015610876578273ffffffffffffffffffffffffffffffffffffffff168473ffffffffffffffffffffffffffffffffffffffff167f8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b9258460405161086d9190610c36565b60405180910390a35b50505050565b5f73ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff16036108cc578060025f8282546108c09190610e23565b9250508190555061099a565b5f5f5f8573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f2054905081811015610955578381836040517fe450d38c00000000000000000000000000000000000000000000000000000000815260040161094c93929190610da8565b60405180910390fd5b8181035f5f8673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f2081905550505b5f73ffffffffffffffffffffffffffffffffffffffff168273ffffffffffffffffffffffffffffffffffffffff16036109e1578060025f8282540392505081905550610a2b565b805f5f8473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020015f205f82825401925050819055505b8173ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff167fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef83604051610a889190610c36565b60405180910390a3505050565b5f81519050919050565b5f82825260208201905092915050565b8281835e5f83830152505050565b5f601f19601f8301169050919050565b5f610ad782610a95565b610ae18185610a9f565b9350610af1818560208601610aaf565b610afa81610abd565b840191505092915050565b5f6020820190508181035f830152610b1d8184610acd565b905092915050565b5f5ffd5b5f73ffffffffffffffffffffffffffffffffffffffff82169050919050565b5f610b5282610b29565b9050919050565b610b6281610b48565b8114610b6c575f5ffd5b50565b5f81359050610b7d81610b59565b92915050565b5f819050919050565b610b9581610b83565b8114610b9f575f5ffd5b50565b5f81359050610bb081610b8c565b92915050565b5f5f60408385031215610bcc57610bcb610b25565b5b5f610bd985828601610b6f565b9250506020610bea85828601610ba2565b9150509250929050565b5f8115159050919050565b610c0881610bf4565b82525050565b5f602082019050610c215f830184610bff565b92915050565b610c3081610b83565b82525050565b5f602082019050610c495f830184610c27565b92915050565b5f5f5f60608486031215610c6657610c65610b25565b5b5f610c7386828701610b6f565b9350506020610c8486828701610b6f565b9250506040610c9586828701610ba2565b9150509250925092565b5f60ff82169050919050565b610cb481610c9f565b82525050565b5f602082019050610ccd5f830184610cab565b92915050565b5f60208284031215610ce857610ce7610b25565b5b5f610cf584828501610b6f565b91505092915050565b5f5f60408385031215610d1457610d13610b25565b5b5f610d2185828601610b6f565b9250506020610d3285828601610b6f565b9150509250929050565b7f4e487b71000000000000000000000000000000000000000000000000000000005f52602260045260245ffd5b5f6002820490506001821680610d8057607f821691505b602082108103610d9357610d92610d3c565b5b50919050565b610da281610b48565b82525050565b5f606082019050610dbb5f830186610d99565b610dc86020830185610c27565b610dd56040830184610c27565b949350505050565b5f602082019050610df05f830184610d99565b92915050565b7f4e487b71000000000000000000000000000000000000000000000000000000005f52601160045260245ffd5b5f610e2d82610b83565b9150610e3883610b83565b9250828201905080821115610e5057610e4f610df6565b5b9291505056fea2646970667358221220c2ace90351a6254148d1d6fc391d67d42f65e41f9290478674caf67a0ec34ec964736f6c634300081b0033

Reconstructing state or Data Availability

Rollups, unlike validiums, need to have their state on L1. If the only thing that's published to it is the state root, then the sequencer could withhold data on the state of the chain. Because it is the one proposing and executing blocks, if it refuses to deliver certain data (like a merkle path to prove a withdrawal on L1), people may not have any place to get it from and get locked out of the chain or some of their funds.

This is called the Data Availability problem. Sending the entire state of the chain on every new L2 batch is impossible; state is too big. As a first next step, what we could do is:

• For every new L2 batch, send as part of the commit transaction the list of transactions in the batch. Anyone who needs to access the state of the L2 at any point in time can track all commit transactions, start executing them from the beginning and reconstruct the state.

This is now feasible; if we take 200 bytes as a rough estimate for the size of a single transfer between two users (see this post for the calculation on legacy transactions) and 128 KB as a reasonable transaction size limit we get around ~650 transactions at maximum per commit transaction (we are assuming we use calldata here, blobs can increase this limit as each one is 128 KB and we could use multiple per transaction).

Going a bit further, instead of posting the entire transaction, we could just post which accounts have been modified and their new values (this includes deployed contracts and their bytecode of course). This can reduce the size a lot for most cases; in the case of a regular transfer as above, we only need to record balance updates of two accounts, which requires sending just two (address, balance) pairs, so (20 + 32) * 2 = 104 bytes, or around half as before. Some other clever techniques and compression algorithms can push down the publishing cost of this and other transactions much further.

This is called state diffs. Instead of publishing entire transactions for data availability, we only publish whatever state they modified. This is enough for anyone to reconstruct the entire state of the chain.

Detailed documentation on the state diffs spec.

How do we prevent the sequencer from publishing the wrong state diffs?

Once again, state diffs have to be part of the public input. With them, the prover can show that they are equal to the ones returned by the VM after executing all blocks in the batch. As always, the actual state diffs are not part of the public input, but their hash is, so the size is a fixed 32 bytes. This hash is then part of the batch commitment. The prover then assures us that the given state diff hash is correct (i.e. it exactly corresponds to the changes in state of the executed blocks).

There's still a problem however: the L1 contract needs to have the actual state diff for data availability, not just the hash. This is sent as part of calldata of the commit transaction (actually later as a blob, we'll get to that), so the sequencer could in theory send the wrong state diff. To make sure this can't happen, the L1 contract hashes it to make sure that it matches the actual state diff hash that is included as part of the public input.

With that, we can be sure that state diffs are published and that they are correct. The sequencer cannot mess with them at all; either it publishes the correct state diffs or the L1 contract will reject its batch.

Compression

Because state diffs are compressed to save space on L1, this compression needs to be proven as well. Otherwise, once again, the sequencer could send the wrong (compressed) state diffs. This is easy though, we just make the prover run the compression and we're done.

EIP 4844 (a.k.a. Blobs)

While we could send state diffs through calldata, there is a (hopefully) cheaper way to do it: blobs. The Ethereum Cancun upgrade introduced a new type of transaction where users can submit a list of opaque blobs of data, each one of size at most 128 KB. The main purpose of this new type of transaction is precisely to be used by rollups for data availability; they are priced separately through a blob_gas market instead of the regular gas one and for all intents and purposes should be much cheaper than calldata.

Using EIP 4844, our state diffs would now be sent through blobs. While this is cheaper, there's a new problem to address with it. The whole point of blobs is that they're cheaper because they are only kept around for approximately two weeks and ONLY in the beacon chain, i.e. the consensus side. The execution side (and thus the EVM when running contracts) does not have access to the contents of a blob. Instead, the only thing it has access to is a KZG commitment of it.

This is important. If you recall, the way the L1 ensured that the state diff published by the sequencer was correct was by hashing its contents and ensuring that the hash matched the given state diff hash. With the contents of the state diff now no longer accessible by the contract, we can't do that anymore, so we need another way to ensure the correct contents of the state diff (i.e. the blob).

The solution is to make the prover take the KZG commitment as a public input and the KZG proof as a private input, compute the state diffs after correctly executing a batch of blocks, and verify the proof to check that the commitment binds to the correct state diffs.

Because the KZG commitment is a public input, we can use the BLOBHASH EVM opcode to retrieve the blob rolling hash (which is just the hash of the KZG commitment and some other constant data) and compare it to the public input KZG commitment (which needs to be hashed too).

Execution witness

The purpose of the execution witness is to allow executing blocks without having access to the whole Ethereum state, as it wouldn't fit in a zkVM program. It contains only the state values needed during the execution.

An execution witness contains all the initial state values (state nodes, codes, storage keys, block headers) that will be read or written to during the blocks' execution.

An execution witness is created from a prior execution of the blocks. This execution can be done by a synced node, and it would expose the data to a RPC endpoint. This is the debug_executionWitness endpoint which is implemented by ethrex and other clients.

If this endpoint is not available, the prover needs to do the following:

1. execute the blocks (also called "pre-execution") to identify which state values will be accessed.
2. log every initial state value accessed or updated during this execution.
3. retrieve an MPT proof for each value, linking it (or its non-existence) to the initial state root hash, using the eth_getProof RPC endpoint of a synced node.

Steps 1 and 2 are data collection steps only - no validation is performed at this stage. The actual validation happens later inside the zkVM guest program. Step 3 involves more complex logic due to potential issues when restructuring the pruned state trie after value removals. In sections initial state validation and final state validation we explain what are pruned tries and in which case they get restructured.

If a value is removed during block execution (meaning it existed initially but not finally), two pathological cases can occur where the witness lacks sufficient information to update the trie structure correctly:

Case 1

Here, only leaf 1 is part of the execution witness, so we lack the proof (and thus the node data) for leaf 2. After removing leaf 1, branch 1 becomes redundant. During trie restructuring, it's replaced by leaf 3, whose path is the path of leaf 2 concatenated with a prefix nibble (k) representing the choice taken at the original branch 1, and keeping leaf 2's value.

branch1 = {c_1, c_2, ..., c_k, ..., c_16} # Only c_k = hash(leaf2) is non-empty
leaf2 = {value, path}
leaf3 = {value, concat(k, path)} # New leaf replacing branch1 and leaf2

Without leaf 2's data, we cannot construct leaf 3. The solution is to fetch the final state proof for the key of leaf 2. This yields an exclusion proof containing leaf 3. By removing the prefix nibble k, we can reconstruct the original path and value of leaf 2. This process might need to be repeated if similar restructuring occurred at higher levels of the trie.

Case 2

In this case, restructuring requires information about branch/ext 2 (which could be a branch or extension node), but this node might not be in the witness. Checking the final extension node might seem sufficient to deduce branch/ext 2 in simple scenarios. However, this fails if similar restructuring occurred at higher trie levels involving more removals, as the final extension node might combine paths from multiple original branches, making it ambiguous to reconstruct the specific missing branch/ext 2 node.

Comparative Analysis: debug_executionWitness Latency

This document presents the results of measurements conducted to analyze the latency improvements of the debug_executionWitness RPC method across different configurations.

Configurations

• Pre-Serialized (added in PR #5956): Execution witnesses are generated during payload execution, converted to RpcExecutionWitness, and stored as pre-serialized JSON bytes in the database. On read, the bytes are parsed directly to serde_json::Value without any additional traversal or serialization.
• Pre-Generated: Execution witnesses are generated during payload execution and stored in the database (but require encode_subtrie() traversal and serialization on each read).
• On-Demand: Execution witnesses are generated when calling debug_executionWitness.

Measurements

Conclusions

Pre-Serialized vs On-Demand

The average latency drops from 242 ms (on-demand) to 94 ms (pre-serialized), representing an improvement of approximately 61%. Median latency shows a similar improvement, decreasing from 224 ms to 91 ms.

Pre-Serialized vs Pre-Generated

The average latency drops from 131 ms (pre-generated) to 94 ms (pre-serialized), representing an additional improvement of approximately 28%. This improvement comes from eliminating the encode_subtrie() depth-first traversal that was previously performed on every read to convert ExecutionWitness to RpcExecutionWitness.

Pre-Generated vs On-Demand

The average latency drops from 242 ms (on-demand) to 131 ms (pre-generated), representing an improvement of approximately 46%. Median latency shows a similar improvement, decreasing from 224 ms to 130 ms.

Overall

Pre-serialized execution witnesses exhibit the tightest latency distribution. On-demand requests frequently experience high-latency spikes due to witness generation at request time. Pre-generating and pre-serializing execution witnesses during payload execution effectively eliminates most of these spikes and results in more predictable response times.

How These Measurements Were Done

These metrics were obtained from an ethrex node synced to the Ethereum mainnet.

Each configuration was measured over a contiguous range of blocks, and latency was recorded for each request.

How to Get Metrics

1. Ensure the node is synced.
   Use the --precompute-witnesses flag to generate and store execution witnesses upon receiving a newPayload message.

2. Enable debug logging for ethrex-replay by editing src/main.rs:21:

   add_directive(Directive::from(tracing::Level::DEBUG))
   

3. Run ethrex-replay using the --endless flag:

   cargo run -r -- blocks --endless --rpc-url [RPC_URL]
   

4. Filter logs to capture execution witness latency:

   cargo run -r -- blocks --endless --rpc-url [RPC_URL] 2>&1 | grep --line-buffered 'Got execution witness for block' | tee execution_witness_times.txt
   

5. Post-process the captured measurements to compute min, max, average, and median latencies.

Appendix

Full Measurement Data

Pre-Serialized

Total blocks analyzed: 199

Pre-Generated

Total blocks analyzed: 176

On-Demand

Total blocks analyzed: 176