Hive Tests

End-to-End tests with Hive. Hive is a system which sends RPC commands to Ethereum clients and validates their responses. You can read more about it here.

Overview

This project uses three key repositories for Hive testing:

ethereum/hive - The main Hive testing framework
- Current commit: 0921fb7833e3de180eacdc9f26de6e51dcab0dba
ethereum/execution-specs - Test fixtures and vectors (the former ethereum/execution-spec-tests repo is archived)
- Mainnet: tests@v20.0.0 (Osaka + BPO1 + BPO2); Amsterdam: tests-glamsterdam-devnet@v7.2.0
ethereum/execution-specs - Fork specifications
- Current branch: forks/amsterdam

Prerequisites

Required Tools

Docker - Required for running containerized clients

# Install Docker following the official guide for your OS
# Verify installation
docker --version

Go - Required to build the Hive framework

Using asdf:
```
asdf plugin add golang https://github.com/asdf-community/asdf-golang.git
```
Uncomment the golang line in the .tool-versions file:
```
rust 1.90.0
golang 1.23.2
```
Then install:
```
asdf install
```
If you need to set GOROOT, follow these instructions.

Rust - Required to build the ethrex client

# Already configured in .tool-versions
asdf install rust

Build the ethrex Docker Image

Before running Hive tests, you need to build the ethrex Docker image:

make build-image

Running Hive Tests

1. Setup Hive Framework

The first time you run tests, Hive will be automatically cloned and built. You can also set it up manually:

make setup-hive

This will:

Clone the ethereum/hive repository at the configured commit
Build the Hive binary using Go
Place it in the ./hive directory

2. Basic Test Execution

Run Specific Simulation

Hive tests are organized by “simulations”. To run a specific simulation:

make run-hive SIMULATION=<simulation-name> TEST_PATTERN="<pattern>"

Available Simulations:

ethereum/rpc-compat - RPC API compatibility tests
devp2p - P2P networking tests (Discovery V4, Eth, Snap)
ethereum/engine - Engine API tests (Paris, Auth, Cancun, Withdrawals)
ethereum/sync - Node synchronization tests
ethereum/eels/consume-engine - EVM Execution Layer tests (Engine format)
ethereum/eels/consume-rlp - EVM Execution Layer tests (RLP format)
ethereum/eels/execute-blobs - Blob execution tests

Example: RPC Compatibility Tests

This is an example of a Hive simulation called ethereum/rpc-compat, which will specifically run chain id and transaction by hash rpc tests:

make run-hive SIMULATION=ethereum/rpc-compat TEST_PATTERN="/eth_chainId|eth_getTransactionByHash"

Run all RPC tests:

make run-hive SIMULATION=ethereum/rpc-compat TEST_PATTERN="*"

Run All Simulations

To run every available simulation:

make run-hive-all

3. Debug Mode

For detailed debug output including Docker container logs:

make run-hive-debug SIMULATION=ethereum/rpc-compat TEST_PATTERN="*"

Debug mode sets:

--sim.loglevel 4 (maximum verbosity)
--docker.output (shows Docker container output)

4. Amsterdam Fork Tests (EELS)

To test Amsterdam fork compatibility using the Ethereum Execution Layer Specification tests:

Run Engine Format Tests

Tests all forks including Amsterdam:

make run-hive-eels-engine

Test specific forks:

make run-hive-eels EELS_SIM=ethereum/eels/consume-engine TEST_PATTERN_EELS=".*fork_Amsterdam.*"

Test multiple forks:

make run-hive-eels EELS_SIM=ethereum/eels/consume-engine TEST_PATTERN_EELS=".*fork_Prague.*|.*fork_Amsterdam.*"

Run RLP Format Tests

make run-hive-eels-rlp

Or test specific forks:

make run-hive-eels EELS_SIM=ethereum/eels/consume-rlp TEST_PATTERN_EELS=".*fork_Amsterdam.*"

Run Blob Execution Tests

make run-hive-eels-blobs

5. Customizing Test Execution

Adjust Parallelism

Control how many tests run in parallel (default: 16):

make run-hive SIMULATION=ethereum/rpc-compat SIM_PARALLELISM=8

Adjust Log Level

Set log verbosity from 1 (least verbose) to 4 (most verbose):

make run-hive SIMULATION=ethereum/rpc-compat SIM_LOG_LEVEL=1

Test Pattern Examples

Match specific test names:

TEST_PATTERN="/eth_chainId"                    # Single test
TEST_PATTERN="/eth_chainId|eth_blockNumber"    # Multiple tests
TEST_PATTERN="/eth_get.*"                       # Regex pattern
TEST_PATTERN="*"                                # All tests

For fork-specific EELS tests:

TEST_PATTERN_EELS=".*fork_Amsterdam.*"                                    # Amsterdam only
TEST_PATTERN_EELS=".*fork_Paris.*|.*fork_Shanghai.*|.*fork_Cancun.*"     # Multiple forks

Viewing Results

Hive Web Interface

After running tests, view results in a web interface:

make view-hive

This starts a local server at http://127.0.0.1:8080 showing:

Test pass/fail status
Detailed logs for each test
Client information and errors

The web interface is automatically opened after make run-hive completes.

Command Line Results

Results are also available in the terminal output and in:

./hive/workspace/logs/

Log files include:

*.json - Test results in JSON format
*.log - Detailed execution logs
Client-specific logs for debugging failures

Generate Hive Report

To generate a formatted report from test results:

cargo run --manifest-path tooling/Cargo.toml -p hive_report

This reads all JSON files in hive/workspace/logs/ and produces:

Summary by category (Engine, P2P, RPC, Sync, EVM)
Pass/fail statistics per simulation
Fork-specific results (Paris, Shanghai, Cancun, Prague, Amsterdam)

Cleaning Up

Remove Hive logs and workspace:

make clean-hive-logs

Remove the entire Hive directory to force a fresh clone:

rm -rf ./hive

Repository Configuration

The project pins specific versions of the three repositories for Amsterdam fork support:

Makefile Configuration

# ethereum/hive branch
HIVE_BRANCH ?= master

Workflow Configuration (.github/workflows/daily_hive_report.yaml)

Fixtures come from ethereum/execution-specs (the archived ethereum/execution-spec-tests repo is no longer used). The workflow uses fork-specific fixtures to ensure comprehensive test coverage, pinned via config files under .github/config/hive/:

# Amsterdam tests use the glamsterdam-devnet bundle (.github/config/hive/amsterdam.yaml)
if [[ "$SIM_LIMIT" == *"fork_Amsterdam"* ]]; then
  FLAGS+=" --sim.buildarg fixtures=$AMSTERDAM_FIXTURES"
  FLAGS+=" --sim.buildarg branch=$AMSTERDAM_EELS_COMMIT"
else
  # Other forks use the mainnet bundle (.github/config/hive/mainnet.yaml),
  # which includes comprehensive coverage including ported static tests.
  FLAGS+=" --sim.buildarg fixtures=$MAINNET_FIXTURES"
  FLAGS+=" --sim.buildarg branch=$MAINNET_EELS_COMMIT"
fi

Fixtures URL Files

Local ef_tests runners (blockchain, state, engine) share one set of fixture URL files under tooling/ef_tests/:

tooling/ef_tests/.fixtures_url — mainnet bundle
tooling/ef_tests/.fixtures_url_amsterdam — Amsterdam (glamsterdam-devnet) fixtures with BAL support
tooling/ef_tests/.fixtures_url_zkevm — zkEVM (EIP-8025 stateless) fixtures

Contents:

# .fixtures_url
https://github.com/ethereum/execution-specs/releases/download/tests%40v20.0.0/fixtures.tar.gz

# .fixtures_url_amsterdam
https://github.com/ethereum/execution-specs/releases/download/tests-glamsterdam-devnet%40v7.2.0/fixtures_glamsterdam-devnet.tar.gz

The CI hive config lives in .github/config/hive/{mainnet,amsterdam}.yaml, each pinning a fixtures URL and an eels_commit (the execution-specs commit used to build the hive consumer).

Updating Repository Versions

To update to a different fork or newer versions:

Update Hive commit in Makefile:
```
HIVE_BRANCH ?= <new-commit-hash>
```
Update the hive fixture versions in .github/config/hive/:

For mainnet forks, edit .github/config/hive/mainnet.yaml:
```
fixtures: https://github.com/ethereum/execution-specs/releases/download/tests%40<version>/fixtures.tar.gz
eels_commit: <execution-specs-commit>
```
For Amsterdam, edit .github/config/hive/amsterdam.yaml the same way with the glamsterdam-devnet bundle.

Update fixtures URL files shared by the local runners (blockchain / state / engine):

# Mainnet fixtures
echo "https://github.com/ethereum/execution-specs/releases/download/tests%40<version>/fixtures.tar.gz" > tooling/ef_tests/.fixtures_url
# Amsterdam fixtures
echo "https://github.com/ethereum/execution-specs/releases/download/tests-glamsterdam-devnet%40<version>/fixtures_glamsterdam-devnet.tar.gz" > tooling/ef_tests/.fixtures_url_amsterdam

Update fork references in code if switching to a different fork:
- .github/workflows/daily_hive_report.yaml - Test names and patterns
- tooling/hive_report/src/main.rs - Fork ranking and result processing

Troubleshooting

Docker Permission Errors

If you encounter Docker permission errors:

sudo usermod -aG docker $USER
newgrp docker

Hive Build Failures

If Go build fails:

# Clean and rebuild
rm -rf ./hive
make setup-hive

Missing Test Fixtures

If EELS tests fail due to missing fixtures:

# Verify fixtures URL is accessible
curl -I $(cat tooling/ef_tests/.fixtures_url)

# Re-download fixtures (they're downloaded automatically during test execution)

Client Container Failures

If the ethrex client container fails to start:

# Rebuild the Docker image
make build-image

# Check Docker images
docker images | grep ethrex

CI/CD Integration

The project runs Hive tests automatically via GitHub Actions:

Daily runs: .github/workflows/daily_hive_report.yaml - Comprehensive test coverage
PR validation: .github/workflows/pr-main_l1.yaml - Subset of critical tests

Results are posted to Slack and available in GitHub Actions artifacts.

Ethrex