Test Fixtures

Generated Artifacts Layer — JSON test vectors from executable specification

Definition

Test fixtures are structured test vectors generated directly from the executable specification. They define input states, operations, and expected outputs in a format that any implementation can consume to verify conformance.

Role in the Framework

Test fixtures are the enforcement mechanism for the executable specification. They translate “the spec says this” into “prove your implementation does this.”

Executable Specification
    ↓
Test Fixtures ← generated from spec execution
    ↓
├── Client A: runs fixtures → pass/fail
├── Client B: runs fixtures → pass/fail
└── Client C: runs fixtures → pass/fail

Why Generated, Not Written

Test fixtures are generated from the executable specification, not hand-written:

Generated Fixtures	Hand-Written Tests
Guaranteed to match spec	May diverge from spec
Comprehensive coverage	May miss edge cases
Single source of truth	Multiple sources that may conflict
Automatically updated	Require manual maintenance

When the specification changes, fixtures regenerate automatically—ensuring tests always reflect current requirements.

Fixture Structure

Ethereum’s execution-spec-tests use a standardized structure:

{
  "test_name": {
    "description": "Validates SPEC-4.2.3: Transaction nonce verification",
    "pre": {
      "0x1234...": {
        "balance": "0x100000",
        "nonce": "0x5",
        "code": "0x",
        "storage": {}
      }
    },
    "transaction": {
      "sender": "0x1234...",
      "to": "0x5678...",
      "nonce": "0x5",
      "value": "0x1000",
      "gasLimit": "0x5208",
      "gasPrice": "0x1"
    },
    "post": {
      "0x1234...": {
        "balance": "0xfed78",
        "nonce": "0x6"
      },
      "0x5678...": {
        "balance": "0x1000"
      }
    },
    "postStateRoot": "0xabcd..."
  }
}

Generation Process

Fixtures are generated by running the executable specification:

def generate_state_transition_fixtures() -> List[Fixture]:
    """
    Generate state transition test fixtures from executable spec.
    """
    fixtures = []
    
    for scenario in load_test_scenarios():
        # Set up initial state per scenario
        pre_state = scenario.create_pre_state()
        
        # Execute using THE SPECIFICATION
        post_state = execute_block(pre_state, scenario.block)
        
        # Capture the fixture
        fixture = Fixture(
            name=scenario.name,
            spec_reference=scenario.spec_section,
            pre=serialize_state(pre_state),
            block=serialize_block(scenario.block),
            post=serialize_state(post_state),
            post_state_root=compute_state_root(post_state)
        )
        fixtures.append(fixture)
    
    return fixtures
 
def generate_vm_fixtures() -> List[Fixture]:
    """
    Generate EVM opcode test fixtures.
    """
    fixtures = []
    
    for opcode in ALL_OPCODES:
        for test_case in opcode_test_cases(opcode):
            # Set up EVM state
            evm = create_evm(
                code=test_case.code,
                stack=test_case.initial_stack,
                memory=test_case.initial_memory,
                gas=test_case.initial_gas
            )
            
            # Execute using THE SPECIFICATION
            execute_opcode(evm, opcode)
            
            fixture = Fixture(
                name=f"{opcode.name}_{test_case.name}",
                spec_reference=f"EVM-{opcode.value}",
                pre=serialize_evm_state(test_case),
                post={
                    "stack": evm.stack,
                    "memory": evm.memory.hex(),
                    "gas_remaining": evm.gas,
                    "pc": evm.pc
                }
            )
            fixtures.append(fixture)
    
    return fixtures

Fixture Categories

State Transition Tests

Verify block processing:

Pre-state + Block → Post-state
State root verification
Receipt verification

Transaction Tests

Verify transaction handling:

Valid transaction acceptance
Invalid transaction rejection (various reasons)
Gas calculation

VM Tests

Verify EVM operations:

Each opcode’s behavior
Stack operations
Memory operations
Gas consumption

Difficulty/Consensus Tests

Verify consensus rules:

Block header validation
Fork choice rules
Finality conditions

Fixture Format Details

{
  "_meta": {
    "generated_by": "execution-spec-tests",
    "spec_version": "cancun",
    "generated_at": "2024-03-15T10:30:00Z"
  },
  
  "state_transition_001": {
    "spec_reference": "FUNC-4.2.3",
    "description": "Simple value transfer",
    
    "env": {
      "currentNumber": "0x1",
      "currentTimestamp": "0x1234",
      "currentGasLimit": "0x1000000"
    },
    
    "pre": {
      "0xsender": {
        "balance": "0x10000",
        "nonce": "0x0",
        "code": "0x",
        "storage": {}
      }
    },
    
    "transaction": {
      "type": "0x2",
      "chainId": "0x1",
      "nonce": "0x0",
      "to": "0xrecipient",
      "value": "0x1000",
      "maxFeePerGas": "0x10",
      "maxPriorityFeePerGas": "0x1",
      "gasLimit": "0x5208",
      "data": "0x"
    },
    
    "expect": {
      "result": "valid",
      "post": {
        "0xsender": {
          "balance": "0xed98",
          "nonce": "0x1"
        },
        "0xrecipient": {
          "balance": "0x1000"
        }
      },
      "gasUsed": "0x5208",
      "stateRoot": "0x..."
    }
  }
}

Client Integration

Clients consume fixtures to verify conformance:

// Geth test runner
func TestStateTransitions(t *testing.T) {
    fixtures := LoadFixtures("state_transition_tests.json")
    
    for _, fixture := range fixtures {
        t.Run(fixture.Name, func(t *testing.T) {
            // Build pre-state
            state := BuildState(fixture.Pre)
            
            // Apply transaction
            result, err := ApplyTransaction(state, fixture.Transaction)
            
            // Verify against fixture
            if fixture.Expect.Result == "valid" {
                require.NoError(t, err)
                require.Equal(t, fixture.Expect.StateRoot, state.Root())
                require.Equal(t, fixture.Expect.GasUsed, result.GasUsed)
            } else {
                require.Error(t, err)
            }
        })
    }
}

Downstream Dependencies

Downstream	Relationship
Client Implementations	Must pass all fixtures
Conformance Tests	Execute fixtures against clients
Compliance Matrix	Track fixture pass rates
Audit Trails	Record fixture test results

Fixture Maintenance

When specifications change:

1. Specification updated (prose or executable)
           ↓
2. Fixture generation re-run
           ↓
3. New/changed fixtures identified
           ↓
4. Clients updated to pass new fixtures
           ↓
5. Old fixtures deprecated/removed as appropriate

Quality Criteria

Generated: From executable spec, not hand-written
Comprehensive: Cover all specified behaviors
Isolated: Each fixture tests one thing
Reproducible: Same inputs always give same fixtures
Documented: Include spec references and descriptions
Versioned: Track which spec version generated them

Best Practices

Never edit generated fixtures manually
Include spec references in fixture metadata
Generate fixtures for positive AND negative cases
Version fixtures with specification version
Provide fixture validation tools for clients
Document fixture format comprehensively

Logos Assembly

Explorer

09-test-fixtures