Benchmarks & Testing

Zentinel includes a comprehensive testing framework validating performance, stability, and resilience. All results below are from actual test runs.

Test Coverage Overview

Proxy Comparison Benchmarks

Five reverse proxies benchmarked head-to-head on the same hardware, same backend, same load. Native binaries only — no Docker overhead.

Test Environment

Methodology: Each proxy is started as a native binary, warmed up, then benchmarked for 60 seconds with oha. Resource usage (CPU, memory) is sampled every second via ps. Proxies are tested sequentially with 10-second cooldowns between runs to avoid contention.

Throughput

Latency Distribution

Resource Efficiency

Summary

Component Latency

Core operations like rate limiting and geo filtering are sub-100μs, meaning they add negligible overhead to request processing. The agent IPC overhead (100-500μs) is the primary cost of extensibility, which is why latency-critical features like rate limiting were moved into the core.

WAF Engine Benchmarks (zentinel-modsec)

Pure Rust ModSecurity implementation with full OWASP CRS compatibility (v0.1.4). Benchmarks run on macOS ARM64 using Criterion.

Throughput

Detection Operators

Rule Parsing

Optimized with lazy regex compilation for fast rule loading.

Rule parsing is now 3.9x faster than libmodsecurity for complex rules.

Transformations

Body Processing Throughput

Key Findings:

• Sub-microsecond transaction processing enables >900K requests/second
• XSS detection is faster than SQLi (57ns vs 123ns) due to optimized Aho-Corasick + RegexSet
• Pattern matching (@pm) uses Aho-Corasick for O(n) multi-pattern search
• Body processing scales linearly at ~40 MB/s throughput
• Pure Rust - zero C/C++ dependencies, full OWASP CRS compatibility

Rust vs C++: zentinel-modsec vs libmodsecurity

Rule Parsing Comparison

Soak Test Results

Extended duration tests validate stability and detect memory leaks or resource exhaustion.

1-Hour Soak Test (2026-01-01)

Memory Analysis

Key Findings:

• Memory decreased over time, demonstrating efficient Rust memory management
• Throughput remained stable throughout the test
• 99% of requests completed in under 62ms
• Connection errors (0.05%) occurred only during startup/shutdown phases

Leak Detection Methodology

Our analysis uses linear regression to detect memory growth patterns:

Analysis Method:

• Linear regression on memory samples over time
• R-squared correlation to detect consistent growth
• Monotonic increase ratio (>80% samples increasing = suspicious)

Test Configuration

From tests/soak/soak-config.kdl:

system {
    worker-threads 2
    max-connections 10000
    graceful-shutdown-timeout-secs 30
}

Chaos Testing

Fault injection validates resilience under failure conditions. All scenarios are in tests/chaos/.

Test Scenarios (10 Total)

Agent Failure Tests

Upstream Failure Tests

Resilience Tests

Circuit Breaker Behavior

The circuit breaker protects the system from cascading failures when agents become unhealthy. It operates as a state machine with three states:

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│   ┌──────────┐   5 failures   ┌──────────┐   10s timeout       │
│   │  CLOSED  │ ─────────────> │   OPEN   │ ─────────────┐      │
│   │ (normal) │                │(fast-fail│               │      │
│   └────▲─────┘                └──────────┘               │      │
│        │                                                 ▼      │
│        │  2 successes         ┌───────────┐                     │
│        └───────────────────── │ HALF-OPEN │ ◄───────────┘      │
│                               │  (probe)  │                     │
│                               └───────────┘                     │
│                                     │                           │
│                                     │ failure                   │
│                                     └──────────> back to OPEN   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

State Descriptions:

• CLOSED (green): Normal operation. All requests pass through to the agent. Failures are counted.
• OPEN (red): Circuit is tripped. Requests are immediately failed without contacting the agent. This prevents hammering a broken service.
• HALF-OPEN (yellow): After a timeout, the circuit allows a limited number of probe requests. If they succeed, the circuit closes. If they fail, it reopens.

Configuration from chaos-config.kdl:

circuit-breaker {
    failure-threshold 5      // Opens after 5 consecutive failures
    success-threshold 2      // Closes after 2 successes in half-open
    timeout-seconds 10       // Time before transitioning to half-open
    half-open-max-requests 2 // Probe requests allowed in half-open
}

Recovery Timeline

Recovery Times

Security Validation

WAF Testing with wafworth

598 test cases across 18 OWASP-aligned attack categories, tested against three WAF engine configurations on Zentinel. Results from wafworth run on 2026-03-04.

Overall Comparison

Per-Category Detection Rates

Integration Test Environment

Full integration tests run in Docker Compose with:

Running the Tests

Unit Tests

cargo test --workspace

Integration Tests

cd tests
./integration_test.sh           # Full suite
./integration_test.sh --quick   # Smoke tests only

Chaos Tests

cd tests/chaos
make quick                      # 4 core scenarios (~5 min)
make test                       # All 10 scenarios (~20 min)
make test-circuit-breaker       # Single scenario

Soak Tests

cd tests/soak
./run-soak-test.sh --duration 1    # 1 hour quick test
./run-soak-test.sh --duration 24   # Standard 24h test
./run-soak-test.sh --duration 72   # Extended 72h test

# Analyze results
python3 analyze-results.py results/<timestamp>/

Prometheus Metrics

Milestone Achievements

How to Reproduce

All benchmark scripts live in the main repository. To reproduce any result on your own hardware:

Proxy Comparison

git clone https://github.com/zentinelproxy/zentinel
cd zentinel

# Install proxies you want to compare (Zentinel, nginx, HAProxy, Caddy, Envoy)
# Then run the benchmark suite:
cd tests/bench
./proxy-bench.sh                    # All proxies, default settings
./proxy-bench.sh --proxy zentinel   # Single proxy
./proxy-bench.sh --duration 120     # Custom duration (seconds)
./proxy-bench.sh --connections 200  # Custom concurrency

The script uses oha as the load generator. Each proxy is started as a native binary (no Docker), warmed up for 5 seconds, then benchmarked for 60 seconds at the configured concurrency. CPU and memory are sampled every second via ps. Proxies are tested sequentially with 10-second cooldowns between runs.

Configuration: All proxies use equivalent minimal configs — single upstream, no TLS, no caching, keepalive enabled. Configs are in tests/bench/configs/.

WAF Benchmarks

cd tests/bench
./waf-bench.sh                      # zentinel-modsec vs libmodsecurity

Soak & Chaos Tests

cd tests/soak
./run-soak-test.sh --duration 1     # 1-hour quick test

cd tests/chaos
make quick                          # 4 core scenarios
make test                           # All 10 scenarios

Results are written to results/<timestamp>/ as JSON. The raw data behind the charts on this page is available at benchmark-results.json.

Known Gaps

We’re actively working on:

Last updated: March 2026 | Zentinel v0.4.2