Performance Benchmarks

Comparing leanfe vs pyfixest and fixest

Overview

leanfe is designed for speed and memory efficiency. It automatically selects the optimal algorithm based on your data:

  • YOCO compression + sparse matrices for low/medium-cardinality FEs (< 10K levels per FE, < 20K total)
  • FWL iterative demeaning for high-cardinality FEs (> 10K levels per FE or > 20K total)

Both approaches produce identical results to traditional methods, just faster.

TipComprehensive Benchmarks

For detailed benchmarks across all 4 backend/language combinations (Python/Polars, Python/DuckDB, R/Polars, R/DuckDB) and all dataset sizes (100K to 100M), see Comprehensive Benchmarks.

How It Works

Smart Strategy Selection

leanfe analyzes your fixed effects and automatically chooses the fastest approach:

FE Cardinality Strategy Why
Low (< 10K levels) YOCO + Sparse Compression reduces data dramatically; sparse matrices are efficient
High (> 10K levels) FWL Demeaning Avoids huge sparse matrices; iterative demeaning scales better

You don’t need to configure anything — leanfe picks the optimal strategy automatically.

YOCO Compression (Low-Cardinality FEs)

From Wong et al. (2021):

  1. Compress data: GROUP BY (regressors + FEs) → compute sufficient statistics per group
  2. Build sparse design matrix: FE dummies as scipy.sparse/Matrix
  3. Solve weighted least squares: On compressed data with sparse operations

FWL Demeaning (High-Cardinality FEs)

For FEs with many levels (e.g., customer IDs):

  1. Iteratively demean Y and X within each FE group
  2. Sort FEs by cardinality (low-card first) for faster convergence
  3. Run OLS on fully demeaned data

Dual Backend Architecture

Backend Strengths Best For
Polars Blazing fast, lazy evaluation, zero-copy Speed when data fits in memory
DuckDB Streams from disk, constant memory Datasets larger than RAM

Both backends use the same smart strategy selection. The choice is about speed vs memory, not features.

See Clustered SE Benchmarks for detailed performance analysis of cluster-robust inference.

Key Findings

Package Speed Memory Best For
leanfe-Polars ⚡ Fastest Moderate Maximum speed, in-memory data
leanfe-DuckDB Fast 💾 Minimal Large datasets, memory-constrained
pyfixest Slow High Compatibility with fixest syntax
fixest Fast High R users who need maximum compatibility

Key takeaways:

  • Python: leanfe-Polars is 10-15x faster than pyfixest with 20x less memory
  • R: leanfe competes with fixest on speed while using less memory
  • DuckDB backend: Enables processing datasets larger than RAM

Benchmark Results

All benchmarks use standardized data generation (see Methodology):

  • FE1: 500 levels, FE2: 100 levels
  • Regressors: Binary treatment + discrete x1 (0, 1, 2)
  • True coefficients: treatment=2.5, x1=1.5

Python Benchmarks

Python IID Standard Errors Benchmark (Two-way FE)
=========================================================================================================
         Obs  leanfe-Polars  leanfe-DuckDB     pyfixest    Speedup     Strategy
---------------------------------------------------------------------------------------------------------
     100,000          0.46s          0.66s       14.19s      30.6x     compress
   1,000,000          1.49s          2.01s        8.43s       5.6x     compress
  10,000,000          2.31s          3.41s      skipped          -     compress

Note: pyfixest skipped for 10M rows (too slow)

R Benchmarks

Note

R benchmarks compare leanfe (Polars and DuckDB backends) against fixest, the gold standard for fixed effects in R.

R IID Standard Errors Benchmark (Two-way FE)
========================================================================================================= 
         Obs  leanfe-Polars  leanfe-DuckDB       fixest    Speedup     Strategy
--------------------------------------------------------------------------------------------------------- 
     100,000          0.52s          0.67s        0.06s       0.1x     compress
   1,000,000          0.60s          0.68s        0.46s       0.8x     compress
  10,000,000          1.13s          2.85s        1.75s       1.5x     compress

Performance Charts (Python)

Python Performance Comparison: leanfe vs pyfixest

Memory Efficiency

DuckDB can process datasets larger than available RAM by streaming from parquet files. The memory measurements above show peak Python heap usage during regression.

When to Use Each Backend

Use Polars (default) when:

  • Speed is the priority
  • Data fits comfortably in memory
  • Running many regressions on the same data

Use DuckDB when:

  • Dataset is larger than available RAM
  • Memory is constrained
  • Reading directly from parquet files

Compression Ratio

The compression ratio depends on the number of unique (regressor, FE) combinations. With discrete regressors (binary treatment + categorical x1), YOCO achieves significant compression:

# Show compression across different dataset sizes
print("Compression Ratio by Dataset Size")
Compression Ratio by Dataset Size
print("=" * 50)
==================================================
print(f"{'Observations':>15} {'Compressed':>15} {'Ratio':>10}")
   Observations      Compressed      Ratio
print("-" * 50)
--------------------------------------------------
for r in py_results:
    n_compressed = int(r["n_obs"] * r["compression"])
    print(f"{r['n_obs']:>15,} {n_compressed:>15,} {r['compression']:>9.1%}")
        100,000          83,280     83.3%
      1,000,000         278,180     27.8%
     10,000,000         300,000      3.0%

Coefficient Validation

All benchmarks verify that both backends produce identical coefficients:

# Verify coefficients match
np.random.seed(42)
n = 100_000
df = pl.DataFrame({
    "y": np.random.normal(0, 1, n),
    "treatment": np.random.binomial(1, 0.3, n).astype(float),
    "x1": np.random.normal(0, 1, n),
    "fe1": np.random.randint(1, 101, n),
    "fe2": np.random.randint(1, 51, n),
})

result_polars = leanfe(formula="y ~ treatment + x1 | fe1 + fe2", data=df, vcov="iid", backend="polars")
result_duckdb = leanfe(formula="y ~ treatment + x1 | fe1 + fe2", data=df, vcov="iid", backend="duckdb")

print("Coefficient Comparison:")
Coefficient Comparison:
print(f"  Polars treatment: {result_polars['coefficients']['treatment']:.6f}")
  Polars treatment: -0.002710
print(f"  DuckDB treatment: {result_duckdb['coefficients']['treatment']:.6f}")
  DuckDB treatment: -0.002710
print(f"  Difference: {abs(result_polars['coefficients']['treatment'] - result_duckdb['coefficients']['treatment']):.2e}")
  Difference: 3.90e-17

Methodology

See Benchmark Methodology for details on:

  • Data generation process
  • Hardware and software specifications
  • Validation approach
  • Reproducible benchmark scripts