leanfe

Lean, Fast Fixed Effects Regression

🍃💨 leanfe

Lean, Fast Fixed Effects Regression for Python and R

High-dimensional fixed effects regression that’s fast when you need speed, and memory-efficient when you’re working with data larger than RAM.

Get Started View on GitHub

Why leanfe?

⚡ Speed

YOCO + sparse matrices compress data dramatically and solve much faster. Process millions of observations in seconds.

💾 Memory Efficient

DuckDB backend processes datasets larger than RAM by streaming from disk. Memory usage stays low regardless of dataset size.

🔄 Unified API

Same syntax in Python and R. Switch languages without relearning. The API works identically in both.

📊 Complete Features

Full econometrics toolkit: clustered standard errors, factor variables, interactions, IV/2SLS, and more.

How It Works: Smart Strategy Selection

leanfe automatically chooses the optimal algorithm based on your data characteristics. You don’t need to configure anything — leanfe analyzes your fixed effects and picks the fastest approach.

Two Algorithms, Automatically Selected

YOCO Compression + Sparse Matrices

Best for: Low/medium-cardinality FEs (< 10K levels)

From Wong et al. (2021): many observations share the same regressor values. leanfe groups identical rows together:

Original: 5M rows with binary treatment × 500 FE1 × 100 FE2
Compressed: ~100K unique combinations (98% reduction!)

Then builds sparse FE dummy matrices and solves weighted least squares on the compressed data. The math is lossless — coefficients and SEs are identical to full computation.

FWL Demeaning (Frisch-Waugh-Lovell)

Best for: High-cardinality FEs (> 10K levels)

When FEs have many levels (e.g., customer IDs), sparse matrices become huge. Instead, leanfe uses iterative within-group demeaning:

1. Demean Y and X within each FE group
2. Repeat until convergence
3. Run OLS on demeaned data

FEs are automatically sorted by cardinality (low-card first) for ~14% faster convergence.

Clustered Standard Errors: Sparse Cluster Matrix

For clustered SEs, leanfe implements Section 5.3.1 of Wong et al. (2021) using a sparse cluster indicator matrix:

• **W̃_C ∈ R^{G×C}**: Maps G groups to C clusters (one 1 per row)
• Vectorized aggregation: cluster_scores = W̃_Cᵀ @ (X × ẽ⁰) — no loops over clusters!

This makes clustered SEs just as fast as IID/HC1, even with thousands of clusters.

The Result

Dataset	pyfixest	leanfe	Speedup
1M rows	8.5s	0.6s	14x
10M rows	135s	9s	15x

View detailed benchmarks →

Two Backends: Speed vs Memory

leanfe offers two backends optimized for different scenarios:

⚡ Polars Backend (Default)

Best for: Maximum speed when data fits in memory

• Blazing fast in-memory DataFrame operations
• Lazy evaluation and query optimization
• Zero-copy data sharing with NumPy
• Ideal for interactive analysis and iteration

# Default: fastest option
result = leanfe(data=df, formula="y ~ x | fe", backend="polars")

💾 DuckDB Backend

Best for: Large datasets that exceed available RAM

• Streams data from disk (parquet files)
• Constant memory usage regardless of dataset size
• SQL-based query engine with automatic optimization
• Can process 100GB+ datasets on a laptop

# Memory-efficient: handles data larger than RAM
result = leanfe(data="huge_data.parquet", formula="y ~ x | fe", backend="duckdb")

Scenario	Recommended Backend
Data fits in memory, need speed	Polars
Data larger than RAM	DuckDB
Reading from parquet files	DuckDB
Running many regressions on same data	Polars
Memory-constrained environment	DuckDB

Both backends use YOCO compression + sparse matrices automatically for all SE types.

Quick Example

Two-way fixed effects regression with clustered standard errors:

Python (Polars)

from leanfe import leanfe

# Fast in-memory computation (default)
result = leanfe(
    formula="outcome ~ treatment + controls | unit_id + time_id",
    data=df,
    vcov="cluster",
    cluster_cols=["unit_id"],
    backend="polars"  # default, fastest
)

Python (DuckDB)

from leanfe import leanfe

# Memory-efficient, can handle data larger than RAM
result = leanfe(
    formula="outcome ~ treatment + controls | unit_id + time_id",
    data="large_data.parquet",  # read directly from file
    vcov="cluster",
    cluster_cols=["unit_id"],
    backend="duckdb"  # low memory usage
)

R (Polars)

library(leanfe)

# Fast in-memory computation (default)
result <- leanfe(
    formula = "outcome ~ treatment + controls | unit_id + time_id",
    data = df,
    vcov = "cluster",
    cluster_cols = c("unit_id"),
    backend = "polars"
)

R (DuckDB)

library(leanfe)

# Memory-efficient, can handle data larger than RAM
result <- leanfe(
    formula = "outcome ~ treatment + controls | unit_id + time_id",
    data = "large_data.parquet",
    vcov = "cluster",
    cluster_cols = c("unit_id"),
    backend = "duckdb"
)

Performance

YOCO + sparse matrices make leanfe extremely fast for all standard error types — IID, HC1, and clustered. DuckDB uses far less memory and can process datasets larger than RAM.

NoteLive Benchmarks

All performance numbers are generated from live benchmarks on standardized synthetic data. See the benchmark pages for current results on your hardware.

Key characteristics:

• Polars: Fastest execution, higher memory usage
• DuckDB: Slightly slower, minimal memory (can handle data larger than RAM)
• YOCO compression: Reduces data significantly for discrete regressors
• Scales to 50M+ observations: Tested on large datasets

View IID/HC1 benchmarks → | Clustered SE benchmarks →

Installation

Python

pip install leanfe

Or install from source:

pip install git+https://github.com/diegogentilepassaro/leanfe.git#subdirectory=package/python

R

# Install from GitHub
remotes::install_github("diegogentilepassaro/leanfe", subdir = "package/r")

Ready to dive in?

📖 Tutorials

Step-by-step guides for common use cases: DiD, event studies, IV regression.

Browse tutorials →

📚 API Reference

Complete documentation of all functions and parameters.

Python API | R API

🔬 Benchmarks

Reproducible performance comparisons with PyFixest and fixest.

View benchmarks →