Comprehensive Benchmarks

All backends, all sizes, all strategies

Overview

This page provides comprehensive benchmarks across all 4 backend/language combinations and all dataset sizes, clearly showing which computational strategy (YOCO+sparse vs FWL demeaning) was used in each case.

Strategy Selection

leanfe automatically selects the optimal strategy based on FE cardinality:

Condition	Strategy	Why
Max FE ≤ 10K, Total FE ≤ 20K, No IV	YOCO + Sparse	Compression + sparse matrices are efficient
Max FE > 10K or Total FE > 20K	FWL Demeaning	Avoids huge sparse matrices
IV/2SLS	FWL Demeaning	YOCO doesn’t support instruments

In these benchmarks, we use FE1=500 and FE2=100 levels (total=600), so YOCO + Sparse is always selected.

Benchmark Configuration

Dataset sizes: 100K, 1M, 10M, 100M observations
Fixed effects: FE1 (500 levels) + FE2 (100 levels)
Regressors: Binary treatment + discrete x1 (0, 1, 2)
Standard errors: IID (all use YOCO path)
Metrics: Runtime (seconds), Peak memory (MB), Strategy used

Python Benchmarks

Python Comprehensive Benchmark (IID Standard Errors)

==============================================================================================================

         Obs    Backend       Time       Memory     Strategy   Compressed      Ratio

--------------------------------------------------------------------------------------------------------------

     100,000     polars      0.48s       20.0MB     compress       83,280     83.28%
     100,000     duckdb      1.60s       51.1MB     compress       82,981     82.98%
   1,000,000     polars      1.47s       67.6MB     compress      278,175     27.82%
   1,000,000     duckdb      1.95s       86.7MB     compress      278,210     27.82%
  10,000,000     polars      2.34s       73.9MB     compress      300,000      3.00%
  10,000,000     duckdb      3.32s       94.5MB     compress      300,000      3.00%
 100,000,000     polars      9.81s       73.9MB     compress      300,000      0.30%
 100,000,000     duckdb     16.45s       94.5MB     compress      300,000      0.30%


Note: Strategy 'compress' = YOCO + Sparse Matrix path

R Benchmarks

R Comprehensive Benchmark (IID Standard Errors)

====================================================================================================

         Obs    Backend       Time     Strategy   Compressed      Ratio

----------------------------------------------------------------------------------------------------

     100,000     polars      0.44s     compress       83,016     83.02%
     100,000     duckdb      0.56s     compress       83,136     83.14%
   1,000,000     polars      0.45s     compress      278,491     27.85%
   1,000,000     duckdb      0.81s     compress      278,213     27.82%
  10,000,000     polars      1.02s     compress      300,000      3.00%
  10,000,000     duckdb      2.65s     compress      300,000      3.00%
 100,000,000     polars      9.44s     compress      300,000      0.30%
 100,000,000     duckdb     25.14s     compress      300,000      0.30%


Note: Strategy 'compress' = YOCO + Sparse Matrix path

Summary Table

The table below summarizes all 4 combinations across all dataset sizes:

Observations	Language	Backend	Strategy	Why This Strategy
100K - 100M	Python	Polars	YOCO + Sparse	FE1=500, FE2=100 → total 600 < 20K threshold
100K - 100M	Python	DuckDB	YOCO + Sparse	Same FE cardinality, same strategy
100K - 100M	R	Polars	YOCO + Sparse	Same FE cardinality, same strategy
100K - 100M	R	DuckDB	YOCO + Sparse	Same FE cardinality, same strategy

Why YOCO + Sparse is Always Selected Here

With FE1=500 and FE2=100 levels:

Max single FE: 500 ≤ 10,000 threshold ✓
Total FE levels: 600 ≤ 20,000 threshold ✓
No IV/2SLS: ✓

All conditions for YOCO are met, so the sparse matrix path is used for all benchmarks.

FWL Demeaning Benchmark (High-Cardinality FEs)

To demonstrate the FWL path, we use high-cardinality FEs (50K levels) which exceed the YOCO threshold:

High-Cardinality FE Benchmark (FE1=50K, FE2=1K → triggers FWL)

==========================================================================================

         Obs    Backend       Time     Strategy   Iterations

------------------------------------------------------------------------------------------

768
0
     100,000     polars      0.45s          fwl           13
0
0
     100,000     duckdb      1.35s          fwl           13
0
0
   1,000,000     polars      0.48s          fwl            4
0
0
   1,000,000     duckdb      1.55s          fwl            4


Note: With FE1=50K > 10K threshold, FWL demeaning is selected instead of YOCO

Performance Insights

Python Polars vs DuckDB

Polars: Fastest for in-memory data, higher memory usage
DuckDB: Slightly slower but constant memory regardless of dataset size

R Polars vs DuckDB

Polars: Generally faster, uses R’s polars bindings
DuckDB: Better memory efficiency, uses R’s DBI interface

Scaling Behavior

With YOCO compression, runtime scales with compressed data size, not original data size. The compression ratio depends on the number of unique (regressor, FE) combinations.