Get Started

Install leanfe and run your first regression in 5 minutes

Installation

Install leanfe using pip:

pip install leanfe

Or install the latest development version from GitHub:

pip install git+https://github.com/diegogentilepassaro/leanfe.git#subdirectory=package/python

Dependencies (installed automatically):

polars >= 0.19.0 - Fast DataFrame library
duckdb >= 0.9.0 - Embedded analytical database
numpy >= 1.20.0 - Numerical computing
pyarrow >= 12.0.0 - Arrow interchange format

Install from GitHub using remotes:

# Install remotes if needed
install.packages("remotes")

# Install leanfe
remotes::install_github("diegogentilepassaro/leanfe", subdir = "package/r")

Dependencies (installed automatically):

polars - Fast DataFrame library for R
duckdb - Embedded analytical database
arrow - For parquet file support

Quick Start

Let’s run a fixed effects regression on synthetic panel data. This example uses 1 million observations to demonstrate leanfe’s efficiency.

Why is leanfe so fast?

leanfe uses YOCO compression + sparse matrices from Wong et al. (2021) for all SE types:

Compression: Groups rows with identical regressors → dramatically fewer rows to process
Sparse matrices: FE dummies are mostly zeros → only store and compute non-zeros
Sparse cluster matrix: For clustered SEs, vectorized score aggregation (no loops over clusters!)

The result: Much faster than traditional methods, with identical coefficients and standard errors. See benchmarks for specific speedups.

Generate Sample Data

First, let’s create some synthetic panel data:

import numpy as np
import polars as pl

# Set seed for reproducibility
np.random.seed(42)

# Generate 1M observations
n_obs = 1_000_000
n_firms = 1_000
n_years = 20

# Create panel structure
df = pl.DataFrame({
    "firm_id": np.random.randint(1, n_firms + 1, n_obs),
    "year": np.random.randint(2000, 2000 + n_years, n_obs),
    "treatment": np.random.binomial(1, 0.3, n_obs).astype(float),
    "x1": np.random.normal(0, 1, n_obs),
    "x2": np.random.normal(0, 1, n_obs),
})

# Generate outcome with known coefficients
# True effect: treatment=2.5, x1=1.5, x2=0.8
firm_fe = np.random.normal(0, 1, n_firms + 1)[df["firm_id"].to_numpy()]
year_fe = np.random.normal(0, 0.5, n_years + 1)[df["year"].to_numpy() - 2000]
error = np.random.normal(0, 1, n_obs)

df = df.with_columns(
    y = pl.lit(2.5) * pl.col("treatment") 
      + pl.lit(1.5) * pl.col("x1") 
      + pl.lit(0.8) * pl.col("x2")
      + pl.lit(firm_fe) 
      + pl.lit(year_fe) 
      + pl.lit(error)
)

print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns}")
df.head(5)

Dataset shape: (1000000, 6)
Columns: ['firm_id', 'year', 'treatment', 'x1', 'x2', 'y']

shape: (5, 6)

firm_id	year	treatment	x1	x2	y
i64	i64	f64	f64	f64	f64
103	2017	0.0	1.041042	1.039711	3.346384
436	2016	1.0	-0.297066	-0.112271	1.074762
861	2018	0.0	-0.316687	-0.00083	1.82663
271	2016	0.0	-0.259329	-0.594086	0.165373
107	2017	1.0	0.845921	-0.013099	3.812536

library(dplyr)

# Set seed for reproducibility
set.seed(42)

# Generate 1M observations
n_obs <- 1000000
n_firms <- 1000
n_years <- 20

# Create panel structure
df <- tibble(
  firm_id = sample(1:n_firms, n_obs, replace = TRUE),
  year = sample(2000:(2000 + n_years - 1), n_obs, replace = TRUE),
  treatment = rbinom(n_obs, 1, 0.3),
  x1 = rnorm(n_obs),
  x2 = rnorm(n_obs)
)

# Generate outcome with known coefficients
# True effect: treatment=2.5, x1=1.5, x2=0.8
firm_fe <- rnorm(n_firms + 1)[df$firm_id]
year_fe <- rnorm(n_years + 1, sd = 0.5)[df$year - 1999]
error <- rnorm(n_obs)

df <- df %>%
  mutate(y = 2.5 * treatment + 1.5 * x1 + 0.8 * x2 + firm_fe + year_fe + error)

cat("Dataset shape:", nrow(df), "x", ncol(df), "\n")
head(df)

Run Fixed Effects Regression

Now let’s estimate the model with two-way fixed effects (firm and year):

from leanfe import leanfe

# Two-way fixed effects regression
# Formula: y ~ treatment + x1 + x2 | firm_id + year
result = leanfe(
    formula="y ~ treatment + x1 + x2 | firm_id + year",
    data=df,
    vcov="iid"  # IID standard errors
)

print(result)

{'coefficients': {'treatment': np.float64(2.4972912858420298), 'x1': np.float64(1.500822251196955), 'x2': np.float64(0.799761237842511)}, 'std_errors': {'treatment': np.float64(0.002185711151156035), 'x1': np.float64(0.0010012680232441217), 'x2': np.float64(0.0010015898528594395)}, 'n_obs': 1000000, 'n_compressed': 1000000, 'compression_ratio': 1.0, 'vcov_type': 'iid', 'strategy': 'compress', 'df_resid': 998979, 'rss': np.float64(999828.5831272217), 'n_clusters': None, 'iterations': 0, 'is_iv': False, 'n_instruments': None, 'formula': 'y ~ treatment + x1 + x2 | firm_id + year', 'fe_cols': ['firm_id', 'year']}

library(leanfe)

# Two-way fixed effects regression
result <- leanfe(
    formula = "y ~ treatment + x1 + x2 | firm_id + year",
    data = df,
    vcov = "iid"
)

print(result)

The estimated coefficients should be close to the true values: treatment ≈ 2.5, x1 ≈ 1.5, x2 ≈ 0.8.

Clustered Standard Errors

For panel data, you typically want clustered standard errors. leanfe supports two-way clustering:

# Two-way clustering (firm and year)
result_clustered = leanfe(
    formula="y ~ treatment + x1 + x2 | firm_id + year",
    data=df,
    vcov="cluster",
    cluster_cols=["firm_id", "year"]
)

print(result_clustered)

{'coefficients': {'treatment': np.float64(2.497291285841986), 'x1': np.float64(1.5008222511968927), 'x2': np.float64(0.7997612378424777)}, 'std_errors': {'treatment': np.float64(0.0023989980909454854), 'x1': np.float64(0.0009938208115052051), 'x2': np.float64(0.0010405597233752895)}, 'n_obs': 1000000, 'n_compressed': 1000000, 'compression_ratio': 1.0, 'vcov_type': 'cluster', 'strategy': 'compress', 'df_resid': 998979, 'rss': np.float64(999828.5831272218), 'n_clusters': 1000, 'iterations': 0, 'is_iv': False, 'n_instruments': None, 'formula': 'y ~ treatment + x1 + x2 | firm_id + year', 'fe_cols': ['firm_id', 'year']}

# Two-way clustering (firm and year)
result_clustered <- leanfe(
    formula = "y ~ treatment + x1 + x2 | firm_id + year",
    data = df,
    vcov = "cluster",
    cluster_cols = c("firm_id", "year")
)

print(result_clustered)

Choose Your Backend

leanfe offers two backends optimized for different scenarios:

Backend	Best For	Trade-off
Polars (default)	Maximum speed	Requires data to fit in memory
DuckDB	Large datasets, memory efficiency	Slightly slower, but minimal memory

Why two backends?

Polars: Uses lazy evaluation and zero-copy operations for blazing fast in-memory computation. Ideal when you’re iterating quickly or running many regressions on the same data.
DuckDB: Streams data from disk and uses SQL-based query optimization. Can process datasets larger than your RAM by reading directly from parquet files. Memory usage stays constant regardless of dataset size.

Both backends use YOCO compression + sparse matrices automatically for all SE types (IID, HC1, and clustered).

# Use DuckDB backend for memory efficiency
result_duckdb = leanfe(
    formula="y ~ treatment + x1 + x2 | firm_id + year",
    data=df,
    vcov="iid",
    backend="duckdb"  # Use DuckDB instead of Polars
)

print(result_duckdb)

{'coefficients': {'treatment': np.float64(2.497291285841983), 'x1': np.float64(1.500822251196876), 'x2': np.float64(0.7997612378425281)}, 'std_errors': {'treatment': np.float64(0.0021857111511560355), 'x1': np.float64(0.001001268023244112), 'x2': np.float64(0.0010015898528594384)}, 'n_obs': 1000000, 'n_compressed': 1000000, 'compression_ratio': 1.0, 'vcov_type': 'iid', 'strategy': 'compress', 'df_resid': 998979, 'rss': np.float64(999828.5831272218), 'n_clusters': None, 'iterations': 0, 'is_iv': False, 'n_instruments': None, 'formula': 'y ~ treatment + x1 + x2 | firm_id + year', 'fe_cols': ['firm_id', 'year']}

# Use DuckDB backend for memory efficiency
result_duckdb <- leanfe(
    formula = "y ~ treatment + x1 + x2 | firm_id + year",
    data = df,
    vcov = "iid",
    backend = "duckdb"
)

print(result_duckdb)

Next Steps

Now that you’ve run your first regression, explore more features:

Basic Usage Tutorial - Deep dive into formula syntax and options
Standard Errors - IID, HC1, and clustered SE options
Factor Variables - Categorical regressors with i() syntax
Difference-in-Differences - Causal inference with TWFE
Large Datasets Guide - Working with data larger than RAM