LakeForge — open-source data lakehouse

Overview

Local lakehouse in Docker Compose: batch ingestion from HuggingFace into MinIO and ClickHouse with a Prefect flow, dbt bronze, silver and gold transformations, a React dashboard on the gold layer, Grafana and Superset already provisioned.

The challenge

Show a complete batch pipeline that starts with one command and handles millions of rows, not slides.

Architecture approach

Ingestion, storage, transformation and serving as separate services, coordinated by Prefect and observable from a single compose file.

The solution

Docker Compose with health checks and dependency ordering, benchmarks included, a custom dashboard on Amazon reviews (1 million rows by default, up to about 4 million) with sub-300 ms query benchmarks.

Key capabilities

• dbt medallion architecture with schema tests
• ClickHouse for analytics, benchmarks included (1 M rows by default)
• React dashboard on nginx with no credentials in the browser
• Compose with health checks, Prometheus, Grafana and Superset

Technology direction

Code on GitHub ↗

The facts

• 1 million rows by default, up to about 4 million
• 7 of 8 queries under 300 ms on ClickHouse
• 27 dbt models with schema tests, Compose with health checks