LakeForge — open-source data lakehouse

Overview

Local lakehouse in Docker Compose: batch ingestion from HuggingFace into MinIO and ClickHouse with a Prefect flow, dbt bronze, silver and gold transformations, a React dashboard on the gold layer, Grafana and Superset already provisioned.

The challenge

Show a complete batch pipeline that starts with one command and handles millions of rows, not slides.

Architecture approach

Ingestion, storage, transformation and serving as separate services, coordinated by Prefect and observable from a single compose file.

The solution

Docker Compose with health checks and dependency ordering, benchmarks included, a custom dashboard on Amazon reviews (1 million rows by default, up to about 4 million) with sub-300 ms query benchmarks.

Key capabilities

• dbt medallion architecture with schema tests
• ClickHouse for analytics, benchmarks included (1 M rows by default)
• React dashboard on nginx with no credentials in the browser
• Compose with health checks, Prometheus, Grafana and Superset

Technology direction

Code on GitHub ↗

Project scope

The case study describes the project scope and the technical choices. Open-source projects have public code on GitHub; details of confidential projects are available on request.