Checking systems...

Data Platform

Everything you need to build, monitor, and ship analytics across all environments.

4Environments·75+dbt Models·2AWS Regions·16DAGs

Platform Health

Loading

Pipeline delivery, data quality, and freshness across all environments.

Quick Access

Jump directly into any platform tool.

ClickHouse Cloud

Query console, cluster management, and performance monitoring

Data Warehouse

Hex

Notebooks, dashboards, and ad-hoc analysis for the team

BI & Analytics

Airflow (MWAA)

DAG monitoring, orchestration, and pipeline scheduling

StagingProd 1 (US)Prod 2 (EU)

Elementary

Data observability, test results, and anomaly reports

StagingProd 1 (US)Prod 2 (EU)

DataHub

Data catalog, lineage, and metadata discovery — coming soon

Catalog

CircleCI

CI/CD pipelines, deploy status, and build logs

CI/CD

Environments

Four ClickHouse Cloud clusters spanning two AWS regions.

Staging

Single Source
Region
us-east-1
Host
Console Airflow

Prod 1 (US)

Single Source
Region
us-east-1
Host
Console Airflow

Prod 2 (EU)

Single Source
Region
eu-west-1
Host
Console Airflow

Prod Analytics

Federated
Region
us-east-1
Host
Console

Architecture

Medallion architecture: from raw events to business-ready metrics.

Sources
Kafka (MSK)
Event streams
Kafka Connect
Iceberg Sink
S3 Iceberg
Parquet + metadata
Zuora AQUA
Billing exports
S3 Iceberg
Billing tables
Bronze
icebergS3() Views
Query-time, no copy
Materialized Tables
Local copies
Silver
Staging Models
ReplacingMergeTree
Intermediate
Business logic
Gold
Marts
KPIs, aggregations
Federation
remoteSecure()
Consumers
Hex
BI & dashboards
In-App
Customer-facing
Cube.js
SQL API
Elementary
Observability
DataHub
Catalog (planned)

Observability

Data quality, test results, and model health powered by Elementary.

Staging

US East

Prod 1 (US)

US East

Prod 2 (EU)

EU West

From the Team

Engineering deep-dives, platform updates, and lessons learned.

Mar 26, 2026Alex Silva

Introducing the Kustomer Data Platform Portal

A single hub for every tool, environment, and dashboard the data team needs — from ClickHouse to Elementary.

portalplatformlaunch
Read
Mar 20, 2026Alex Silva

How We Reduced ClickHouse Test Costs by 95%

Running dbt tests on every DAG run was burning through ClickHouse credits. Here's how we restructured testing to cut costs by 95% without losing coverage.

clickhousetestingcost
Read
Mar 15, 2026Alex Silva

Elementary: Data Observability Across All Environments

How we deployed Elementary data observability across staging, prod1, and prod2 — and made the reports available through the Data Platform Portal.

elementaryobservabilityairflow
Read
View all posts

Documentation

Architecture decisions, runbooks, and operational guides.

Architecture

System overview: environments, data layers, CI/CD, and credential management.

Architecture

Pipeline Walkthrough

Visual guide from Kafka events through Bronze/Silver/Gold to dashboards.

Architecture

Environment Setup Runbook

ClickHouse provisioning, grants, bronze setup, federation, and recovery.

Operations

Lessons Learned

Production incidents, OPTIMIZE FINAL pitfalls, vertical scaling, and debugging tips.

Operations

Test Optimization Strategy

How we reduced ClickHouse test costs by 95% with nightly test DAGs.

Testing

PII/PCI Masking

Macro-based masking at the silver layer for emails, financials, and free-text.

Security

ADR-001: Query Engine

Why we consolidated on ClickHouse and the tradeoffs considered.

ADR

Full Refresh Runbook

When and how to run dbt full-refresh safely across environments.

Operations

DAG Orchestration

Airflow DAG design, scheduling patterns, and concurrency strategy.

Operations

Getting Started

Up and running in three steps.

1

Clone & Install

Set up the dbt project and install dependencies.

git clone git@github.com:kustomer/kustomer-analytics.git
cd kustomer-analytics
python3 -m venv venv && source venv/bin/activate
pip install dbt-clickhouse==1.10.0
dbt deps
2

Local Development

Start local ClickHouse via Tilt and build models.

# In your local clouddev checkout
tilt up

# Back in kustomer-analytics
cp profiles.yml.example ~/.dbt/profiles.yml
dbt build --target clickhouse_local
3

Deploy

Open a PR — CI deploys to staging on merge. Prod needs approval.

git checkout -b feat/my-model
git push -u origin HEAD
# CircleCI: build → staging auto-deploy
# Approve prod1 / prod2 in CircleCI UI