Building a Scalable Data Product for Behavioral Analytics in Luxury Fashion with dbt and AI
The company collected raw data through Google Analytics 4 (GA4), accumulating billions of interactions. However, the existing architecture, based on traditional business intelligence and visualization tools (such as Qlik and Denodo), could not process these volumes directly.
Attempts to run complex aggregations or joins on complete historical series caused the frontend systems to crash or to respond with unacceptable latency. In practice, the data existed but was “locked,” preventing the marketing and e-commerce teams from gaining insights into user segmentation, conversion funnels, and performance by geographic area (EMEAI, APAC, AMER, CHINA).
- Computational bottleneck: the visualization tools (Qlik/Denodo) could not act as a compute engine for billions of rows.
- No historical depth: deep historical trends could not be analyzed without compromising performance.
- Complex maintenance: no version control for SQL code and fragmented documentation, which made it hard to onboard new team members and maintain business logic.
- Limits on predictive analytics: without clean, structured datasets, machine learning algorithms could not be applied to cluster users.
The key aspect of this case study is the paradigm shift from plain ETL to a “Data Product”. It is not just about moving data, but about creating a certified, documented, ready-to-use data product. Thanks to the “Documentation-First” approach and the modular architecture, onboarding time for new developers was reduced by 70%. In addition, an incremental strategy made it possible to process only each day's new data, sharply cutting compute costs on BigQuery and delivering fresh data with a maximum delay (latency) that is controlled and monitored.
To overcome the infrastructure limits and enable advanced analytics, a cloud-based Modern Data Stack was implemented. The engagement covered the entire data value chain:
- Data Transformation Layer (dbt Labs): adoption of dbt (data build tool) to move transformation logic inside the data warehouse. A Medallion architecture (Bronze/Landing, Silver/Private, Gold/Public) was implemented, ensuring progressive data cleansing and separation of responsibilities.
- Orchestration and automation: Airflow schedules the pipelines and manages dependencies, ensuring that data is updated and available to the business every morning.
- Quality Assurance & CI/CD: integration of SQLFluff for code standardization and GitHub for version control. Automated alerts (via Slack) were implemented for data freshness, test failures, and cost thresholds being exceeded.
- AI evolution (behavioral clustering): building on the clean, structured data, a Vertex AI / BigQuery ML module is being implemented. Using unsupervised algorithms in Python, the system will be able to determine user behavior clusters on its own, moving beyond manual segmentation and providing advanced input to the frontend systems.
Let's now take a closer look at the Medallion architecture.
- Raw GA4 Data
- Minimal ETL
- Data Ingestion
- Core Models
- Aggregations
- Brand Filter
- Business Views
- Country Enrich
- Consumer Ready
- dbt Core (v1.10+) – Transformation framework used to write complex logic in modular SQL, with dependency management and automatic documentation.
- Google BigQuery – Serverless data warehouse chosen for its ability to scale to petabytes of data and for its native integration with GA4.
- Apache Airflow – Orchestration platform for managing workflows and monitoring pipelines.
- Python (v3.11+) – Used as the runtime environment and for future custom machine learning algorithms.
- SQLFluff & GitHub – Tools for code linting and versioning (CI/CD), ensuring software stability and quality.
- Vertex AI / BigQuery ML – Suite for implementing machine learning models (unsupervised clustering) directly on the data stored in the warehouse.