System & Governance

Methodology, Data Limitations & Next Data Strategy

Technical specifications, dataset boundaries, explicit data limitations, and the next-generation data acquisition roadmap.

Processing Architecture & Offline Pipeline

RAW CSVs (~13.7 GB)
  ↓ DuckDB Engine CLI (main.py)
In-Memory Relational Views & Single-Pass SQL
  ↓ Compact Aggregations
22 Analytical JSON Outputs (<500 KB total)
  ↓ Next.js Vercel Production Runtime (Sub-millisecond sub-assembly)

Heavy computation runs offline via analytics/main.py using DuckDB disk-spilling and configurable memory limits (`--memory-limit 2GB`).

What This Dataset Cannot Tell Us (Data Limitations)

Observation Window Limit: Dataset covers only 2 calendar months (Oct–Nov 2019). Long-term customer lifetime value (LTV) cannot be calculated.
Missing Order Groupings: No `order_id` field. Multiple product purchases within a session are evaluated as discrete item purchase events.
No Financial Margins / COGS: Revenue represents Gross Merchandise Value (GMV) proxy. Net profitability and COGS are unobserved.
No Marketing Attribution: No UTM campaign tags, referral sources, or Ad spend data. Customer Acquisition Cost (CAC) cannot be calculated.
No Demographics or Geography: User segmentation is strictly behavioral based on clickstream activity.

Next Data Acquisition Roadmap (Event Schema V2)

Recommended schema upgrades for the data engineering team to unlock profitability modeling and marketing attribution.

Field to Collect	Business Question Enabled	Decision Enabled	Priority
order_id & item_quantity	Can we measure exact order value, basket size, and unit volumes?	Cart basket optimization and exact order fulfillment cost modeling.	Critical (P1)
discount_amount & coupon_code	Which promo campaigns actually drive profitable incremental lift vs margin erosion?	Promotion spend allocation and pricing strategy.	Critical (P1)
marketing_channel & UTM source	What is our customer acquisition cost (CAC) and channel ROI?	Ad spend reallocation across Google, Facebook, Affiliate, and Organic.	High (P2)
payment_gateway_status & failure_code	How much cart leakage is caused by payment gateway failures vs user hesitation?	Payment infrastructure reliability and fallback routing.	High (P2)