PoC Validates Neural Recommendations Over Rule-Based Matching

Background 

Rule-based recommendation systems can model product attributes accurately, but they cannot learn from behavior. When a platform reaches millions of active users, the commercial ceiling of attribute-only matching becomes visible: cross-category purchase patterns go unrecognized, and users stay within known preference profiles regardless of what similar buyers are doing. For marketplaces where discovery drives conversion, the absence of collaborative intelligence signals is not a technical limitation so much as a revenue one. 

Challenge 

The client’s goal was to move its recommendation engine from product-attribute similarity into behavioral intelligence. Rather than matching wines on characteristics alone, the platform needed a system that could learn from actual purchase behavior across its user base and surface wines a user was likely to buy, not just likely to enjoy on paper. 

The existing system used pure rule-based similarity matching with manual feature weighting. It had no capacity to learn from purchase history and optimized only for wine similarity, not for conversion, retention, or revenue. At this scale, the system produced a filter bubble effect: users received recommendations that stayed within their established taste profile, with no signal from what comparable buyers were actually purchasing. Cross-category purchase patterns, and the commercial opportunity they represent, went unrealized. 

Before committing to a full build, the client wanted a validated proof of concept. Stage 1 required demonstrating that an ML-based retrieval architecture was viable on the client’s existing data, met defined offline metric thresholds, and passed qualitative review by subject matter experts. 

The specific requirements for Stage 1: 

• Design a retrieval model that learns from user purchase behavior, not just product attributes 
• Validate ML viability on existing AWS RDS MySQL data: purchase history, user profiles, wine catalog, and ratings 
• Implement cold start handling for new wines and new users 
• Evaluate retrieval quality using offline metrics (Recall@K) 
• Demonstrate viable candidate generation meeting Stage 1 success criteria 
• Obtain SME qualitative review sign-off and stakeholder approval to proceed to Stage 2 

Build the data foundation your
ML models need to perform

DISCOVER MORE

Solution

We began with a problem framing and data readiness assessment before any model development. Reviewing the client’s existing AWS RDS MySQL data, including purchase history, user profiles, wine catalog, and ratings, confirmed the data was viable for ML and shaped the architecture decisions that followed. 

1. Embedding Model Design. We designed and trained a Two-Tower (Dual-Encoder) Neural Network as the core of the retrieval layer. The model learns to represent both users and wines as vectors in a shared embedding space, enabling retrieval driven by behavioral signals rather than manually assigned feature weights. Users with similar purchase histories converge in embedding space, so the model can surface wines that comparable buyers have chosen even when they fall outside a user’s established taste profile. 
2. Vector Store and ANN Retrieval. We built a vector store using Milvus to hold the wine embeddings and enable fast approximate nearest-neighbor (ANN) search. On each query, the retrieval layer surfaces the top-K candidate wines most relevant to the requesting user’s learned preference vector. The ANN approach keeps retrieval fast at the platform’s catalog scale without sacrificing meaningful recommendation quality. 
3. Cold Start Handling. New wines receive immediate embeddings via wine metadata, bypassing the need for purchase history before they can be recommended. New users receive segment-based fallback recommendations until behavioral data accumulates. This meant the system could function from launch without requiring a purchase history from every user on the platform. 
4. PoC Validation and Baseline Evaluation. Stage 1 success was defined against specific, agreed criteria before any build began: offline metrics had to meet minimum thresholds, candidate generation had to be demonstrably viable, and sample recommendations had to pass SME qualitative review. To measure retrieval quality rigorously, we evaluated the model against two popularity baselines on real held-out purchase data: engagement-driven popularity (what users pay attention to) and purchase-driven popularity (what users actually buy). The comparison covered three candidate thresholds (top-100, top-1,000, and top-5,000) on both per-user and purchase-weighted measurements. 

Key technical implementations: 

• AWS SageMaker as the ML training and experimentation environment 
• Two-Tower (Dual-Encoder) Neural Network for shared user and wine embedding 
• Milvus Vector Store for embedding storage and ANN search 
• Recall@K as the primary offline retrieval evaluation metric 
• Batch processing pipeline for embedding generation and updates 
• MLFlow for experiment tracking and configuration management 
• Metadata-based embedding for cold start on new wines; segment-based fallback for new users 

We treated Stage 1 as a gated validation exercise, not a shortcut to production. Every architectural decision was made with Stage 2 scale in mind, so the client would have a foundation to extend rather than rebuild. 

Every major design choice was documented as we went. That includes the places where we caught ourselves early, reversed course, or simplified scope deliberately. 

Results

We evaluated the retrieval model against two popularity baselines on real held-out purchase data: engagement-driven popularity and purchase-driven popularity. The model outperformed both baselines across every measured candidate threshold. 

Key findings: 

• Strongest gains at the top-100 threshold, where retrieval quality matters most for user experience 
• Lift held across both per-user and purchase-weighted measurements, confirming genuine personalization rather than rediscovery of popularity 
• Sample recommendations passed SME qualitative review 
• All Stage 1 success criteria met; project approved to proceed to ranking-model development (Stage 2) 

From proof of concept to
production-ready AI that scales

SEE HOW