{"body":"

How combining gradient boosting and reinforcement learning creates hyper-personalised online grocery shopping experiences

For retailers, offering customers a wide range of product promotions is key to balancing stock and offering value. Online, a large volume of promotions can mean consumers struggle to spot the offers most relevant to their preferences. So how can we improve this and serve individual consumers with highly personalised promotions?

Traditional machine learning models can make generalisations and offer broad recommendations. Reinforcement learning gives retailers an opportunity to be even more attuned and responsive to dynamically changing customer preferences.

How we developed our hyper-personal machine learning model

We created a machine learning model that uses customer shopping data as an input, combining gradient boosting with reinforcement learning to serve hyper-relevant promotions, effectively automatically building personalised virtual stores for millions of customers.

The Gradient Boosting Decision Tree (GBDT) is a state-of-the-art machine learning model that we combined with a contextual multi-armed bandit (MAB) approach.

The approach is based on training a GBDT offline, preprocessing its output and using it as one of the inputs to the contextual MAB Thompson sampling model. In a simulation using artificially generated data, we demonstrated that the new model outperformed GBDT alone. Online A/B testing proved that presenting customers with more personalised offers using our new model increased promotional purchase rates by more than 50%.

The power of gradient boosting…

GBDT models need very little data preprocessing. They also cope well with interactions in the model, missing data, outliers, insignificant variables, and a combination of categorical and numerical features. To understand which promotions customers are most likely to be interested in, we ranked them by click-through rate probability. This was estimated by a GBDT from TensorFlow.

We used a range of categorical and numerical features, including:

item attributes, e.g. price and recent popularity,
discount level and type,
general information, e.g. day of the week,
user-item interaction history, e.g. if a user has bought a similar item before.

… meets reinforcement learning

GBDTs allow us to make general assumptions about the effect features have across all items and users – reducing the need to consider promotions changing regularly. But we also need the flexibility to factor in item-specific information which might not be reflected in any of the features – like new, seasonal products, or very attractive new types of promotions. In grocery eCommerce, it ’s not possible to use product identifiers for this purpose, because the product range is constantly changing. We used algorithms designed for a multi-armed bandit problem, specifically, Thompson sampling, to help tackle this.

We combined gradient boosted trees and multi-armed bandit by inputting predictions from the GBDT (split into buckets and one-hot encoded) and one-hot encoded product identifiers into a linear Thompson sampling model.

Thompson sampling model

Thompson sampling solves the multi-armed bandit problem using a Bayesian approach, i.e. we update the model as more information becomes available. It is a logistic regression model where the coefficient vector is treated as a random variable; let ’s call the coefficient vector ?.

We assume a prior distribution of ?. At each moment in time ?, we can update this distribution ? based on all the actions, features, and rewards observed up to time ?; this is known as the posterior distribution. Subsequently, we generate ?_t by sampling from this posterior distribution. Having ?_t we can calculate the probability of purchasing a product and use it to rank products that will be shown to a customer.

The important part of our new model is hidden in the coefficient ?. In the beginning, we assume that it follows a normal distribution with a mean ? equal to 0 and a variance ?² (this is a hyperparameter of the model). As a result we initially get random coefficient values which can be either positive or negative – this is the exploration phase. In this phase, we show products randomly to discern which promotions are especially popular.

As the posterior distribution of ? is updated, the coefficient ’s mean converges to a specific value and ?² decreases, such that the spread of ?_t gets smaller with time. We can now use the gathered data to better determine the probability of purchase and be confident in which promotions are more attractive to customers.

This is visible on the plots below, which show the values of ? and ?² depending on the time for different products.

Figure 1: Mean of the posterior distribution of the model coefficients corresponding to one-hot encoded product identifiers.

Figure 2: Variance of the posterior distribution of the model coefficients corresponding to one-hot encoded product identifiers.

Results

By enhancing GBDT with Thompson sampling, we increased the proportion of customers who purchased something from our suggestions by 50%. This new ML model showed outstanding results in serving customers with more relevant promotions. In this way, we ’ve been able to create hyper-personalised shopping experiences – meaning retailers see more products sold, and customers get better deals.

","presentation":{"classes":"","theme":"theme-default","columns":""}}

Reinforcement Learning: creating a truly personalised virtual store