(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Markus Ojala writes:
Bayesian modeling is becoming mainstream in many application areas. Applying it needs still a lot of knowledge about distributions and modeling techniques but the recent development in probabilistic programming languages have made it much more tractable. Stan is a promising language that suits single analysis cases well. With the improvements in approximation methods, it can scale to production level if care is taken in defining and validating the model. The model described here is the basis for the model we are running in production with various additional improvements.
He begins with some background:
Online advertisers are moving to optimizing total revenue on ad spend instead of just pumping up the amount of conversions or clicks. Maximizing revenue is tricky as there is huge random variation in the revenue amounts brought in by individual users. If this isn’t taken into account, it’s easy to react to the wrong signals and waste money on less successful ad campaigns. Luckily, Bayesian inference allows us to make justified decisions on a granular level by modeling the variation in the observed data.
Probabilistic programming languages, like Stan, make Bayesian inference easy. . . .
Sure, we know all that. But then comes the new part, at least it’s new to me:
In this blog post, we describe our experiences in getting Stan running in production.
Ojala discusses the “Use case: Maximizing the revenue on ad spend” and provides lots of helpful detail—not just the Stan code itself, but background on how they set up the model, and intermediate steps such as the first try which didn’t work because the model was insufficiently constrained and they needed to add prior information. As Ojala puts it:
What’s nice about Stan is that our model definition turns almost line-by-line into the final code. However, getting this model to fit by Stan is hard, as we haven’t specified any limits for the variables, or given sensible priors.
His solution is multilevel modeling:
The issue with the first approach is that the ad set estimates would be based on just the data from the individual ad sets. In this case, one random large $1,000 purchase can affect the mean estimate of a single ad set radically if there are only tens of conversion events (which is a common case). As such large revenue events could have happened also in other ad sets, we can get better estimates by sharing information between the ad sets.
With multilevel modeling, we can implement a partial pooling approach to share information. . . .
It’s BDA come to life! (But for some reason this paper is labeled, “Tauman, Chapter 6.”)
He continues with model development and posterior predictive checking! I’m lovin it.
Also this excellent point:
After you get comfortable in writing models, Stan is an expressive language that takes away the need to write custom optimization or sampling code to fit your model. It allows you to modify the model and add complexity easily.
Now let’s get to the challenges:
The model fitting can easily crash if the learning diverges. In most cases that can be fixed by adding sensible limits and informative priors for the variables and possibly adding a custom initialization for the parameters. Also non-centered parametrization is needed for hierarchical models.
These are must-haves for running the model in production. You want the model to fit 100% of the cases and not just 90% which would be fine in interactive mode. However, finding the issues with the model is hard.
What to do?
The best is to start with really simple model and add stuff step-by-step. Also running the model against various data sets and producing posterior plots automatically helps in identifying the issues early.
And some details:
We are using the PyStan Python interface that wraps the compilation and calling of the code. To avoid recompiling the models always, we precompile them and pickle them . . .
For scheduling, we use Celery which is a nice distributed task queue. . . .
We are now running the revenue model for thousands of different accounts every night with varying amount of campaigns, ad sets and revenue observations. The longest run takes couple minutes. Most of our customers still use the conversion optimization but are transitioning to use the revenue optimization feature. Overall, about one million euros of advertising spend on daily level is managed with our Predictive Budget Allocation. In future, we see that Stan or some other probabilistic programming language plays a big role in the optimization features of Smartly.io.
That’s awesome. We used the BSD license for Stan so it could be free and open source and anyone could use it inside their software, however they like. This sort of thing is exactly what we were hoping to see.
The post How smartly.io productized Bayesian revenue estimation with Stan appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
