Insights · Models that decide
Which threshold should your propensity model use?
If your team delivered a propensity model with good metrics and the campaign still fell short of the promised return, the problem probably isn't the model. It's the number that decides who gets the offer.
The question the model doesn't answer
A propensity model ranks customers by their likelihood of buying, churning, or paying. It doesn't tell you who to call tomorrow. That decision depends on three numbers that aren't in the model: what a contact costs, what a conversion is worth, and how many contacts the team can make.
A company has a customer base of 10,000 and a sales team. Each call costs €8 and each sale leaves €120 in margin. The model is good, with an AUC of about 0.78. With the same model, three threshold choices lead to three very different results:
- call everyone: a loss of about €3,600;
- use the library default threshold (0.5): almost no one gets a call, and the result is close to zero;
- use the cost-based threshold: a profit of about €27,600.
The numbers come from a simulation on synthetic data, modeled on a real problem. The difference between loss and profit isn't in the algorithm.
What usually goes wrong
- The default threshold stays in place. Almost every library defaults to 0.5. That value is right only when a sale is worth exactly twice what a contact costs.
- The threshold is picked by a technical metric. F1, Youden's index, "the point closest to the corner of the ROC curve": none of them uses what each error costs your business.
- The score isn't a probability. Models trained on rebalanced data or with class weights, and some algorithms (in the classic study, AdaBoost-style boosting and SVMs), tend to rank well but get the scale wrong. Apply the cost rule to a miscalibrated score and the threshold lands in the wrong place, even with the AUC intact.
- The threshold gets frozen. The purchase rate dips with the season, the margin changes, the team's cost changes, and the threshold is still the one set on deployment day.
- Nobody owns the threshold. The data team delivers the model, sales works the list, and the number that connects the two has no owner.
How to decide, in four steps
- Put a price on the error. The cost of a contact (C) and the margin on a conversion (V): what the sale leaves after costs and discounts, not the price. If more than one kind of error is expensive, write them all down.
- Make sure the score is a probability. Measure the calibration curve on a holdout set and calibrate if needed. Without that, any expected-value rule gets it wrong.
- Set the threshold by expected value, and respect capacity. With a calibrated score, and assuming people you don't contact wouldn't buy on their own, the optimal threshold is C/V: it pays to contact anyone whose probability of converting exceeds cost divided by value. If many would buy anyway, the question becomes who buys because of the contact, and that is an uplift problem. If the team can't reach everyone above the line, the list is cut off at capacity, and what matters is precision at the top.
- Review the threshold, not just the model. Recalculate the threshold when cost or margin changes, and recalibrate the score when the base rate shifts. And keep two small random groups: a few people below the threshold who do get contacted, to see what the threshold is missing, and a control group above it who don't, to measure the real effect of the outreach.
- a score decides who gets a contact, an offer, credit, or a review;
- the threshold was chosen by the technical team, without factoring in business costs;
- nobody recalculates the threshold when margin or cost changes;
- the model was trained on rebalanced data and nobody measured calibration;
- you don't know what happened to the people below the threshold.
In a short diagnostic, we look at your model, the cost of your errors, and your team's capacity, and tell you which threshold to use and how much money is on the table. If it isn't worth changing, we'll tell you that too.
In the Lab NotebookWant the math, the simulator, and the failure modes in detail?How to make money with a propensity model
Read the technical version →References
- Elkan, C. (2001). The Foundations of Cost-Sensitive Learning. IJCAI.
- scikit-learn: Tuning the decision threshold for class prediction.
- Niculescu-Mizil, A. & Caruana, R. (2005). Predicting Good Probabilities With Supervised Learning. ICML.
- Gutierrez, P. & Gérardy, J.-Y. (2017). Causal Inference and Uplift Modelling: A Review of the Literature. PMLR 67.