Hype or Reality? Stealing Machine Learning Models via Prediction APIs

Wired magazine just published an article with the interesting title How to Steal an AI, where the author explores the topic of reverse engineering Machine Learning algorithms based on a recently published academic paper: Stealing Machine Learning Models via Prediction APIs.

BigML was contacted by the author via email prior to the publication and within 24 hours we responded via a lengthy email that sums up our stance on the topic. Unfortunately, the article incorrectly stated that BigML did not respond. We are in the process of helping the author correct that omission. Update: the Wired article has now been updated and includes a short paragraph that summarizes BigML’s response. In the meanwhile, to set the record straight, we are publishing the highlights of our response below for the benefit of the BigML community as we take any security and privacy related issue very seriously:

WIRED Author:

“I’d really appreciate if anyone at BigML can comment on the security or privacy threat this [ways of “stealing” machine learning models from black-box platforms] might represent to BigML’s machine learning platform, given that it seems certain models can be reverse engineered via a series of inputs and outputs by anyone who accesses them on BigML’s public platform.”

BigML:

• Models built using BigML’s platform are only accessible to their owners who already have complete and white-box access to them, so this research does not expose or represent any security or privacy threat to BigML’s platform at all.

• BigML’s users can access the underlying structure of their own models. This means that they can not only introspect their models using BigML’s visualizations but also fully download their model and use them in their own applications as they wish. BigML does not charge users for making predictions with their own models. So there is no need to reverse-engineer them as might be the case when you use Azure ML or Amazon ML. These services charge the owners of the models for making predictions with their own models.

• BigML allows users to share models with other BigML users either in a white-box mode or in a black-box mode. In the latter case, if a user wanted to monetize her model by charging for predictions to another user, the user being charged might try to reproduce such model and avoid to continue paying for predictions. There is currently no BigML user charging for predictions. Again, this research does not expose or represent any security or privacy threat to BigML’s platform at all.

On Obviousness

• Anyone versed in Machine Learning can see that many of the results of the publication are obvious. Any machine-learned model that is made available becomes a “data labeling API”, so it can, unsurprisingly, be used to label enough data to reproduce the model to some degree.  These researchers are focused on elaborate attacks that learn decision trees exactly (which does seem interesting academically), but far simpler algorithms will and always have been able to generate a lossy reproduction of a machine-learned model.  In fact, this is the exact trick that Machine Learning itself pulls on human experts: The human provides labeled data and the machine learns a model that replicates (to the degree made possible by the data) the modeling process of the human labeler.  It is therefore utterly unremarkable that this also works if it is a machine providing the labeled data.

• As an instructive example, imagine you want to reverse-engineer the pricing strategy of an airline. It is unimportant how the model used by the airline was created; using a Machine Learning API,  an open source ML package, or a collection of rules provided by experts.  If one looks up at the price for enough flights, days, and lead times, one will soon have enough data to replicate the pricing strategy.

On Charging for Predictions:

• BigML does not charge customers for predictions with their own models.  We think that this research might be relevant for services like Amazon ML or Azure ML, since they are charging users for predictions. Users of those services could try to reproduce the model or simply cache model responses to avoid being charged. Selling predictions is not a long-term money-making proposition unless you keep improving the classifier so that your predictions keep improving too. In other words, this shows how charging for predictions is a poor business strategy, and how BigML’s business model (charging for overall computational capacity to build many models for many different predictive use cases in an organization) is therefore more reasonable.

• In BigML, this research would only be significant in the scenario where a BigML user publicly offers their trained model for paid predictions but wants to keep it secret.  We do not currently  have any customers exposing black-box models (except the ones created by these researchers).  But if that were the case, a user can guarantee that reconstructing the model will have a prohibitive cost by setting a higher price for each prediction.

On Applicability:

• Some models are easier to reproduce while others are considerably harder. This research shows that their most elaborate method is only useful for single trees.  When the confidence level of a prediction is provided, the difficulty of the learning problem decreases.  However, when the models are more complex (such as Random Decision Forests) the process to replicate a model is not amenable to many of the techniques described in the paper, so models can only be approximated via the method we describe above.

• If we wanted to offer a monetized black-box prediction platform in a serious way (and we are sure that we do not), we would encourage users to use complex models rather than individual trees. We can easily detect and throttle the kind of systematic search across the input space that would be required to efficiently reconstruct a complex classifier.

On Machine Learning APIs:

• One thing is very clear to us though, Machine Learning APIs help researchers in many areas to start experimenting with machine-learned models in a way that other tools have never allowed. Mind you this is coming from a team with backgrounds in ML research. In fact, the research these folks carried out would be far more difficult to pursue using old-fashioned Machine Learning tools such as R or SAS that are tedious and complicated.

Finally, some comments in defense of other Machine Learning services that are potentially subject to this issue.

On Legality: 

• We assume that to a researcher in security trying to find things on which to publish a paper, everything looks like a “security issue”. Putting things in the same category data privacy or identity theft issues makes them sound dangerous and urgent. However, the vast majority of the paper describes security issues closer in nature to defeating copy protection in commercial software, or developing software that functions exactly as an existing commercial product.  While this sort of security breach is certainly unfortunate and something to be minimized, it is important to distinguish things that are often dangerous to the public at large from those that, in the vast majority of cases, do not pose as big a threat.

• Software theft and reverse engineering isn’t new or unique to Machine Learning as a Service, and society typically relies on the legal system to provide incentives against such behavior.  Said another way, even if stealing software were easy, there is still an important disincentive to do so in that it violates intellectual property law.  To our knowledge, there has been no major IP litigation to date involving compromise of machine-learned models, but as machine learning grows in popularity the applicable laws will almost certainly mature and offer some recourse against the exploits that the authors describe.