Grading our 2020 Oscars Machine Learning Predictions

The 2020 Oscars were presented on Sunday evening in Hollywood, California in what turned out a night of some historic firsts. The second in a row hostless edition of the award ceremony was reportedly watched live by an audience of 23.6 million, which tracked lower than last year’s total of 29.6 million viewers. While all award show broadcasts these days are reporting declines in viewership, the Academy Awards still rule the roost by a healthy margin with continued reverberations in social media days after the event.

So how did we do with our predictions this year?

The short answer is we got 5 out of 8 predictions right. It’s not so shabby when considering that if we hired chimpanzees to randomly throw darts on boards with the names of the nominees on them, it would take our furry friends 3,125 tries to correctly guess the five awards we got right. (NOTE: For the ubergeek in you, a score of perfect 8 out 8 would take the chimp squad 703,125 tries on average.) As for human experts, we’re not aware of any high profile movie critics that got every award right either.

So it’s safe to say our models were definitely picking up some legitimate signals in what is essentially a fairly small sample (1288 records) of past award data. To be more specific, the number of positive examples (there are just 20 award winners in a given category in the last 20 years) in each held out test set is pretty tiny. This opens the door for potential overfitting. Contrast that against millions of data points collected from sensors on a piece of machinery that can be modeled to make much more robust predictions.

As it turned out, BigML was not alone in missing the mark on some of the high-profile award predictions this year. Some were counting on 1917, fresh off its Golden Globes win, to take home the big one while others applied the “wisdom of the crowd” approach to arrive at the same conclusion. It seems those approaches fell short not only in terms of prediction accuracy (which can happen with a smaller dataset) but also because they neither described a repeatable, end-to-end process nor shared a public dataset for interested parties to utilize — more work needed.

As we analyze our misses as seen in the table above, we swung but missed the Best Picture, Best Director and Best Adapted Screenplay awards but did very well with Best Original Screenplay as well as all four of the Actor and Actress categories. Without a doubt, South Korea’s Parasite bucking the trend and becoming the first foreign-language film to win the big awards, Best Picture and Best Director (Bong Joon Ho), had a lot to do with our respectable but less than perfect results.

In general, Machine Learning models such as the classification models we built for this project rely on the assumption that the newly presented data will not drastically deviate from historical datasets they were trained on. While this helps produce robust results that are statistically significant most of the time, it may also miss important points of deflection from the norm like what we witnessed on Sunday.

To be fair, our model had Parasite sport the second-best score of 82/100 for the Best Picture award behind only Once Upon a Time…in Hollywood. And we did mention in our post that a Parasite win could not be ruled out. Similarly, Bong Joon Ho was given a score of 55/100 for the Best Director award, which can be interpreted as a win probability of 55% which is significant enough by itself.

If we put Parasite under the “microscope” (sorry, couldn’t help it :), we see that the producers and the director of the original crime drama tirelessly campaigned to generate grassroots interest in many International film festivals, which helped carry their momentum into the box office to the tune of $35M+ in the U.S. and $165M worldwide. That’s pretty impressive for an Asian production with a modest budget of $11M.

In the past decades, movie studios like U.S.-based Lionsgate had perfected the game of getting smaller budget flicks to punch above their weight as was the case for the 2006 surprise winner, Crash. On the other hand, last year, we witnessed Alejandro González Iñárritu’s Mexican production Roma (another foreign-language release with subtitles) counted among the favorites. However, Roma grossed barely above $1M despite great reviews by art-house critics. So, in the 2020 edition of the Oscars, a perfect storm of a foreign language movie not only well-liked by the critics but also by worldwide and U.S. audiences may have been brewing in front of our eyes.

It’s perhaps very fitting that Parasite‘s big wins coincided with the name change of what was called the “Best Foreign Language Film” award to “Best International Feature Film” as the welcome change shows the Academy is adapting to the more inclusive point of view considering world cinema not “foreign” or “other,” but part of the broader movie landscape. This gives us the motivation to expand our dataset for next year to cover more international festivals such as Cannes and the Toronto International Film Festival (TIFF). This may help capture a more complete sentiment of worldwide movie fans.

The last award we predicted incorrectly was the Adapted Screenplay. The winner, Jojo Rabbit, had a very low score of 1 out of 100 yet managed to beat The Irishman that had 67. We’ll chalk that up to the “big miss” bucket and admittedly have to do some deeper digging down to see if there was an angle we overlooked or even an underlying data issue.

With this wrap, as the pioneers of ML-as-a-Service here at BigML, we welcome you to build your own models that hopefully will beat what we’ve shared here so we can learn a few tricks from you too. The movies 2000-2019 dataset is public and calling for your time and creativity. In this exercise, you will have the benefit of knowing the winners in advance, but it may still make great practice for the 2021 Oscars. Let us know how your results come out on Twitter @bigmlcom or shoot us a note at feedback@bigml.com anytime!