Recommendation Engine Article Summary
Article is about recommendations and uses Netflix Prize as the central theme. Netflix announced a prize of $1 million for anyone that beat their Cinematch (movie recommendation) algorithm by more than 10%. The author mentions various people competing for the prize and explain the problems of developing a good recommendation system from different angles.
Good recommendations systems are hard to build. After a point, doesn’t matter what you do, it improves negligibly. Netflix CEO Reed Hastings came up with the idea of a competition with a huge prize to entice them to come up with better recommendation systems. Hundreds of teams started working on improving it part time, spending on average 20 hours per week after office. The top 10 teams used most complex systems, one of which is Singular Value Decomposition which Netflix earlier didn’t use for recommendations.
The article also debates in part about the ability of a computer to understand human taste and generate good recommendations. As there is a lot of data about people’s preferences, it takes those into account to figure out how much would you an individual like a movie. These systems perform fairly well for movie that generate similar reactions by the users. But there are movies like “Napolean Dynamites” that generate extreme reactions by people. This is where recommendation systems fail. As teams apply more complex algorithms, they lose the ability to identify why it works. Movies without any apparent similarity are clustered in together and intuitively, it’s difficult to understand why. Maybe algorithm is able to recognize our tastes that even we don’t understand ourselves. Then there is problem of inherent user biases. Some users are always punitive in giving ratings, so teams inflate ratings from these users, on the other hand other users are very liberal and teams deflate ratings from them. It also depends on the user’s mood, if they are happy, they’ll be more forgiving but if they’re in a bad mood, they’ll be stringent.
It is difficult to build recommendations systems that work.
