Recommender Systems Across Industries

Recommendation engines are everywhere. If you have ever been on the Internet you have already used them.

Av Nima Darabi

Whenever you see “People who bought this bought that too” on an online shopping service or “People you may know” on a social network, there is a recommendation engine behind the scene that comes up with those suggestions from an enormous pool of products or people. Much of the ads, relevant news and videos that are shown to you are the outcome of those engines. Today, more and more industries consider taking advantage of the power of recommendation engines in order to make their customers more satisfied and to take their ecommerce effort to the next level.

What is a recommendation engine?

Recommendation engines (or recommender systems) are digital services that seek to predict the “rating” or “preference” that a user would give to an item [i].  They are meant to make it easy for users to find the relevant contents or products and thus to improve the customer satisfaction and increase the retention. The inventory subject to the recommendation can be digitally available contents such as music, movies, TV programs, or news. They can also be physical (or so-called brick and mortar) products such as grocery or other consumer products, or even job positions or dating profiles. Basically any list of inventories big enough to make the customers confused in a big pool of choices can benefit from a recommender system.

How do recommendation engines work?

A recommendation engine is a feature that filters items by predicting how users may rate them. It solves the problem of connecting existing users with the correct items in a big enough inventory of products or contents. There are two different approaches in the making of recommendation engines: Collaborative Filtering and Content-Based Filtering. Collaborative filtering systems merely use the abstract interaction between the existing user and contents without looking into what contents consist of. On the other hand, Content-based filtering systems look deeper into the nature of contents to find out which of them may be more relevant to a specific user. There exist also hybrid methods that can benefit from both approaches.

Advanced recommendation engines also take advantage of contextual data. They pay attention to seasonal events, weather, location, and sometimes to the political events and fashionable trends. Such algorithms rely on the time of the historic purchases or viewings so that they can recommend something at the right time and location that not only suits the user’s historic taste but also fits his contextual surroundings.

Who needs a recommendation engine?

If you don’t have an established user base and a massive inventory, a recommendation engine does not truly solve a problem for you [ii]. But what is massive? If we want to force a rule of thumb, a big enough inventory means more than ten thousands. Are you a media service with more than ten thousands programs and you have problem how to pick the contents to deliver to each specific user? Are you a job market service with more dozens of thousands of job positions and may be even more number of CVs? Then a recommendation engine can start to pay off for your business.

It is a common misconception that recommenders necessarily need “Big Data”. Recommender algorithms are mostly implemented and developed on big data technologies such as Hadoop, but this is because they primarily came from the information technology and historically needed to scale quite fast. If your business or the size of your offerings is not growing exponentially and if you can already handle your data with a traditional data warehouse, you already set up your own recommender on top of that and start seeing benefits from it.

That being said, recommenders are not the killer feature of your minimum viable product that will bring the users to you in numbers [iii]. You need some existing users and an established service with big enough list of offerings. Given that, it can be an important factor of growth for an already existing popular service or even an essential capability in order to maintain the popularity depending on the competition with other players.

Why being proactive?

Better recommendation of the web pages was the reason that Google’s search algorithm outperformed Altavista at the dawn of web search engines [v]. At the time the number of web pages were extremely smaller than what they are now [vi] and a recommendation engine could be seen as a luxurious feature. Some of us will still remember those days when the pioneer tech companies still tried to establish themselves as the “front page” of the Internet by bringing up the content that matters the most to all. Google saw the future better than others and invested in the recommendation engine to boost the magical power of its search engine and we all know how it became the core component of growth for a small start-up that is now one of the biggest firms on the planet.

The same trends of using data are coming from information technolgy to other industries: health, retail, media and job market. It is time today and not tomorrow, for many firms out there to start thinking about incorporating a more efficient listing of their products or contents in order to stay relevant. Otherwise, it is conceivable for many big players to lose the ground to their competitors even if they are well-established services.

Why recommendation engine is not a product?

It is charming, beautiful and ideal to buy a plug and play recommendation engine from a vendor for your business. You buy one out of the box and start using it and users come to your service in numerous numbers. It is fantastic, except for that it is – sadly – not possible. Recommenders are not digital products but are solutions that should be developed for a specific business. Depending on the size of the business they can be made by external consultants or partners or local data scientists. Building a recommendation engine needs field knowledge, feasibility research, data quality assessment, choice of the algorithm [iv] and the estimate of the efficient technology.

What are the risks of using recommendation engines?

Before you risk rolling out the beta version of your recommendation engine you should look in to the risks it may have for your business:

First and foremost, recommenders can be risky in terms of bringing contents with low quality. Popular contents are safe options to introduce and normally have an assured quality as opposed to unknown contents. Companies must be catious and take all measures to make sure that the recommender delivers content that is not only relavant but also of good quality.

Recommendation engines may also fail if the market that they are serving is not “long-tail”. A decade ago it was assumed that by the help of the Internet a digitized future will come to all businesses where more money will be made from nieche offerings than from bluckbuster hits. This future came to happen in some industries but failed to be seen in some others. Despite all the success stories, the long tail turned out to be a risky entrepreneurial assumption for many start-ups. The theory reportedly caused launching of more than a thousand startups [vii], but failed to deliver the promise of the long-tail profitability. Before starging to build your own recommender you need to look into if the market has long-tail profitability or the users are already happy by consuming a limited list of the nieche products.

Recommenders, if implemented wrong, can also decrease the diversity of digital consumption while the opposite is intended by them. While recommenders are built to help consumers discover new products and thus increase sales diversity, a blind recommendation algorithms based on sales and ratings (mostly collaborative filters) fail to expose products with limited historical data and will reinforce the popularity of hit products leading to a rich-get-richer effect [viii]. It is crucial to tweak the parameters of your recommender correctly so such artifacts do not happen.

These are the reasons you need experienced data scientist to build the recommendation engine for you.

Where is value?

Recommender systems increase the user satisfaction by delivering a personalized user experience. Making choices is a difficult task for human and users do not want to waste their precious time getting lost in a list of your inventory, no matter how colorful and appealing that list is. If another service gives them what they want in a shorter time they will switch. Recommenders are shown to have a crucial role on improving the retention and avoiding churn.

Recommendation engines do not only save time for your customers. They save time for your staff, too. If your staff spend a lot of time to sort out new delivered commodities, to consult customers on what suits them best among a long list of products, or to put editorial effort on new produced contents, you may want to consider to save lots of money that you have been spending on expensive human resource by delegating such tasks to the machines. Use your staff for more creative jobs, if the machines can do the labor-intesive tasks better, faster, and cheaper.


[i] https://en.wikipedia.org/wiki/Recommender_system

[ii] http://www.datacommunitydc.org/blog/2013/05/recommendation-engines-why-you-shouldnt-build-one

[iii] Sarwar, Badrul, et al. “Item-based collaborative filtering recommendation algorithms.” Proceedings of the 10th international conference on World Wide Web. ACM, 2001.

[iv] https://www.toptal.com/algorithms/predicting-likes-inside-a-simple-recommendation-engine

[v] http://www.forbes.com/sites/lutzfinger/2014/09/02/recommendation-engines-the-reason-why-we-love-big-data/#1a41c38c218e

[vi] http://www.internetlivestats.com/total-number-of-websites/

[vii] http://anand.typepad.com/datawocky/2008/07/the-real-long-tail-why-both-chris-anderson-and-anita-elberse-are-wrong.html

[viii] Fleder, Daniel, and Kartik Hosanagar. “Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity.” Management science 55.5 (2009): 697-712.

[ix] https://hbr.org/2006/06/profiting-from-the-long-tail