Recommender system

A recommender system (RecSys), or a recommendation system (sometimes replacing system with terms such as platform, engine, or algorithm), sometimes only called "the algorithm" or "algorithm"<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> is a subclass of information filtering system that provides suggestions for items that are most pertinent to a particular user.<ref name="handbook" /><ref>Template:Cite magazine</ref><ref>Template:Cite journal</ref> Recommender systems are particularly useful when an individual needs to choose an item from a potentially overwhelming number of items that a service may offer.<ref name="handbook" /><ref name="ResnickVarian"/> Modern recommendation systems such as those used on large social media sites make extensive use of AI, machine learning and related techniques to learn the behavior and preferences of each user and categorize content to tailor their feed individually.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref>

Typically, the suggestions refer to various decision-making processes, such as what product to purchase, what music to listen to, or what online news to read.<ref name="handbook" /> Recommender systems are used in a variety of areas, with commonly recognised examples taking the form of playlist generators for video and music services, product recommenders for online stores, or content recommenders for social media platforms and open web content recommenders.<ref name="twitterwtf2">Template:Cite conference</ref><ref>Template:Cite journal</ref> These systems can operate using a single type of input, like music, or multiple inputs within and across platforms like news, books and search queries. There are also popular recommender systems for specific topics like restaurants and online dating. Recommender systems have also been developed to explore research articles and experts,<ref name="expertseer">H. Chen, A. G. Ororbia II, C. L. Giles ExpertSeer: a Keyphrase Based Expert Recommender for Digital Libraries, in arXiv preprint 2015</ref> collaborators,<ref name="collabseer">Template:Cite conference</ref> and financial services.<ref name="financialrec">Template:Cite conference</ref>

A content discovery platform is an implemented software recommendation platform which uses recommender system tools. It utilizes user metadata in order to discover and recommend appropriate content, whilst reducing ongoing maintenance and development costs. A content discovery platform delivers personalized content to websites, mobile devices and set-top boxes. A large range of content discovery platforms currently exist for various forms of content ranging from news articles and academic journal articles<ref name="nature1">Template:Cite journal</ref> to television.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> As operators compete to be the gateway to home entertainment, personalized television is a key service differentiator. Academic content discovery has recently become another area of interest, with several companies being established to help academic researchers keep up to date with relevant academic content and serendipitously discover new content.<ref name="nature1"/>

Overview

Recommender systems usually make use of either or both collaborative filtering and content-based filtering, as well as other systems such as knowledge-based systems. Collaborative filtering approaches build a model from a user's past behavior (items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items (or ratings for items) that the user may have an interest in.<ref name="Recommender2010">Template:Cite encyclopedia</ref> Content-based filtering approaches utilize a series of discrete, pre-tagged characteristics of an item in order to recommend additional items with similar properties.<ref name="mooney99">Template:Cite conference</ref>

The differences between collaborative and content-based filtering can be demonstrated by comparing two early music recommender systems, Last.fm and Pandora Radio.

• Last.fm creates a "station" of recommended songs by observing what bands and individual tracks the user has listened to on a regular basis and comparing those against the listening behavior of other users. Last.fm will play tracks that do not appear in the user's library, but are often played by other users with similar interests. As this approach leverages the behavior of users, it is an example of a collaborative filtering technique.<ref>Template:Cite journal</ref>
• Pandora uses the properties of a song or artist (a subset of the 400 attributes provided by the Music Genome Project) to seed a "station" that plays music with similar properties. User feedback is used to refine the station's results, deemphasizing certain attributes when a user "dislikes" a particular song and emphasizing other attributes when a user "likes" a song. This is an example of a content-based approach.

Each type of system has its strengths and weaknesses. In the above example, Last.fm requires a large amount of information about a user to make accurate recommendations. This is an example of the cold start problem, and is common in collaborative filtering systems.<ref name=":3">Template:Cite journal</ref><ref name="rubens2016">Template:Cite book</ref><ref>Template:Cite journal</ref><ref name="elahi2016"> Template:Cite journal </ref><ref name="schein02"> Template:Cite conference </ref><ref name="bi2017">Template:Cite journal</ref> Whereas Pandora needs very little information to start, it is far more limited in scope (for example, it can only make recommendations that are similar to the original seed).

Recommender systems are a useful alternative to search algorithms since they help users discover items they might not have found otherwise. Of note, recommender systems are often implemented using search engines indexing non-traditional data.

Recommender systems have been the focus of several granted patents,<ref>Stack, Charles. "System and method for providing recommendation of goods and services based on recorded purchasing history." U.S. Patent 7,222,085, issued May 22, 2007.</ref><ref>Herz, Frederick SM. "Customized electronic newspapers and advertisements." U.S. Patent 7,483,871, issued January 27, 2009. </ref><ref> Herz, Frederick, Lyle Ungar, Jian Zhang, and David Wachob. "System and method for providing access to data using customer profiles." U.S. Patent 8,056,100, issued November 8, 2011. </ref><ref> Harbick, Andrew V., Ryan J. Snodgrass, and Joel R. Spiegel. "Playlist-based detection of similar digital works and work creators." U.S. Patent 8,468,046, issued June 18, 2013. </ref><ref> Linden, Gregory D., Brent Russell Smith, and Nida K. Zada. "Automated detection and exposure of behavior-based relationships between browsable items." U.S. Patent 9,070,156, issued June 30, 2015.</ref> and there are more than 50 software libraries<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> that support the development of recommender systems including LensKit,<ref>Template:Cite book</ref><ref>Template:Cite book</ref> RecBole,<ref>Template:Cite book</ref> ReChorus<ref>Template:Cite book</ref> and RecPack.<ref>Template:Cite book</ref>

History

Elaine Rich created the first recommender system in 1979, called Grundy.<ref>BEEL, Joeran, et al. Paper recommender systems: a literature survey. International Journal on Digital Libraries, 2016, 17. Jg., Nr. 4, S. 305–338.</ref><ref>RICH, Elaine. User modeling via stereotypes. Cognitive science, 1979, 3. Jg., Nr. 4, S. 329–354.</ref> She looked for a way to recommend users books they might like. Her idea was to create a system that asks users specific questions and classifies them into classes of preferences, or "stereotypes", depending on their answers. Depending on users' stereotype membership, they would then get recommendations for books they might like.

Another early recommender system, called a "digital bookshelf", was described in a 1990 technical report by Jussi Karlgren at Columbia University, <ref>Karlgren, Jussi. "An Algebra for Recommendations.Template:Webarchive. Syslab Working Paper 179 (1990). " </ref> and implemented at scale and worked through in technical reports and publications from 1994 onwards by Jussi Karlgren, then at SICS,<ref>Karlgren, Jussi. "Newsgroup Clustering Based On User Behavior-A Recommendation Algebra Template:Webarchive." SICS Research Report (1994).</ref><ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> and research groups led by Pattie Maes at MIT,<ref> Shardanand, Upendra, and Pattie Maes. "Social information filtering: algorithms for automating "word of mouth"." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 210–217. ACM Press/Addison-Wesley Publishing Co., 1995. </ref> Will Hill at Bellcore,<ref>Hill, Will, Larry Stead, Mark Rosenstein, and George Furnas. "Recommending and evaluating choices in a virtual community of use Template:Webarchive." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 194–201. ACM Press/Addison-Wesley Publishing Co., 1995.</ref> and Paul Resnick, also at MIT,<ref>Resnick, Paul, Neophytos Iacovou, Mitesh Suchak, Peter Bergström, and John Riedl. "GroupLens: an open architecture for collaborative filtering of netnews." In Proceedings of the 1994 ACM conference on Computer supported cooperative work, pp. 175–186. ACM, 1994.</ref><ref name="ResnickVarian">Resnick, Paul, and Hal R. Varian. "Recommender systems." Communications of the ACM 40, no. 3 (1997): 56–58.</ref> whose work with GroupLens was awarded the 2010 ACM Software Systems Award.

Montaner provided the first overview of recommender systems from an intelligent agent perspective.<ref>Template:Cite journal.</ref> Adomavicius provided a new, alternate overview of recommender systems.<ref name="Toward the Next Generation of Recommender Systems">Template:Cite journal.</ref> Herlocker provides an additional overview of evaluation techniques for recommender systems,<ref>Template:Cite journal.</ref> and Beel et al. discussed the problems of offline evaluations.<ref name=":0">Template:Cite book</ref> Beel et al. have also provided literature surveys on available research paper recommender systems and existing challenges.<ref>Template:Cite book</ref><ref>Template:Cite journal</ref>

Approaches

Collaborative filtering

{{#invoke:Labelled list hatnote|labelledList|Main article|Main articles|Main page|Main pages}}

File:Collaborative filtering.gif

An example of collaborative filtering based on a rating system

One approach to the design of recommender systems that has wide use is collaborative filtering.<ref name="Breese98">Template:Cite conference </ref> Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past. The system generates recommendations using only information about rating profiles for different users or items. By locating peer users/items with a rating history similar to the current user or item, they generate recommendations using this neighborhood. Collaborative filtering methods are classified as memory-based and model-based. A well-known example of memory-based approaches is the user-based algorithm,<ref>Template:Cite report</ref> while that of model-based approaches is matrix factorization (recommender systems).<ref>Template:Cite journal</ref>

A key advantage of the collaborative filtering approach is that it does not rely on machine analyzable content and therefore it is capable of accurately recommending complex items such as movies without requiring an "understanding" of the item itself. Many algorithms have been used in measuring user similarity or item similarity in recommender systems. For example, the k-nearest neighbor (k-NN) approach<ref>{{#invoke:citation/CS1|citation |CitationClass=web }},</ref> and the Pearson Correlation as first implemented by Allen.<ref>Template:Citation</ref>

When building a model from a user's behavior, a distinction is often made between explicit and implicit forms of data collection.

Examples of explicit data collection include the following:

• Asking a user to rate an item on a sliding scale.
• Asking a user to search.
• Asking a user to rank a collection of items from favorite to least favorite.
• Presenting two items to a user and asking him/her to choose the better one of them.
• Asking a user to create a list of items that he/she likes (see Rocchio classification or other similar techniques).

Examples of implicit data collection include the following:

• Observing the items that a user views in an online store.
• Analyzing item/user viewing times.<ref>Template:Cite conference.</ref>
• Keeping a record of the items that a user purchases online.
• Obtaining a list of items that a user has listened to or watched on his/her computer.
• Analyzing the user's social network and discovering similar likes and dislikes.

Collaborative filtering approaches often suffer from three problems: cold start, scalability, and sparsity.<ref name="Lee2007">Sanghack Lee and Jihoon Yang and Sung-Yong Park, Discovery of Hidden Similarity on Collaborative Filtering to Overcome Sparsity Problem, Discovery Science, 2007.</ref>

• Cold start: For a new user or item, there is not enough data to make accurate recommendations. Note: one commonly implemented solution to this problem is the multi-armed bandit algorithm.<ref>Template:Cite book</ref><ref name=":3" /><ref name="rubens2016"/><ref name="elahi2016"/><ref name="bi2017"/>
• Scalability: There are millions of users and products in many of the environments in which these systems make recommendations. Thus, a large amount of computation power is often necessary to calculate recommendations.
• Sparsity: The number of items sold on major e-commerce sites is extremely large. The most active users will only have rated a small subset of the overall database. Thus, even the most popular items have very few ratings.

One of the most famous examples of collaborative filtering is item-to-item collaborative filtering (people who buy x also buy y), an algorithm popularized by Amazon.com's recommender system.<ref name="patft.uspto.gov">Collaborative Recommendations Using Item-to-Item Similarity Mappings Template:Webarchive</ref>

Many social networks originally used collaborative filtering to recommend new friends, groups, and other social connections by examining the network of connections between a user and their friends.<ref name="handbook">Template:Cite book</ref> Collaborative filtering is still used as part of hybrid systems.

Content-based filtering

Another common approach when designing recommender systems is content-based filtering. Content-based filtering methods are based on a description of the item and a profile of the user's preferences.<ref name="Aggarwal16Book">Template:Cite book</ref><ref>Template:Cite book</ref> These methods are best suited to situations where there is known data on an item (name, location, description, etc.), but not on the user. Content-based recommenders treat recommendation as a user-specific classification problem and learn a classifier for the user's likes and dislikes based on an item's features.

In this system, keywords are used to describe the items, and a user profile is built to indicate the type of item this user likes. In other words, these algorithms try to recommend items similar to those that a user liked in the past or is examining in the present. It does not rely on a user sign-in mechanism to generate this often temporary profile. In particular, various candidate items are compared with items previously rated by the user, and the best-matching items are recommended. This approach has its roots in information retrieval and information filtering research.

To create a user profile, the system mostly focuses on two types of information:

1. A model of the user's preference.
2. A history of the user's interaction with the recommender system.

Basically, these methods use an item profile (i.e., a set of discrete attributes and features) characterizing the item within the system. To abstract the features of the items in the system, an item presentation algorithm is applied. A widely used algorithm is the tf–idf representation (also called vector space representation).<ref>Template:Cite journal</ref> The system creates a content-based profile of users based on a weighted vector of item features. The weights denote the importance of each feature to the user and can be computed from individually rated content vectors using a variety of techniques. Simple approaches use the average values of the rated item vector while other sophisticated methods use machine learning techniques such as Bayesian Classifiers, cluster analysis, decision trees, and artificial neural networks in order to estimate the probability that the user is going to like the item.<ref>Template:Cite news</ref>

A key issue with content-based filtering is whether the system can learn user preferences from users' actions regarding one content source and use them across other content types. When the system is limited to recommending content of the same type as the user is already using, the value from the recommendation system is significantly less than when other content types from other services can be recommended. For example, recommending news articles based on news browsing is useful. Still, it would be much more useful when music, videos, products, discussions, etc., from different services, can be recommended based on news browsing. To overcome this, most content-based recommender systems now use some form of the hybrid system.

Content-based recommender systems can also include opinion-based recommender systems. In some cases, users are allowed to leave text reviews or feedback on the items. These user-generated texts are implicit data for the recommender system because they are potentially rich resources of both feature/aspects of the item and users' evaluation/sentiment to the item. Features extracted from the user-generated reviews are improved metadata of items, because as they also reflect aspects of the item like metadata, extracted features are widely concerned by the users. Sentiments extracted from the reviews can be seen as users' rating scores on the corresponding features. Popular approaches of opinion-based recommender system utilize various techniques including text mining, information retrieval, sentiment analysis (see also Multimodal sentiment analysis) and deep learning.<ref>X.Y. Feng, H. Zhang, Y.J. Ren, P.H. Shang, Y. Zhu, Y.C. Liang, R.C. Guan, D. Xu, (2019), "The Deep Learning–Based Recommender System "Pubmender" for Choosing a Biomedical Publication Venue: Development and Validation Study", Journal of Medical Internet Research, 21 (5): e12957</ref>

Hybrid recommendations approaches

Most recommender systems now use a hybrid approach, combining collaborative filtering, content-based filtering, and other approaches. There is no reason why several different techniques of the same type could not be hybridized. Hybrid approaches can be implemented in several ways: by making content-based and collaborative-based predictions separately and then combining them; by adding content-based capabilities to a collaborative-based approach (and vice versa); or by unifying the approaches into one model.<ref name="Toward the Next Generation of Recommender Systems" /> Several studies that empirically compared the performance of the hybrid with the pure collaborative and content-based methods and demonstrated that the hybrid methods can provide more accurate recommendations than pure approaches. These methods can also be used to overcome some of the common problems in recommender systems such as cold start and the sparsity problem, as well as the knowledge engineering bottleneck in knowledge-based approaches.<ref>Rinke Hoekstra, The Knowledge Reengineering Bottleneck, Semantic Web – Interoperability, Usability, Applicability 1 (2010) 1, IOS Press</ref>

Netflix is a good example of the use of hybrid recommender systems.<ref>Template:Cite journal</ref> The website makes recommendations by comparing the watching and searching habits of similar users (i.e., collaborative filtering) as well as by offering movies that share characteristics with films that a user has rated highly (content-based filtering).

Some hybridization techniques include:

• Weighted: Combining the score of different recommendation components numerically.
• Switching: Choosing among recommendation components and applying the selected one.
• Mixed: Recommendations from different recommenders are presented together to give the recommendation.
• Cascade: Recommenders are given strict priority, with the lower priority ones breaking ties in the scoring of the higher ones.
• Meta-level: One recommendation technique is applied and produces some sort of model, which is then the input used by the next technique.<ref name=hybrids>Robin Burke, Hybrid Web Recommender Systems Template:Webarchive, pp. 377-408, The Adaptive Web, Peter Brusilovsky, Alfred Kobsa, Wolfgang Nejdl (Ed.), Lecture Notes in Computer Science, Springer-Verlag, Berlin, Germany, Lecture Notes in Computer Science, Vol. 4321, May 2007, 978-3-540-72078-2.</ref>

Technologies

Session-based recommender systems

These recommender systems use the interactions of a user within a session<ref name="a">Template:Cite arXiv</ref> to generate recommendations. Session-based recommender systems are used at YouTube<ref name="yt">Template:Cite arXiv</ref> and Amazon.<ref name="amzn">Template:Cite book</ref> These are particularly useful when history (such as past clicks, purchases) of a user is not available or not relevant in the current user session. Domains where session-based recommendations are particularly relevant include video, e-commerce, travel, music and more. Most instances of session-based recommender systems rely on the sequence of recent interactions within a session without requiring any additional details (historical, demographic) of the user. Techniques for session-based recommendations are mainly based on generative sequential models such as recurrent neural networks,<ref name="a" /><ref>Template:Cite book</ref> transformers,<ref>Template:Cite arXiv</ref> and other deep-learning-based approaches.<ref>Template:Cite book</ref><ref>Template:Cite book</ref>

Reinforcement learning for recommender systems

The recommendation problem can be seen as a special instance of a reinforcement learning problem whereby the user is the environment upon which the agent, the recommendation system acts upon in order to receive a reward, for instance, a click or engagement by the user.<ref name="yt" /><ref name="srl">Template:Cite arXiv</ref><ref name="sQ">Template:Cite journal</ref> One aspect of reinforcement learning that is of particular use in the area of recommender systems is the fact that the models or policies can be learned by providing a reward to the recommendation agent. This is in contrast to traditional learning techniques which rely on supervised learning approaches that are less flexible, reinforcement learning recommendation techniques allow to potentially train models that can be optimized directly on metrics of engagement, and user interest.<ref name="jd">Template:Cite book</ref>

Multi-criteria recommender systems

Multi-criteria recommender systems (MCRS) can be defined as recommender systems that incorporate preference information upon multiple criteria. Instead of developing recommendation techniques based on a single criterion value, the overall preference of user u for the item i, these systems try to predict a rating for unexplored items of u by exploiting preference information on multiple criteria that affect this overall preference value. Several researchers approach MCRS as a multi-criteria decision making (MCDM) problem, and apply MCDM methods and techniques to implement MCRS systems.<ref>Template:Cite journal</ref> See this chapter<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> for an extended introduction.

Risk-aware recommender systems

The majority of existing approaches to recommender systems focus on recommending the most relevant content to users using contextual information, yet do not take into account the risk of disturbing the user with unwanted notifications. It is important to consider the risk of upsetting the user by pushing recommendations in certain circumstances, for instance, during a professional meeting, early morning, or late at night. Therefore, the performance of the recommender system depends in part on the degree to which it has incorporated the risk into the recommendation process. One option to manage this issue is DRARS, a system which models the context-aware recommendation as a bandit problem. This system combines a content-based technique and a contextual bandit algorithm.<ref name="Bouneffouf2013">Template:Cite thesis</ref>

Mobile recommender systems

Template:Further Mobile recommender systems make use of internet-accessing smartphones to offer personalized, context-sensitive recommendations. This is a particularly difficult area of research as mobile data is more complex than data that recommender systems often have to deal with. It is heterogeneous, noisy, requires spatial and temporal auto-correlation, and has validation and generality problems.<ref name="taxirecommender">Template:Cite conference </ref>

There are three factors that could affect the mobile recommender systems and the accuracy of prediction results: the context, the recommendation method and privacy.<ref>Template:Cite journal</ref> Additionally, mobile recommender systems suffer from a transplantation problem – recommendations may not apply in all regions (for instance, it would be unwise to recommend a recipe in an area where all of the ingredients may not be available).

One example of a mobile recommender system are the approaches taken by companies such as Uber and Lyft to generate driving routes for taxi drivers in a city.<ref name="taxirecommender"/> This system uses GPS data of the routes that taxi drivers take while working, which includes location (latitude and longitude), time stamps, and operational status (with or without passengers). It uses this data to recommend a list of pickup points along a route, with the goal of optimizing occupancy times and profits.

Generative recommenders

Generative recommenders (GR) represent an approach that transforms recommendation tasks into sequential transduction problems, where user actions are treated like tokens in a generative modeling framework. In one method, known as HSTU (Hierarchical Sequential Transduction Units),<ref>Template:Cite arXiv</ref> high-cardinality, non-stationary, and streaming datasets are efficiently processed as sequences, enabling the model to learn from trillions of parameters and to handle user action histories orders of magnitude longer than before. By turning all of the system’s varied data into a single stream of tokens and using a custom self-attention approach instead of traditional neural network layers, generative recommenders make the model much simpler and less memory-hungry. As a result, it can improve recommendation quality in test simulations and in real-world tests, while being faster than previous Transformer-based systems when handling long lists of user actions. Ultimately, this approach allows the model’s performance to grow steadily as more computing power is used, laying a foundation for efficient and scalable “foundation models” for recommendations.

The Netflix Prize

{{#invoke:Labelled list hatnote|labelledList|Main article|Main articles|Main page|Main pages}}

One of the events that energized research in recommender systems was the Netflix Prize. From 2006 to 2009, Netflix sponsored a competition, offering a grand prize of $1,000,000 to the team that could take an offered dataset of over 100 million movie ratings and return recommendations that were 10% more accurate than those offered by the company's existing recommender system. This competition energized the search for new and more accurate algorithms. On 21 September 2009, the grand prize of US$1,000,000 was given to the BellKor's Pragmatic Chaos team using tiebreaking rules.<ref name="nytimes.com">Template:Cite news</ref>

The most accurate algorithm in 2007 used an ensemble method of 107 different algorithmic approaches, blended into a single prediction. As stated by the winners, Bell et al.:<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref>

Predictive accuracy is substantially improved when blending multiple predictors. Our experience is that most efforts should be concentrated in deriving substantially different approaches, rather than refining a single technique. Consequently, our solution is an ensemble of many methods.

Many benefits accrued to the web due to the Netflix project. Some teams have taken their technology and applied it to other markets. Some members from the team that finished second place founded Gravity R&D, a recommendation engine that's active in the RecSys community.<ref name="nytimes.com"/><ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> 4-Tell, Inc. created a Netflix project–derived solution for ecommerce websites.

A number of privacy issues arose around the dataset offered by Netflix for the Netflix Prize competition. Although the data sets were anonymized in order to preserve customer privacy, in 2007 two researchers from the University of Texas were able to identify individual users by matching the data sets with film ratings on the Internet Movie Database (IMDb).<ref>Rise of the Netflix Hackers Template:Webarchive</ref> As a result, in December 2009, an anonymous Netflix user sued Netflix in Doe v. Netflix, alleging that Netflix had violated United States fair trade laws and the Video Privacy Protection Act by releasing the datasets.<ref>Template:Cite magazine</ref> This, as well as concerns from the Federal Trade Commission, led to the cancellation of a second Netflix Prize competition in 2010.<ref name="nfcancel">{{#invoke:citation/CS1|citation |CitationClass=web }}</ref>

Evaluation

Performance measures

Evaluation is important in assessing the effectiveness of recommendation algorithms. To measure the effectiveness of recommender systems, and compare different approaches, three types of evaluations are available: user studies, online evaluations (A/B tests), and offline evaluations.<ref name=":0" />

The commonly used metrics are the mean squared error and root mean squared error, the latter having been used in the Netflix Prize. The information retrieval metrics such as precision and recall or DCG are useful to assess the quality of a recommendation method. Diversity, novelty, and coverage are also considered as important aspects in evaluation.<ref>Lathia, N., Hailes, S., Capra, L., Amatriain, X.: Temporal diversity in recommender systemsTemplate:Dead linkTemplate:Cbignore. In: Proceedings of the 33rd International ACMSIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, pp. 210–217. ACM, New York</ref> However, many of the classic evaluation measures are highly criticized.<ref name="Turpin2001">Template:Cite book</ref>

Evaluating the performance of a recommendation algorithm on a fixed test dataset will always be extremely challenging as it is impossible to accurately predict the reactions of real users to the recommendations. Hence any metric that computes the effectiveness of an algorithm in offline data will be imprecise.

User studies are rather a small scale. A few dozens or hundreds of users are presented recommendations created by different recommendation approaches, and then the users judge which recommendations are best.

In A/B tests, recommendations are shown to typically thousands of users of a real product, and the recommender system randomly picks at least two different recommendation approaches to generate recommendations. The effectiveness is measured with implicit measures of effectiveness such as conversion rate or click-through rate.

Offline evaluations are based on historic data, e.g. a dataset that contains information about how users previously rated movies.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref>

The effectiveness of recommendation approaches is then measured based on how well a recommendation approach can predict the users' ratings in the dataset. While a rating is an explicit expression of whether a user liked a movie, such information is not available in all domains. For instance, in the domain of citation recommender systems, users typically do not rate a citation or recommended article. In such cases, offline evaluations may use implicit measures of effectiveness. For instance, it may be assumed that a recommender system is effective that is able to recommend as many articles as possible that are contained in a research article's reference list. However, this kind of offline evaluations is seen critical by many researchers.<ref name=":4">Template:Cite journal</ref><ref>Template:Cite book</ref><ref name=":1">Template:Cite book</ref><ref name=":0" /> For instance, it has been shown that results of offline evaluations have low correlation with results from user studies or A/B tests.<ref name=":1" /><ref>Template:Cite book</ref> A dataset popular for offline evaluation has been shown to contain duplicate data and thus to lead to wrong conclusions in the evaluation of algorithms.<ref name="BasaranNtoutsi2017">Template:Cite book</ref> Often, results of so-called offline evaluations do not correlate with actually assessed user-satisfaction.<ref>Template:Cite book</ref> This is probably because offline training is highly biased toward the highly reachable items, and offline testing data is highly influenced by the outputs of the online recommendation module.<ref name=":4" /><ref name="cañamares2018">Template:Cite conference</ref> Researchers have concluded that the results of offline evaluations should be viewed critically.<ref name="cañamares2020">Template:Cite journal</ref>

Beyond accuracy

Typically, research on recommender systems is concerned with finding the most accurate recommendation algorithms. However, there are a number of factors that are also important.

• Diversity – Users tend to be more satisfied with recommendations when there is a higher intra-list diversity, e.g. items from different artists.<ref name="Ziegler2005">Template:Cite book</ref><ref name="castells2015">Template:Cite book</ref>

• Recommender persistence – In some situations, it is more effective to re-show recommendations,<ref name="Beel2013e">Template:Cite book</ref> or let users re-rate items,<ref name="Cosley2003">Template:Cite book</ref> than showing new items. There are several reasons for this. Users may ignore items when they are shown for the first time, for instance, because they had no time to inspect the recommendations carefully.
• Privacy – Recommender systems usually have to deal with privacy concerns<ref name="Pu2012">Template:Cite journal</ref> because users have to reveal sensitive information. Building user profiles using collaborative filtering can be problematic from a privacy point of view. Many European countries have a strong culture of data privacy, and every attempt to introduce any level of user profiling can result in a negative customer response. Much research has been conducted on ongoing privacy issues in this space. The Netflix Prize is particularly notable for the detailed personal information released in its dataset. Ramakrishnan et al. have conducted an extensive overview of the trade-offs between personalization and privacy and found that the combination of weak ties (an unexpected connection that provides serendipitous recommendations) and other data sources can be used to uncover identities of users in an anonymized dataset.<ref name="privacyoverview">Template:Cite journal

</ref>

• User demographics – Beel et al. found that user demographics may influence how satisfied users are with recommendations.<ref name="Beel2013f">Template:Cite book</ref> In their paper they show that elderly users tend to be more interested in recommendations than younger users.
• Robustness – When users can participate in the recommender system, the issue of fraud must be addressed.<ref name="Konstan2012">Template:Cite journal</ref>
• Serendipity – Serendipity is a measure of "how surprising the recommendations are".<ref name="Ricci2011">Template:Cite book</ref><ref name=castells2015/> For instance, a recommender system that recommends milk to a customer in a grocery store might be perfectly accurate, but it is not a good recommendation because it is an obvious item for the customer to buy. "[Serendipity] serves two purposes: First, the chance that users lose interest because the choice set is too uniform decreases. Second, these items are needed for algorithms to learn and improve themselves".<ref>Template:Cite journal</ref>
• Trust – A recommender system is of little value for a user if the user does not trust the system.<ref name="Montaner2002">Template:Cite book</ref> Trust can be built by a recommender system by explaining how it generates recommendations, and why it recommends an item.
• Labelling – User satisfaction with recommendations may be influenced by the labeling of the recommendations.<ref name="Beel2013a">Template:Cite conference</ref> For instance, in the cited study click-through rate (CTR) for recommendations labeled as "Sponsored" were lower (CTR=5.93%) than CTR for identical recommendations labeled as "Organic" (CTR=8.86%). Recommendations with no label performed best (CTR=9.87%) in that study.

Reproducibility

Recommender systems are notoriously difficult to evaluate offline, with some researchers claiming that this has led to a reproducibility crisis in recommender systems publications. The topic of reproducibility seems to be a recurrent issue in some Machine Learning publication venues, but does not have a considerable effect beyond the world of scientific publication. In the context of recommender systems a 2019 paper surveyed a small number of hand-picked publications applying deep learning or neural methods to the top-k recommendation problem, published in top conferences (SIGIR, KDD, WWW, RecSys, IJCAI), has shown that on average less than 40% of articles could be reproduced by the authors of the survey, with as little as 14% in some conferences. The articles considers a number of potential problems in today's research scholarship and suggests improved scientific practices in that area.<ref>Template:Cite journal</ref><ref>Template:Cite book</ref><ref>Template:Cite book</ref> More recent work on benchmarking a set of the same methods came to qualitatively very different results<ref>Template:Cite book</ref> whereby neural methods were found to be among the best performing methods. Deep learning and neural methods for recommender systems have been used in the winning solutions in several recent recommender system challenges, WSDM,<ref>Template:Cite journal</ref> RecSys Challenge.<ref>Template:Cite book</ref> Moreover, neural and deep learning methods are widely used in industry where they are extensively tested.<ref name="ntfx">Yves Raimond, Justin Basilico Deep Learning for Recommender Systems, Deep Learning Re-Work SF Summit 2018</ref><ref name="yt"/><ref name="amzn"/> The topic of reproducibility is not new in recommender systems. By 2011, Ekstrand, Konstan, et al. criticized that "it is currently difficult to reproduce and extend recommender systems research results," and that evaluations are "not handled consistently".<ref>Template:Cite book</ref> Konstan and Adomavicius conclude that "the Recommender Systems research community is facing a crisis where a significant number of papers present results that contribute little to collective knowledge [...] often because the research lacks the [...] evaluation to be properly judged and, hence, to provide meaningful contributions."<ref>Template:Cite book</ref> As a consequence, much research about recommender systems can be considered as not reproducible.<ref name=":2">Template:Cite journal</ref> Hence, operators of recommender systems find little guidance in the current research for answering the question, which recommendation approaches to use in a recommender systems. Said and Bellogín conducted a study of papers published in the field, as well as benchmarked some of the most popular frameworks for recommendation and found large inconsistencies in results, even when the same algorithms and data sets were used.<ref>Template:Cite book</ref> Some researchers demonstrated that minor variations in the recommendation algorithms or scenarios led to strong changes in the effectiveness of a recommender system. They conclude that seven actions are necessary to improve the current situation:<ref name=":2" /> "(1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research."

Artificial intelligence applications in recommendation

Template:More citations needed section Artificial intelligence (AI) applications in recommendation systems are the advanced methodologies that leverage AI technologies, to enhance the performance recommendation engines. The AI-based recommender can analyze complex data sets, learning from user behavior, preferences, and interactions to generate highly accurate and personalized content or product suggestions.<ref>Template:Cite book</ref> The integration of AI in recommendation systems has marked a significant evolution from traditional recommendation methods. Traditional methods often relied on inflexible algorithms that could suggest items based on general user trends or apparent similarities in content. In comparison, AI-powered systems have the capability to detect patterns and subtle distinctions that may be overlooked by traditional methods.<ref>Template:Cite journal</ref> These systems can adapt to specific individual preferences, thereby offering recommendations that are more aligned with individual user needs. This approach marks a shift towards more personalized, user-centric suggestions.

Recommendation systems widely adopt AI techniques such as machine learning, deep learning, and natural language processing.<ref name="Artificial intelligence in recommen">Template:Cite journal</ref> These advanced methods enhance system capabilities to predict user preferences and deliver personalized content more accurately. Each technique contributes uniquely. The following sections will introduce specific AI models utilized by a recommendation system by illustrating their theories and functionalities.Template:Citation needed

KNN-based collaborative filters

Collaborative filtering (CF) is one of the most commonly used recommendation system algorithms. It generates personalized suggestions for users based on explicit or implicit behavioral patterns to form predictions.<ref>Template:Cite journal</ref> Specifically, it relies on external feedback such as star ratings, purchasing history and so on to make judgments. CF make predictions about users' preference based on similarity measurements. Essentially, the underlying theory is: "if user A is similar to user B, and if A likes item C, then it is likely that B also likes item C."

There are many models available for collaborative filtering. For AI-applied collaborative filtering, a common model is called K-nearest neighbors. The ideas are as follows:

1. Data Representation: Create a n-dimensional space where each axis represents a user's trait (ratings, purchases, etc.). Represent the user as a point in that space.
2. Statistical Distance: 'Distance' measures how far apart users are in this space. See statistical distance for computational details
3. Identifying Neighbors: Based on the computed distances, find k nearest neighbors of the user to which we want to make recommendations
4. Forming Predictive Recommendations: The system will analyze the similar preference of the k neighbors. The system will make recommendations based on that similarity

Neural networks

An artificial neural network (ANN), is a deep learning model structure which aims to mimic a human brain. They comprise a series of neurons, each responsible for receiving and processing information transmitted from other interconnected neurons.<ref>Template:Cite journal</ref> Similar to a human brain, these neurons will change activation state based on incoming signals (training input and backpropagated output), allowing the system to adjust activation weights during the network learning phase. ANN is usually designed to be a black-box model. Unlike regular machine learning where the underlying theoretical components are formal and rigid, the collaborative effects of neurons are not entirely clear, but modern experiments has shown the predictive power of ANN.

ANN is widely used in recommendation systems for its power to utilize various data. Other than feedback data, ANN can incorporate non-feedback data which are too intricate for collaborative filtering to learn, and the unique structure allows ANN to identify extra signal from non-feedback data to boost user experience.<ref name="Artificial intelligence in recommen"/> Following are some examples:

• Time and Seasonality: what specify time and date or a season that a user interacts with the platform
• User Navigation Patterns: sequence of pages visited, time spent on different parts of a website, mouse movement, etc.
• External Social Trends: information from outer social media

Two-Tower Model

The Two-Tower model is a neural architecture<ref>Yi, X., Hong, L., Zhong, E., Tewari, A., & Dhillon, I. S. (2019). "A scalable two-tower model for estimating user interest in recommendations." Proceedings of the 13th ACM Conference on Recommender Systems.</ref> commonly employed in large-scale recommendation systems, particularly for candidate retrieval tasks.<ref>Google Cloud Blog. \"Scaling Deep Retrieval with Two-Tower Models.\" Published November 30, 2022. Accessed December 2024.</ref> It consists of two neural networks:

• User Tower: Encodes user-specific features, such as interaction history or demographic data.
• Item Tower: Encodes item-specific features, such as metadata or content embeddings.

The outputs of the two towers are fixed-length embeddings that represent users and items in a shared vector space. A similarity metric, such as dot product or cosine similarity, is used to measure relevance between a user and an item.

This model is highly efficient for large datasets as embeddings can be pre-computed for items, allowing rapid retrieval during inference. It is often used in conjunction with ranking models for end-to-end recommendation pipelines.

Natural language processing

Natural language processing is a series of AI algorithms to make natural human language accessible and analyzable to a machine.<ref>Template:Cite book</ref> It is a fairly modern technique inspired by the growing amount of textual information. For application in recommendation system, a common case is the Amazon customer review. Amazon will analyze the feedbacks comments from each customer and report relevant data to other customers for reference. The recent years have witnessed the development of various text analysis models, including latent semantic analysis (LSA), singular value decomposition (SVD), latent Dirichlet allocation (LDA), etc. Their uses have consistently aimed to provide customers with more precise and tailored recommendations.

Specific applications

Academic content discovery

An emerging market for content discovery platforms is academic content.<ref>Template:Cite journal</ref><ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> Approximately 6000 academic journal articles are published daily, making it increasingly difficult for researchers to balance time management with staying up to date with relevant research.<ref name="nature1"/> Though traditional tools academic search tools such as Google Scholar or PubMed provide a readily accessible database of journal articles, content recommendation in these cases are performed in a 'linear' fashion, with users setting 'alarms' for new publications based on keywords, journals or particular authors.

Google Scholar provides an 'Updates' tool that suggests articles by using a statistical model that takes a researchers' authorized paper and citations as input.<ref name="nature1"/> Whilst these recommendations have been noted to be extremely good, this poses a problem with early career researchers which may be lacking a sufficient body of work to produce accurate recommendations.<ref name="nature1"/>

Decision-making

Template:See also In contrast to an engagement-based ranking system employed by social media and other digital platforms, a bridging-based ranking optimizes for content that is unifying instead of polarizing.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref><ref name=":5">{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> Examples include Polis and Remesh which have been used around the world to help find more consensus around specific political issues.<ref name=":5" /> Twitter has also used this approach for managing its community notes,<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> which YouTube planned to pilot in 2024.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref><ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> Aviv Ovadya also argues for implementing bridging-based algorithms in major platforms by empowering deliberative groups that are representative of the platform's users to control the design and implementation of the algorithm.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref>

Television

As the connected television landscape continues to evolve, search and recommendation are seen as having an even more pivotal role in the discovery of content.<ref>The New Face of TV</ref> With broadband-connected devices, consumers are projected to have access to content from linear broadcast sources as well as internet television. Therefore, there is a risk that the market could become fragmented, leaving it to the viewer to visit various locations and find what they want to watch in a way that is time-consuming and complicated for them. By using a search and recommendation engine, viewers are provided with a central 'portal' from which to discover content from several sources in just one location.

See also

• Algorithmic radicalization
• ACM Conference on Recommender Systems
• Cold start
• Collaborative filtering
• Collective intelligence
• Configurator
• Enterprise bookmarking
• Filter bubble
• Information filtering system
• Information explosion
• Media monitoring service
• Pattern recognition
• Personalized marketing
• Personalized search
• Preference elicitation
• Product finder
• Rating site
• Reputation management
• Reputation system

Further reading

Books

• Kim Falk (d 2019), Practical Recommender Systems, Manning Publications, Template:ISBN
• Template:Cite book
• Template:Cite book
• Seaver, Nick (2022). Computing Taste: Algorithms and the Makers of Music Recommendation. University of Chicago Press.

Scientific articles

• {{#invoke:citation/CS1|citation

|CitationClass=web }}

• Prem Melville, Raymond J. Mooney, and Ramadass Nagarajan. (2002) Content-Boosted Collaborative Filtering for Improved Recommendations. Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI-2002), pp. 187–192, Edmonton, Canada, July 2002.Template:Cbignore