Editing Recommender system (section)

=== Reproducibility ===
Recommender systems are notoriously difficult to evaluate offline, with some researchers claiming that this has led to a [[reproducibility crisis]] in recommender systems publications. The topic of reproducibility seems to be a recurrent issue in some Machine Learning publication venues, but does not have a considerable effect beyond the world of scientific publication. In the context of recommender systems a 2019 paper surveyed a small number of hand-picked publications applying deep learning or neural methods to the top-k recommendation problem, published in top conferences (SIGIR, KDD, WWW, [[ACM Conference on Recommender Systems|RecSys]], IJCAI), has shown that on average less than 40% of articles could be reproduced by the authors of the survey, with as little as 14% in some conferences. The articles considers a number of potential problems in today's research scholarship and suggests improved scientific practices in that area.<ref>{{cite journal |last1=Ferrari Dacrema |first1=Maurizio |last2=Boglio |first2=Simone |last3=Cremonesi |first3=Paolo |last4=Jannach |first4=Dietmar |title=A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research |journal=ACM Transactions on Information Systems |date=8 January 2021 |volume=39 |issue=2 |pages=1–49 |doi=10.1145/3434185 |url=https://dl.acm.org/doi/10.1145/3434185 |arxiv=1911.07698|hdl=11311/1164333 |s2cid=208138060 }}</ref><ref>{{cite book |last1=Ferrari Dacrema |first1=Maurizio |last2=Cremonesi |first2=Paolo |last3=Jannach |first3=Dietmar |title=Proceedings of the 13th ACM Conference on Recommender Systems |chapter=Are we really making much progress? A worrying analysis of recent neural recommendation approaches |series=RecSys '19 |date=2019 |pages=101–109 |doi=10.1145/3298689.3347058 |hdl=11311/1108996 |chapter-url=https://dl.acm.org/authorize?N684126
 |access-date=16 October 2019 |publisher=ACM|arxiv=1907.06902 |isbn=978-1-4503-6243-6 |s2cid=196831663 }}</ref><ref>{{cite book |last1=Rendle |first1=Steffen |last2=Krichene |first2=Walid |last3=Zhang |first3=Li |last4=Anderson |first4=John |title=Fourteenth ACM Conference on Recommender Systems |chapter=Neural Collaborative Filtering vs. Matrix Factorization Revisited |date=22 September 2020 |pages=240–248 |doi=10.1145/3383313.3412488|arxiv=2005.09683 |isbn=978-1-4503-7583-2 |doi-access=free }}</ref>
More recent work on benchmarking a set of the same methods came to qualitatively very different results<ref>{{cite book|last1=Sun|first1=Zhu|last2=Yu|first2=Di|last3=Fang|first3=Hui|last4=Yang|first4=Jie|last5=Qu|first5=Xinghua|last6=Zhang|first6=Jie|last7=Geng|first7=Cong|title=Fourteenth ACM Conference on Recommender Systems |chapter=Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison |chapter-url=https://dl.acm.org/doi/10.1145/3383313.3412489|year=2020|pages=23–32|publisher=ACM|doi=10.1145/3383313.3412489|isbn=978-1-4503-7583-2|s2cid=221785064}}</ref> whereby neural methods were found to be among the best performing methods. Deep learning and neural methods for recommender systems have been used in the winning solutions in several recent recommender system challenges, WSDM,<ref>{{cite journal|last1=Schifferer|first1=Benedikt|last2=Deotte|first2=Chris|last3=Puget|first3=Jean-François|last4=de Souza Pereira|first4=Gabriel|last5=Titericz|first5=Gilberto|last6=Liu|first6=Jiwei|last7=Ak|first7=Ronay|title=Using Deep Learning to Win the Booking.com WSDM WebTour21 Challenge on Sequential Recommendations|url=https://web.ec.tuwien.ac.at/webtour21/wp-content/uploads/2021/03/shifferer.pdf|journal=WSDM '21: ACM Conference on Web Search and Data Mining|publisher=ACM|access-date=April 3, 2021|archive-date=March 25, 2021|archive-url=https://web.archive.org/web/20210325063047/https://web.ec.tuwien.ac.at/webtour21/wp-content/uploads/2021/03/shifferer.pdf|url-status=dead}}</ref> [[RecSys Challenge]].<ref>{{cite book|last1=Volkovs|first1=Maksims|last2=Rai|first2=Himanshu|last3=Cheng|first3=Zhaoyue|last4=Wu|first4=Ga|last5=Lu|first5=Yichao|last6=Sanner|first6=Scott|title=Proceedings of the ACM Recommender Systems Challenge 2018 |chapter=Two-stage Model for Automatic Playlist Continuation at Scale |chapter-url=https://dl.acm.org/doi/10.1145/3267471.3267480|year=2018|pages=1–6|publisher=ACM|doi=10.1145/3267471.3267480|isbn=978-1-4503-6586-4|s2cid=52942462}}</ref>
Moreover, neural and deep learning methods are widely used in industry where they are extensively tested.<ref name="ntfx">Yves Raimond, Justin Basilico [https://www2.slideshare.net/moustaki/deep-learning-for-recommender-systems-86752234 Deep Learning for Recommender Systems], Deep Learning Re-Work SF Summit 2018</ref><ref name="yt"/><ref name="amzn"/> The topic of reproducibility is not new in recommender systems. By 2011, [[Michael Ekstrand|Ekstrand]], [[Joseph A. Konstan|Konstan]], et al. criticized that "it is currently difficult to reproduce and extend recommender systems research results," and that evaluations are "not handled consistently".<ref>{{Cite book|last1=Ekstrand|first1=Michael D.|last2=Ludwig|first2=Michael|last3=Konstan|first3=Joseph A.|last4=Riedl|first4=John T.|title=Proceedings of the fifth ACM conference on Recommender systems |chapter=Rethinking the recommender research ecosystem |date=2011-01-01|series=RecSys '11|location=New York, NY, USA|publisher=ACM|pages=133–140|doi=10.1145/2043932.2043958|isbn=978-1-4503-0683-6|s2cid=2215419}}</ref> Konstan and Adomavicius conclude that "the Recommender Systems research community is facing a crisis where a significant number of papers present results that contribute little to collective knowledge [...] often because the research lacks the [...] evaluation to be properly judged and, hence, to provide meaningful contributions."<ref>{{Cite book|last1=Konstan|first1=Joseph A.|last2=Adomavicius|first2=Gediminas|title=Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation |chapter=Toward identification and adoption of best practices in algorithmic recommender systems research |date=2013-01-01|series=RepSys '13|location=New York, NY, USA|publisher=ACM|pages=23–28|doi=10.1145/2532508.2532513|isbn=978-1-4503-2465-6|s2cid=333956}}</ref> As a consequence, much research about recommender systems can be considered as not reproducible.<ref name=":2">{{Cite journal|last1=Breitinger|first1=Corinna|last2=Langer|first2=Stefan|last3=Lommatzsch|first3=Andreas|last4=Gipp|first4=Bela|date=2016-03-12|title=Towards reproducibility in recommender-systems research|journal=User Modeling and User-Adapted Interaction|language=en|volume=26|issue=1|pages=69–101|doi=10.1007/s11257-016-9174-x|s2cid=388764|issn=0924-1868|url=http://nbn-resolving.de/urn:nbn:de:bsz:352-0-324818}}</ref> Hence, operators of recommender systems find little guidance in the current research for answering the question, which recommendation approaches to use in a recommender systems. [[Alan Said|Said]] and [[Alejandro Bellogín|Bellogín]] conducted a study of papers published in the field, as well as benchmarked some of the most popular frameworks for recommendation and found large inconsistencies in results, even when the same algorithms and data sets were used.<ref>{{Cite book|last1=Said|first1=Alan|last2=Bellogín|first2=Alejandro|title=Proceedings of the 8th ACM Conference on Recommender systems |chapter=Comparative recommender system evaluation |date=2014-10-01|series=RecSys '14|location=New York, NY, USA|publisher=ACM|pages=129–136|doi= 10.1145/2645710.2645746|isbn=978-1-4503-2668-1|hdl=10486/665450|s2cid=15665277}}</ref> Some researchers demonstrated that minor variations in the recommendation algorithms or scenarios led to strong changes in the effectiveness of a recommender system. They conclude that seven actions are necessary to improve the current situation:<ref name=":2" /> "(1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research."