保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Recommendation System in Python: LightFM

A Step-by-Step guide to building a recommender system in Python using LightFM Shashank Kapadia February 25, 20209 min read A recommender system, or a recommendation system, can be thought of as a subclass of information filtering system that seeks to predict the best “rating” or “preference” a user would give to an item which is typically obtained by optimizing for objectives like total clicks, total revenue, and overall sales.

So, what is the basic principle that underlies the working of recommendation algorithms? The basic principle of recommendations is that there are significant dependencies between user- and item-centric activity. For example, a user who is interested in hotels in New York City is more likely to be interested in other NYC hotels, rather than in Boston.

Introduction

While the primary business goal of any recommender system is to provide users’ with a personalized experience; from the problem setting standpoint, we can broadly classify recommenders into two categories: Learn this step by step with the interactive AI and Data Scientist roadmap.

Image Source

Image Source

Basics of recommendation modeling approaches

Broadly speaking, most recommender systems leverage two types of data:

The modeling approach relying on the former data is generally known as the Collaborative Filtering method, and approaches using the latter are referred to as Content Base Filtering method. There is also another category known as Knowledge-Based recommender system that is based on explicitly specified user requirements.

Of course, each of these methods has its strengths and weaknesses depending on which applications they are used for, and the amount of data available. Hybrid Systems are then used to combine the advantages of these approaches to have a robust performing system across a wide variety of applications.

There is more to different types of approaches, and nuances of each; if you are interested in learning more, I would highly recommend the Statistical Methods for Recommender Systems book by Deepak Agarwal.

Primer into LightFM

There are many open-source recommender system frameworks and libraries available across different languages that can help you get started with your first implementation of the recommendation model. For this article, we’ll explore one of them that I have found valuable in my work, and covers a wide variety of underlying algorithms for varied use-cases, known as LightFM. LightFM is a Python implementation of several popular recommendation algorithms for both implicit and explicit feedback types. Importantly, it allows you to incorporate both item and user metadata into the traditional matrix factorization algorithms, making it possible to generalize to new items (via item features) and new users (via user features). To learn more about it, you can check its documentation here.

Implementation

The complete code is available as a Jupyter Notebook on GitHub.

Loading data

For this tutorial, we’ll use book reviews data by Goodreads. The datasets were collected in late 2017 from goodreads.com, where the data from users’ public shelves was scraped, i.e., everyone can see it on the web without login. User IDs and review IDs are anonymized. The dataset is collected for academic use only. Please do not redistribute them or use for commercial purposes. There are three groups of datasets:

These datasets can be merged together by matching book/user/review ids. For this tutorial, we’ll be using only the former two. You can download the dataset we'reusing in this article from here:

python

%%time
books_metadata = pd.read_json('./data/goodreads_books_poetry.json', lines=True)
interactions = pd.read_json('./data/goodreads_interactions_poetry.json', lines=True)

Data inspection & preparation

Books metadata

Let’s start by inspecting the books’ metadata information. To develop a reliable and robust ML model, it is essential to get a thorough understanding of the available data. As the first step, let’s take a look at all the available fields, and sample data: python

books_metadata.sample(2)

Books Metadata (Sample) While all the available information is vital to extract contextual information to be able to train a better recommendation system, for this example, we’ll only focus on the selected fields that require minimal manipulation. python

# Limit the books metadata to selected fields
books_metadata_selected = books_metadata[['book_id', 
'average_rating', 'is_ebook', 'num_pages', 'publication_year', 
'ratings_count', 'language_code']]
books_metadata_selected.sample(5)

Filtered Books Metadata (Sample) Now that we have the data with selected fields, next, we’ll run it through pandas profiler to perform preliminary exploratory data analysis to help us better understand the available data. python

import pandas_profiling
import numpy as np
# replace blank cells with NaN
books_metadata_selected.replace('', np.nan, inplace=True)
# not taking book_id into the profiler report
profile = \
pandas_profiling.ProfileReport(books_metadata_selected[['average_rating', 
'is_ebook', 'num_pages', 'publication_year', 'ratings_count']])
profile.to_file('./results/profiler_books_metadata_1.html')

Considering the results from the profiler, we’ll perform following transformations to the dataset:

python

# using pandas cut method to convert fields into discrete intervals
books_metadata_selected['num_pages'].replace(np.nan, -1, inplace=True)
books_metadata_selected['num_pages'] = pd.to_numeric(books_metadata_selected['num_pages'])
books_metadata_selected['num_pages'] = pd.cut(books_metadata_selected['num_pages'], bins=25)
# rounding ratings to neares .5 score
books_metadata_selected['average_rating'] = books_metadata_selected['average_rating'].apply(lambda x: round(x*2)/2)
# using pandas qcut method to convert fields into quantile-based discrete intervals
books_metadata_selected['ratings_count'] = pd.qcut(books_metadata_selected['ratings_count'], 25)
# replacing missing values to year 2100
books_metadata_selected['publication_year'].replace(np.nan, 2100, inplace=True)
# replacing missing values to 'unknown'
books_metadata_selected['language_code'].replace(np.nan, 'unknown', inplace=True)
# convert is_ebook column into 1/0 where true=1 and false=0
books_metadata_selected['is_ebook'] = books_metadata_selected.is_ebook.map(
    lambda x: 1.0*(x == 'true'))
profile = pandas_profiling.ProfileReport(books_metadata_selected[['average_rating', 'is_ebook', 'num_pages', 
                                                        'publication_year', 'ratings_count']])
profile.to_file('./results/profiler_books_metadata_2.html')

Interactions data

As the first step, let’s take a look at all the available fields and sample data: python

interactions.sample(5)

Limit the data to only the selected fields: python

# Limit the books metadata to selected fields
interactions_selected = interactions[['user_id', 'book_id', 'is_read', 'rating']]
# mapping boolean to string
booleanDictionary = {True: 'true', False: 'false'}
interactions_selected['is_read'] = interactions_selected['is_read'].replace(booleanDictionary)
profile = pandas_profiling.ProfileReport(interactions_selected[['is_read', 'rating']])
profile.to_file('./results/profiler_interactions.html')

Considering the results from the profiler, we’ll convert is_read column to 1/0: python

# convert is_read column into 1/0 where true=1 and false=0
interactions_selected['is_read'] = interactions_selected.is_read.map(
    lambda x: 1.0*(x == 'true'))

Since we have two fields denoting interaction between a user and a book, is_read and rating — let’s see how many data points we have where the user hasn’t read the book but have given the ratings. python

interactions_selected.groupby(['rating', 'is_read']).size().reset_index().pivot(columns='rating', index='is_read', values=0)

From the above results, we can conclusively infer that users with ratings >= 1 have all read the book. Therefore, we’ll use the ratings as the final score, drop interactions where is_read is false, and limit interactions from random 5000 users to limit the data size for further analysis.

Data preprocessing

Now, let’s transform the available data into CSR sparse matrix that can be used for matrix operations. We will start by the process by creating books_metadata matrix which is np.float64 csr_matrix of shape ([n_books, n_books_features]) — Each row contains that book’s weights over features. However, before we create a sparse matrix, we’ll first create an item dictionary for future reference. python

item_dict ={}
df = books_metadata[['book_id', 'title']].sort_values('book_id').reset_index()
for i in range(df.shape[0]):
    item_dict[(df.loc[i,'book_id'])] = df.loc[i,'title']
# dummify categorical features
books_metadata_selected_transformed = pd.get_dummies(books_metadata_selected, columns = ['average_rating', 'is_ebook', 'num_pages',                          'publication_year', 'ratings_count', 'language_code'])
books_metadata_selected_transformed = books_metadata_selected_transformed.sort_values('book_id').reset_index().drop('index', axis=1)
books_metadata_selected_transformed.head(5)
# convert to csr matrix
books_metadata_csr = csr_matrix(books_metadata_selected_transformed.drop('book_id', axis=1).values)

Next, we’ll create an iterations matrix which is np.float64 csr_matrix of shape ([n_users, n_books]). We’ll also create a user dictionary for future use cases. python

user_book_interaction = pd.pivot_table(interactions_selected, index='user_id', columns='book_id', values='rating')
# fill missing values with 0
user_book_interaction = user_book_interaction.fillna(0)
user_id = list(user_book_interaction.index)
user_dict = {}
counter = 0 
for i in user_id:
    user_dict[i] = counter
    counter += 1
# convert to csr matrix
user_book_interaction_csr = csr_matrix(user_book_interaction.values)
user_book_interaction_csr

Model training

Ideally, we would build, train, and evaluate several models for our recommender system to determine which model holds the most promise for further optimization (hyper-parameter tuning). However, for this tutorial, we’ll train the base model with randomly selected input parameters for demonstrations. python

model = LightFM(loss='warp',
                random_state=2016,
                learning_rate=0.90,
                no_components=150,
                user_alpha=0.000005)
model = model.fit(user_book_interaction_csr,
                  epochs=100,
                  num_threads=16, verbose=False)

Top n-recommendations

For getting back the top-n recommendations, we’ll re-purpose the code by Aayush Agrawal here. python

def sample_recommendation_user(model, interactions, user_id, user_dict, 
                               item_dict,threshold = 0,nrec_items = 5, show = True):
    
    n_users, n_items = interactions.shape
    user_x = user_dict[user_id]
    scores = pd.Series(model.predict(user_x,np.arange(n_items), item_features=books_metadata_csr))
    scores.index = interactions.columns
    scores = list(pd.Series(scores.sort_values(ascending=False).index))
    
    known_items = list(pd.Series(interactions.loc[user_id,:] \
                                 [interactions.loc[user_id,:] > threshold].index).sort_values(ascending=False))
    
    scores = [x for x in scores if x not in known_items]
    return_score_list = scores[0:nrec_items]
    known_items = list(pd.Series(known_items).apply(lambda x: item_dict[x]))
    scores = list(pd.Series(return_score_list).apply(lambda x: item_dict[x]))
    if show == True:
        print ("User: " + str(user_id))
        print("Known Likes:")
        counter = 1
        for i in known_items:
            print(str(counter) + '- ' + i)
            counter+=1
print("\n Recommended Items:")
        counter = 1
        for i in scores:
            print(str(counter) + '- ' + i)
            counter+=1

Let’s check the results: python

sample_recommendation_user(model, user_book_interaction, 'ff52b7331f2ccab0582678644fed9d85', user_dict, item_dict)

Closing notes

Machine learning has become increasingly popular over the past decade, and recent advances in computational availability have led to exponential growth to people looking for ways how new methods can be incorporated to advance the field of Recommendation Systems. Often, we treat the Recommendation System as black-box algorithms. Still, hopefully, this post addressed to shed light on the underlying math, and intuitions behind it, and high-level code to get you started with building your first recommendation model. In the next article, we’ll go one step deeper into understanding how you can evaluate the performance of these models, and tune its hyper-parameters to get more intuitive and reliable results.

References

[1] https://en.wikipedia.org/wiki/Recommender_system [2] Agarwal, Deepak K.. Statistical Methods for Recommender Systems. Cambridge University Press. Kindle Edition [3] Charu, A. (1997). Recommender systems The Textbook. In Communications of the ACM (Vol. 40) [4] Mengting Wan, Julian McAuley, “Item Recommendation on Monotonic Behavior Chains”, in RecSys’18 [5] Mengting Wan, Rishabh Misra, Ndapa Nakashole, Julian McAuley, “Fine-Grained Spoiler Detection from Large-Scale Review Corpora”, in ACL’19 [6] https://github.com/aayushmnit/cookbook/blob/master/recsys.py Thanks for reading. If you have any feedback, please feel to reach out by commenting on this post, messaging me on LinkedIn , or shooting me an email (shmkapadia[at]gmail.com)

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

SK Written By

Shashank Kapadia

See all from Shashank Kapadia LightfmMachine LearningPythonRecommendation EngineRecommender Systems Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Data Science for Cycling - How to Visualize Gradient Ranges of a GPX Route

Data Science Part 5/6 - Visualize gradient ranges of a Strava route with Python and Plotly Dario Radečić March 2, 20226 min read

Building An Anomaly Detection System? 5 Requirements It Must Meet

Machine Learning Recommendations for a successful solution - from signal analysis to sharing insights at scale Alvin Chan February 8, 20217 min read

How does an AI Imagine the Universe?

Machine Learning Exploiting Generative Adversarial Networks to generate new worlds, galaxies and stars Davide Coccomini August 12, 20215 min read

Machine Learning on Snowflake

Machine Learning snowflake is one of the leading data platforms out there. In this article, we will explore the capabilities of its' snowpark python... Ransaka Ravihara December 26, 20229 min read

Training and Deploying a Custom Detectron2 Model for Object Detection using PDF Documents (Part 1...

Machine Learning Making your machine learn how to see PDFs like a human Noah Haglund November 29, 202311 min read

Time Series Forecasting with Salesforce's Merlion

Data Science An introduction to ML library Merlion for time series forecasting Konstantin Rink April 13, 20228 min read

Not All RecSys Problems Are Created Equal

Machine Learning How baseline strength, churn, and subjectivity determine complexity Diogo Leitão February 11, 20268 min read

4 Ways to Get the Most Out of Your Data Science Degree

Artificial Intelligence How I approached the balancing act of (1) enjoying my degree and (2) setting myself up for life after university in "the real world" Matt Chapman May 23, 20239 min read

Continuous Learning: A Data Scientist's Odyssey

Data Science Navigating the ever-changing field Zijing Zhu, PhD September 15, 202316 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close