Sales Transactions Dataset Weekly¶
https://archive.ics.uci.edu/ml/machine-learning-databases/00396/
https://archive.ics.uci.edu/ml/datasets/Sales_Transactions_Dataset_Weekly
The dataset contains weekly purchased quantities of 811 products over 52 weeks.
Normalised values are also provided.
The objective is here to build a clustering model to identify natural grouping of the product based on purchase patterns.
Summary of Key information
Number of Products : 811
Number of Products with missing attributes : None
Number of qualified Products : 811
Number of Input Attributes : 53
Number of categorical attributes : 01(Product code)
Number of numerical attributes : 52(Sales qty for 52 weeeks)
Problem Identification : Unsupervised Learning (Clustering)Loading necessary libraries¶
In [1]:
import numpy as np
import pandas as pd
import time
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from sklearn.preprocessing import StandardScaler, MinMaxScaler
In [1]:
from urllib.request import urlopen
url = "https://archive.ics.uci.edu/ml/datasets/Sales_Transactions_Dataset_Weekly"
page = urlopen(url)
html = page.read().decode("utf-8")
start_index = html.find("<title>") + len("<title>")
end_index = html.find("</title>")
title = html[start_index:end_index]
title
Out[1]:
In [559]:
#print(html)
Importing the dataset¶
In [544]:
path = "/Users/bhaskarroy/BHASKAR FILES/BHASKAR CAREER/Data Science/" \
"Practise/Python/UCI Machine Learning Repository/" \
"Sales_Transaction_Weekly_Dataset/"
path1 = path + "Sales_Transactions_Dataset_Weekly.csv"
In [545]:
df = pd.read_csv(path1)
In [548]:
# Inspecting the data
df.head()
Out[548]:
In [547]:
df.describe()
Out[547]:
In [9]:
df.info()
In [10]:
df.columns[:55]
Out[10]:
In [11]:
df.shape
Out[11]:
In [60]:
# Verifying if normalised values provided are correct
In [114]:
df.iloc[0:2, 0:53].set_index('Product_Code')
Out[114]:
In [105]:
df.iloc[0:2,np.r_[0,-52:-1:1]].set_index('Product_Code')
Out[105]:
In [113]:
df.iloc[0:2,0:53].set_index('Product_Code') \
.apply(lambda x : (x-x.min())/(x.max()-x.min()), axis = 1)
Out[113]:
Dataset Info¶
Abstract:
Contains weekly purchased quantities of 800 over products over 52 weeks. Normalised values are provided too.
Attribute Information:
Product_Code
52 weeks: W0, W1, ..., W51.
Normalised vlaues of weekly data: Normalised 0, Normalised 1, ..., Normalised 51
Data Set Characteristics:
Multivariate, Time-Series
Number of Instances:
811
Area:
N/A
Attribute Characteristics:
Integer, Real
Number of Attributes:
53
Date Donated
2017-07-16
Associated Tasks:
Clustering
Missing Values?
N/A
Number of Web Hits: 92132
Defining the Problem¶
We have the sales quantity of 811 products over a period of 52 weeks. The available information can be used for two kinds of problems.
- Firstly, we can create a time series forecasting model to predict the sales qty for next week.
- Secondly, we can build a clustering model to identify natural groupings of the product based on the weekly sales pattern.
In this notebook, we will focus on the clustering problem.
The forecasting problem will taken up in a separate notebook.
Finding Product Clusters¶
We have chosen k-means clustering as it is one of the most extensively used clustering algorithms. There is no hierarchy in k means clustering which suits our problem. Whereas, Hierarchical clustering creates a tree using either of agglomerative(bottom-up) and divisive(top-down) approaches. For our particular example, we don't require hierarchical tree.
In [373]:
#import libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import missingno as msno
import scipy.cluster.hierarchy as sch
from sklearn.preprocessing import MinMaxScaler
from sklearn.cluster import KMeans
from sklearn.cluster import AgglomerativeClustering
Preparing the data¶
In [549]:
# Extracting the sales qty values as the input data
# The product codes are unique and will be used as index
col_num = [0]
col_num.extend(list(np.arange(1,53)))
X = df.iloc[:,col_num] \
.set_index(keys = "Product_Code")
X.head()
Out[549]:
In [551]:
# creating strandard scaled and min max scaled versions of the input data
# https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing
from sklearn.preprocessing import StandardScaler, MinMaxScaler
ssc, mmsc = StandardScaler(), MinMaxScaler()
X_ssc = ssc.fit_transform(X)
X_mmsc = mmsc.fit_transform(X)
In [376]:
# Inspecting the
#print(list(dir(ssc)))
#print('\n'*2)
#print(ssc.feature_names_in_)
#print('\n'*2)
#print(ssc.__getstate__())
K means clustering¶
We will choose the optimal number of clusters (k value) using both the elbow method and Silhouette analysis.
Choosing optimal number of clusters using Elbow method¶
Observations
From elbow method, 3 seems to be the most optimal value.
However, we cannot rule out 4 and 5. We will proceed with the Silhouette coefficient plot to confirm.
In the below code, we have checked the elbow plots with three forms of input data :
- Unscaled (which means no feature scaling)
- Standard scaling
- Minmax scaling
In [475]:
# Elbow plot for unscaled, standard scaled and minmax scaled data
from sklearn.cluster import KMeans
#plt.style.use('classic')
plt.rcdefaults()
lw = 0.75
marker_size = 3
titlefntsize = 10
labelfntsize = 8
labelalpha = 0.5
ticklabelsize = 8
ticklabelalpha = 0.7
gridalpha = 0.4
fig, axs = plt.subplots(3, figsize = (4,5), dpi = 150)
# Creating a user defined function to return elbow plot
def Elbow_plot(X, ax, ax_title, range_strt, range_end):
'''returns a elbow plot for evaluating optimum k value in k-means
X is the prepared dataframe.
ax is the axes.
ax_title is the title.
range_strt is start value of k for evaluating k means.
range_end is end value of k for evaluating k means.
'''
sns.set_style('white')
k_vals = list(range(range_strt, range_end+1))
WCSSE = []
for k in k_vals:
kmeans = KMeans(n_clusters=k, init='k-means++',random_state=0)
model = kmeans.fit(X)
WCSSE.append(model.inertia_)
ax.plot(k_vals, WCSSE, marker='o',
markersize=marker_size,linewidth = lw)
ax.set_title(ax_title, fontsize = titlefntsize)
ax.set_ylabel('Within Cluster SSE',
fontsize = labelfntsize,
alpha = labelalpha)
ax.set_xlabel('k',
fontsize = labelfntsize,
alpha = labelalpha)
for labels in ax.get_xticklabels():
labels.set(fontsize = ticklabelsize, alpha = ticklabelalpha)
for labels in ax.get_yticklabels():
labels.set(fontsize = ticklabelsize, alpha = ticklabelalpha)
ax.yaxis.grid(True, alpha = gridalpha)
# Adjusting visibility of the spines
#ax.set_frame_on(True)
for axspine in ax.spines:
ax.spines[axspine].set(visible = True, alpha = 0.5)
#ax.spines['right'].set_visible(False)
#ax.spines['top'].set_visible(False)
#ax.spines['left'].set(visible = True, alpha = 0.5)
#ax.spines['bottom'].set_visible(True)
#List of Data
Data_list = [X, X_ssc, X_mmsc]
#List of titles
Title_list = ['Unscaled', 'Scaled', 'MinMax Scaled']
for i, (data,title) in enumerate(zip(Data_list, Title_list)):
Elbow_plot(data, ax = axs[i], ax_title = title, range_strt = 2, range_end =20)
plt.tight_layout()
Silhouette Analysis for k-means clustering¶
From Silhouette Analysis, the number of clusters can be 3,4,5 in decreasing order of silhouette scores.
Cluster counts 10,8,6 are ruled out owing to presence of negative silhouette scores.
For standard scaled data, the silhouette scores for different k values are :
For n_clusters = 3, The average silhouette_score is : 0.609876
For n_clusters = 4, The average silhouette_score is : 0.56319
For n_clusters = 5, The average silhouette_score is : 0.449328
For n_clusters = 6, The average silhouette_score is : 0.3887
For n_clusters = 8, The average silhouette_score is : 0.358646
For n_clusters = 10, The average silhouette_score is : 0.290975
Note : Results for standard-scaled data are similar to the un-scaled and minmax scaled data.
In [552]:
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score
def cluster_evaluation(X, range_n_clusters, xlowerlim = -0.1, store_values = False):
'''Returns Silhouette plot for clusterwise samples and a second plot to show
distribution of points in the feature space of first two attributes
X : a dataframe.
range_n_clusters : a list of candidate values for the number of clusters.
xlower_lim : the lower limit of x axis for the silhouette scores plot
Usually, The silhouette coefficient can range from -1, 1.
store_values : Bool. False
If True, function will return a dictionary storing the silhouette scores
at overall level and individual clusters.
'''
# Create a subplot with 1 row and 2 columns
#rowcount = len(range_n_clusters)
#fig, axr = plt.subplots(nrows = rowcount, ncols = 2)
#fig.set_size_inches(18, rowcount*3)
#Intercluster gap for inserting blank space between silhouette
#plots of individual clusters, to demarcate them clearly.
intercluster_gap = 10
#Initializing a dictionary
dict_Kcluster = {}
for i,n_clusters in enumerate(range_n_clusters):
# Create a subplot with 1 row and 2 columns
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_size_inches(10, 4)
# The 1st subplot is the silhouette plot
# The silhouette coefficient can range from -1, 1 but depending on the
# observed minimum silhouette coefficient, we will set the lower limit of x.
ax1.set_xlim([xlowerlim,1])
# The (n_clusters+1)*10 is for inserting blank space between silhouette
# plots of individual clusters, to demarcate them clearly.
ax1.set_ylim([0, len(X) + (n_clusters + 1) * intercluster_gap])
# Initialize the clusterer with n_clusters value and a random generator
# seed of 10 for reproducibility.
clusterer = KMeans(n_clusters=n_clusters, random_state=10)
cluster_labels = clusterer.fit_predict(X)
# The silhouette_score gives the average value for all the samples.
# This gives a perspective into the density and separation of the formed
# clusters
silhouette_avg = silhouette_score(X, cluster_labels)
# Compute the silhouette scores for each sample
sample_silhouette_values = silhouette_samples(X, cluster_labels)
# Store values - silhoutte score in dictionary
if store_values == True:
dict_Kcluster[n_clusters] = {}
dict_Kcluster[n_clusters]['silhouette_score'] = {}
dict_Kcluster[n_clusters]['silhouette_score']['size'] = \
int(sample_silhouette_values.size)
dict_Kcluster[n_clusters]['silhouette_score']['avg'] = \
silhouette_avg.round(6)
dict_Kcluster[n_clusters]['silhouette_score']['max'] = \
sample_silhouette_values.max().round(6)
dict_Kcluster[n_clusters]['silhouette_score']['min'] = \
sample_silhouette_values.max().round(6)
print("For n_clusters =", n_clusters,
"The average silhouette_score is :", silhouette_avg.round(6))
y_lower = intercluster_gap
for i in range(n_clusters):
# Aggregate the silhouette scores for samples belonging to
# cluster i, and sort them
ith_cluster_silhouette_values = \
sample_silhouette_values[cluster_labels == i]
ith_cluster_silhouette_values.sort()
size_cluster_i = ith_cluster_silhouette_values.shape[0]
# Store values - cluster sizes in dictionary
if store_values == True:
dict_Kcluster[n_clusters]['cluster'+str(i)] = {}
dict_Kcluster[n_clusters]['cluster'+str(i)]['size'] = \
int(size_cluster_i)
dict_Kcluster[n_clusters]['cluster'+str(i)]['avg'] = \
ith_cluster_silhouette_values.mean()
dict_Kcluster[n_clusters]['cluster'+str(i)]['max'] = \
ith_cluster_silhouette_values.max()
dict_Kcluster[n_clusters]['cluster'+str(i)]['min'] = \
ith_cluster_silhouette_values.min()
#print(f' Cluster {i}: {size_cluster_i}')
y_upper = y_lower + size_cluster_i
color = cm.nipy_spectral(float(i) / n_clusters)
# Plotting silhouette values corresponding to each value in sample
ax1.fill_betweenx(np.arange(y_lower, y_upper),
0, ith_cluster_silhouette_values,
facecolor=color, edgecolor=color, alpha=0.7)
# Label the silhouette plots with their cluster numbers at the middle
ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))
# Compute the new y_lower for next plot
y_lower = y_upper + 10 # 10 for the 0 samples
ax1.set_title("The silhouette plot for the various clusters.")
ax1.set_xlabel("The silhouette coefficient values")
ax1.set_ylabel("Cluster label")
# The vertical line for average silhouette score of all the values
ax1.axvline(x=silhouette_avg, color="red", linestyle="--")
ax1.set_yticks([]) # Clear the yaxis labels / ticks
ax1.set_xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])
#---------------------------------------------------------------------------
# 2nd Plot showing the actual clusters formed
colors = cm.nipy_spectral(cluster_labels.astype(float) / n_clusters)
ax2.scatter(X.iloc[:, 0], X.iloc[:, 1], marker='.', s=30, lw=0, alpha=0.7,
c=colors, edgecolor='k')
# Labeling the clusters
centers = clusterer.cluster_centers_
# Draw white circles at cluster centers
ax2.scatter(centers[:, 0], centers[:, 1], marker='o',
c="white", alpha=1, s=200, edgecolor='k')
for i, c in enumerate(centers):
ax2.scatter(c[0], c[1], marker='$%d$' % i, alpha=1,
s=50, edgecolor='k')
ax2.set_title("The visualization of the clustered data.")
ax2.set_xlabel("Feature space for the 1st feature")
ax2.set_ylabel("Feature space for the 2nd feature")
plt.suptitle(("Silhouette analysis for KMeans clustering on sample data "
"with n_clusters = %d" % n_clusters),
fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()
# Return dictionary if store_values is True
if store_values == True:
return dict_Kcluster
Unscaled input Data¶
In [553]:
range_n_clusters = [3,4,5,6,8,10]
Kmeanwise_cluster_info = cluster_evaluation(X,range_n_clusters = range_n_clusters,
xlowerlim = -0.3, store_values= True)
In [466]:
pd.DataFrame(Kmeanwise_cluster_info[3]).T.fillna("")
Out[466]:
In [478]:
user_ids = []
frames = []
for user_id, d in Kmeanwise_cluster_info.items():
user_ids.append(user_id)
frames.append(pd.DataFrame.from_dict(d, orient='columns'))
df = pd.concat(frames, keys=user_ids).round(2)
df.fillna("")
Out[478]:
In [479]:
Kmeanwise_cluster_info[3]
Out[479]:
Standard scaled data¶
In [480]:
range_n_clusters = [3,4,5,6,8,10]
cluster_evaluation(pd.DataFrame(X_ssc, columns = ssc.get_feature_names_out().tolist()),
range_n_clusters = range_n_clusters)
From Silhouette Analysis, the number of clusters can be 3,4,5 in decreasing order of silhouette scores.
Cluster counts 10,8,6 are ruled out owing to presence of negative silhouette scores.
In [365]:
print(list(dir(ssc)))
Minmax scaled data¶
In [481]:
range_n_clusters = [3,4,5,6,8,10]
cluster_evaluation(pd.DataFrame(X_mmsc, columns = mmsc.get_feature_names_out().tolist()),
range_n_clusters = range_n_clusters)
Visualising the product clusters¶
On Raw Data¶
In [499]:
optimal_k = 3
clusterer = KMeans(n_clusters= optimal_k, random_state=10)
cluster_labels = clusterer.fit_predict(X_ssc)
In [554]:
X
Out[554]:
In [557]:
X_clustered = pd.concat([X, \
pd.DataFrame(cluster_labels, \
columns = ['cluster_label'], index = X.index)], \
axis = 1)
In [558]:
sns.scatterplot(x = 'W2', y = 'W3', data = X_clustered, hue = 'cluster_label', palette = 'muted')
Out[558]:
On PCA Transformed Data¶
In [534]:
from sklearn.decomposition import PCA
pca = PCA(n_components = 3)
X_pca = pca.fit_transform(X)
X_pca = pd.DataFrame(X_trans, columns = ['PCA1', 'PCA2','PCA3'], index = X.index )
explained_variance = pca.explained_variance_ratio_
In [535]:
X_pca_clustered = pd.concat([X_trans,
pd.DataFrame(cluster_labels,
columns = ['cluster_label'],
index = X.index)], axis = 1)
In [536]:
X_pca_clustered
Out[536]:
In [537]:
explained_variance
Out[537]:
In [539]:
sns.scatterplot(x = 'PCA1', y = 'PCA2', data = X_pca_clustered, hue = 'cluster_label', palette = 'muted')
Out[539]:
Summary¶
We had done below tasks on the data :
- k-means clustering was used to identify natural groupings of products based on Weekly sales quantity pattern.
- From elbow plot, 3 seemed to be most suited to be the optimal value of k. The other candidate values were 4 and 5.
- From Silhouette analysis, we have confirmed k = 3 with the highest silhouette score.
- For k = 5, 6, 8, 10, there were instances of negative silhouette coefficients.
- For both elbow method and silhouette analysis, We had used unscaled, standard scaled and min max scaled data. Results were similar.
- We have then proceeded with k = 3 and visualised the clusters successively using both raw and PCA transformed data.
Useful resources¶
- Grouping of Retail Items by Using K-Means Clustering Kusrini Kusrini
The Third Information Systems International Conference - Paper published by Elsevier in 2015 - https://inseaddataanalytics.github.io/INSEADAnalytics/CourseSessions/Sessions45/ClusterAnalysisReading.html
- https://quantdare.com/hierarchical-clustering/
- Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
https://www.sciencedirect.com/science/article/pii/0377042787901257 - On kmeans and PCA
https://stats.stackexchange.com/questions/92985/when-plotting-clustering-results-in-the-pca-coordinates-does-one-do-pca-or-clus?rq=1 - On drawbacks of k means and the no free lunch theorem
https://stats.stackexchange.com/questions/133656/how-to-understand-the-drawbacks-of-k-means - API design for machine learning software: experiences from the scikit-learn project
https://arxiv.org/pdf/1309.0238.pdf - https://scikit-learn.org/stable/modules/clustering.html#clustering-evaluation
- https://en.wikipedia.org/wiki/Silhouette_(clustering)