How to identify patterns in dataset [closed]

Question

I have a dataset that pertains to calls received to a hospital emergency helpline. My task is to identify patterns in the data.

How do I approach examining the dataset to identify patterns?

Extract from dataset.

Answer 1

Try general linear model to see how the features interact. Decide if the problem is classification or trend or anomaly then post back your discovery then I can help .

I applied K-Means, TSNE dimension reduction, and PCA and saw maybe two clusters. it is not conclusive clustering. There were a few observations discovered by severity, age, and gender of the heart events.

from sklearn.decomposition import PCA
from scipy.cluster.vq import vq, whiten
from scipy.cluster.hierarchy import linkage, fcluster,dendrogram
from sklearn.preprocessing import LabelEncoder
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from sklearn.cluster import KMeans
from sklearn.manifold import TSNE

calltype=['Emergency','Emergency','Emergency','Hosp to Hosp','Emergency','Emergency','Emergency','Emergency','Hosp to Hosp','Hosp to Hosp','Emergency','Hosp to Hosp','Emergency','Emergency','Emergency','Emergency','Emergency','Emergency','Emergency']
district=['District 1','District 2','District 3', 'District 4','District 5','District 1','District 2','District 3','District 4','District 5','District 1','District 2','District 3','District 4','District 5', 'District 1','District 2','District 3','District 4']
gender=['Male','Male','Male','Male','Female','Male','Male','Female','Female','Male','Male','Male','Male','Male','Male','Female','Male','Male','Female']
complaint=['Chest Pain','Heart Problem','Cardiac Arrest','Heart Problem','Cardiac Arrest','Chest Pain','Cardiac Arrest','Heart Problem','Heart Problem','Heart Problem','Heart Problem','Chest Pain','Heart Problem','Chest Pain','Heart Problem','Heart Problem','Heart Problem','Chest Pain','Heart Problem']
 age=[28,50,76,45,52,34,44,55,86,65,60,36,45,65,60,35,80,46,70]
 area=['Rural','Rural','Rural','Urban','Urban','Urban','Urban','Rural','Rural','Rural','Urban','Urban','Urban','Rural','Rural','Urban','Rural','Urban','Urban']




 df=pd.DataFrame({'calltype': calltype,'district':district,'gender':gender,'complaint':complaint,'age':age,'area':area})
 df['severity']=df['complaint'].apply(lambda row: 3 if row=='Chest Pain' else 2 if row=='Heart Problem' else 1 if row=='Cardiac Arrest' else 0)

 encoder=LabelEncoder()
 df['district']=encoder.fit(df['district']).transform(df['district'])
 df['gender']=encoder.fit(df['gender']).transform(df['gender'])
 df['area']=encoder.fit(df['area']).transform(df['area'])
 df['calltype']=encoder.fit(df['calltype']).transform(df['calltype'])
 print(df)

 sns.countplot(x='gender',hue='severity',data=df)
 plt.legend(['Cardiac Arrest','Heart Problem','Chest Pain'])
 plt.show()

 print('Gender 0=Female 1=Male')
 fp=df.pivot_table(index=['severity'],columns=['gender'],values=['age'],aggfunc='mean')
 print(fp)

 print("\n More males had severe heart problems")

 fig,ax=plt.subplots(figsize=(10,8))
 x=np.linspace(0,len(df),len(df))
 ax.bar(x,df['severity'],alpha=0.3,color='green')
 twin_ax=ax.twinx()
 df[['age']].plot(ax=twin_ax, c='red')
 plt.show()

 print('\nAges 30-40 had high frequencies of cardiac arrest')

 print("\nK-Means\n")
 labels=df['gender']
 xs=df['age']
 ys=df['severity']
 points=np.column_stack((xs,ys))
 model = KMeans(n_clusters=3)
 model.fit(points)
 print(model.inertia_)
 new_points=np.random.permutation(points)
 new_labels = model.predict(new_points)
 print(new_labels)
 xs = new_points[:,0]
 ys = new_points[:,1]
 plt.clf()
 # Make a scatter plot of xs and ys, using labels to define the colors
 _=plt.scatter(xs,ys,c=labels, alpha=0.5)
 # Assign the cluster centers: centroids
 centroids = model.cluster_centers_
 # Assign the columns of centroids: centroids_x, centroids_y
 centroids_x = centroids[:,0]
 centroids_y = centroids[:,1]
 # Make a scatter plot of centroids_x and centroids_y
 _=plt.scatter(centroids_x,centroids_y, marker='D',s=50)
 plt.show()

 print("\nTSNE\n")

 model=TSNE(learning_rate=100)
 X=df[['calltype','district','gender','age','area','severity']]
 transformed=model.fit_transform(X)

 xs=transformed[:,0]
 ys=transformed[:,1]

 plt.scatter(xs,ys,c=labels)
 plt.show()


 print("\nPCA\n")
 pca=PCA(n_components=3)
 pca.fit(X)

 transformed=pca.transform(X)
 print(transformed.shape)

 xs=transformed[:,0]
 ys=transformed[:,1]
 plt.scatter(xs,ys,c=labels)
 plt.show()

Answer 2

Your question implies that you want to find "latent (unobserved) variable(s)" in your data.

An example of a latent variable could be a class variable. For example, social scientists may attempt to identify the political party membership (class) of a sample of the population where this variable is unknown. A traditional approach would be:

1. Construct a dataset of numeric data that describes the population under study (e.g. - income, education level, age of individuals in a census area).
2. Arbitrarily set the number of political classes to study (e.g. republican, democrat, independent).
3. Run a "Latent Class Analysis" or KNN clustering algorithm against this data set in order to separate each record (individual) into 3 separate classes.
4. Study the classes to determine how well the algorithm grouped members by attribute (income, education level, age). Example: a heatmap of average values by class.

Your problem is made more complex by the types of input data in your spreadsheet. You will need to convert this data into numeric inputs. For example, a one-hot encoding of "calltype" before running a clustering algorithm against your dataset.

If you're up for a challenge, and you have a large (100k+) dataset, you might find interesting clusters by implementing the following algorithms:

1. Autoencoder --> KNN or Gaussian Mixture Model
2. Autoencoder --> UMAP --> KNN or Gaussian Mixture Model
3. ...some other deep approach + clustering algorithm

If you're up for the latter approach you'll want to study the Python Keras library.

Otherwise, if you are inexperienced with this problem you should start with:

1. converting your text inputs into numeric data
2. Run KNN with your chosen K-classes