Identifying patterns in tabular data

I have a set of tables containing some thousand entries and some tenths of columns from machine status values of production. The entries are of mixed types like string, float, or timestamp. Each table is pre-labeled with a certain failure mode (e.g. valve setting jump, the problem with inlet A, etc.). This could be due to a jump in the mean values in some columns or a special correlation between several columns. This is what I refer to as a pattern. I would like to generate a machine learning-based classification model for this data set that can recognize these patterns. This means that I need to feed in a table of tables as data points. Up to now I only came across either tabular data with individual data points (i.e. integer, float or strings) or images, which are kinds of tables but have a fixed ordering of rows and columns. Any idea on how to tackle this problem?

• @Berhard: I think the response you gave should be an edit to the question Nov 17, 2021 at 7:18

2 Answers

Can you share a sample of the data?

Otherwise, Random Forest Classifier could be a good solution to your problem... https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

thanks Nicolas for your hint. I doesn't work, though. I tried to create a minimal example as I can't share the original data:

from sklearn import datasets
from sklearn.model_selection import train_test_split
from random import randint

#generate 100 datasets with arbitrary values
dataset_x = []
labels = []
for i in range(100):
x_values, _ = datasets.make_sparse_uncorrelated(n_features=20,n_samples=150)
dataset_x.append(x_values)
#label each dataset
if randint(0,10) < 5:
labels.append('failure')
else:
labels.append('no failure')

#split into train and test
x_train, x_test, y_train, y_test = train_test_split(dataset_x,labels,train_size=.8)

#here is where the magic should happen
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(max_depth=5, random_state=0)
clf.fit(x_train,y_train)


as expected, this fails (ValueError: Found array with dim 3. Estimator expected <= 2.).