I have a set of tables containing some thousand entries and some tenths of columns from machine status values of production. The entries are of mixed types like string, float, or timestamp. Each table is pre-labeled with a certain failure mode (e.g. valve setting jump, the problem with inlet A, etc.). This could be due to a jump in the mean values in some columns or a special correlation between several columns. This is what I refer to as a pattern. I would like to generate a machine learning-based classification model for this data set that can recognize these patterns. This means that I need to feed in a table of tables as data points. Up to now I only came across either tabular data with individual data points (i.e. integer, float or strings) or images, which are kinds of tables but have a fixed ordering of rows and columns. Any idea on how to tackle this problem?
Can you share a sample of the data?
Otherwise, Random Forest Classifier could be a good solution to your problem... https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
thanks Nicolas for your hint. I doesn't work, though. I tried to create a minimal example as I can't share the original data:
from sklearn import datasets from sklearn.model_selection import train_test_split from random import randint #generate 100 datasets with arbitrary values dataset_x =  labels =  for i in range(100): x_values, _ = datasets.make_sparse_uncorrelated(n_features=20,n_samples=150) dataset_x.append(x_values) #label each dataset if randint(0,10) < 5: labels.append('failure') else: labels.append('no failure') #split into train and test x_train, x_test, y_train, y_test = train_test_split(dataset_x,labels,train_size=.8) #here is where the magic should happen from sklearn.ensemble import RandomForestClassifier clf = RandomForestClassifier(max_depth=5, random_state=0) clf.fit(x_train,y_train)
as expected, this fails (ValueError: Found array with dim 3. Estimator expected <= 2.).