Identifying patterns in tabular data

Question

I have a set of tables containing some thousand entries and some tenths of columns from machine status values of production. The entries are of mixed types like string, float, or timestamp. Each table is pre-labeled with a certain failure mode (e.g. valve setting jump, the problem with inlet A, etc.). This could be due to a jump in the mean values in some columns or a special correlation between several columns. This is what I refer to as a pattern. I would like to generate a machine learning-based classification model for this data set that can recognize these patterns. This means that I need to feed in a table of tables as data points. Up to now I only came across either tabular data with individual data points (i.e. integer, float or strings) or images, which are kinds of tables but have a fixed ordering of rows and columns. Any idea on how to tackle this problem?

---

thanks Nicolas for your hint. I doesn't work, though. I tried to create a minimal example as I can't share the original data:

from sklearn import datasets
from sklearn.model_selection import train_test_split
from random import randint

#generate 100 datasets with arbitrary values
dataset_x = []
labels = []
for i in range(100):
    x_values, _ = datasets.make_sparse_uncorrelated(n_features=20,n_samples=150)
    dataset_x.append(x_values)
    #label each dataset
    if randint(0,10) < 5:
        labels.append('failure')
    else:
        labels.append('no failure')

#split into train and test
x_train, x_test, y_train, y_test = train_test_split(dataset_x,labels,train_size=.8)  

#here is where the magic should happen
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(max_depth=5, random_state=0)
clf.fit(x_train,y_train)

as expected, this fails (ValueError: Found array with dim 3. Estimator expected <= 2.).

Answer 1

Can you share a sample of the data?

Otherwise, Random Forest Classifier could be a good solution to your problem... https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Answer 2

I've not tried this.

But, it seems a special neural network might be amenable to this. Some LSTM or convolutional neurons could learn to extract time-series information per column like "a jump in the mean values in some columns", while fully-connected neurons can extract "a special correlation between several columns". The former should handle different input table sizes, while the latter might need some additional work (maybe those are also convolutional, which large filter sizes?).