How to predict a class for a file given a small number of files for training?

Question

I have been given the following data:

1. 20 example CSV files, each labeled as belonging to one of six fixed classes, say A, B, C, D, E, F.
2. Each file has roughly 20000 rows and 10 floating point columns.
3. Within each file, the values seem pretty noisy, but the relationships between pairs of columns seem pretty linear (but noisy).
4. I have not been given any domain knowledge related to the content of the files, except A) the files are likely experimental measurements, and B) that the order of the records should not have any effect on the classifier; i.e. classification whould be invariant under permuting rows of files.

I have been asked to see if there is a useful way to predict (with, for example, accuracy > 0.8) a class label for a previously unseen file.

At first I thought it was going to be a no-brainer, given the total number of records over all the files. But as I got into it, it seemed more an more like really I had only 20 training examples, and really felt as if I were exhausting them pretty quickly and data dredging.

It feels difficult.

I am wondering if there is a standard aproach in a situation like this.

Thanks for any help!

Answer 1

Given that only that most classes have less than four samples, it is not useful to do a train/test split. A train/test split is the most useful way to assess generalization.

One option could be to craft rules by hand. Explore the data and manually construct rule-based logic for each of the classes.

Answer 2

I agree with the other answer. Train-test split probably won't work.

Here's an idea: (Assuming the columns are features) Compute the mean of every column. So, for every file, you'll have 10 numbers. Then treat this as the coordinate of each file in a 10D space. Now check if they form any meaningful clusters. You can also try replacing the mean with median or even std.

Answer 3

I mostly agree with all the other answers, however, I'll try to propose another approach to the problem,

Given that the order of the rows doesn't affect the classification task, I would try to split each CSV file into multiple files, augmenting the data this way. For each original CSV with 20k rows, you can generate 20 files with only 1k rows. This way you would move from 20 files to 400 files. Then you can use the typical train/test split.

Of course, this approach has multiple problems. One of them is that train and test datasets would not be independent -aka data leak- which would make your estimation of the accuracy to be too optimistic.