1
$\begingroup$

I have a system where i get as input array of feature strings:

["kol","bol","sol","nol"]

The length of this array is dynamic, i can get 2, 4 or 6 etc, total features <20

I need to make a decision according to this array, the decision is another string:

x = ["feature1","feature5","feature3","feature8"] #in
y = "john" #decide

What I end up doing is creating a table, 1 if exist, 0 otherwise, for each training set (dataframe pandas):

feature1   feature2   feature3   feature4   feature5...  decision
1          0          1          0          1            1 (john mapped to 1, Ly to 2, etc)

I feed this into a Decision Tree Classifier using sklearn. (DecisionTreeClassifier) I train it with 100+ input feature arrays and desired outcomes.

It works, but i do have a feeling that it won't really provide value if the input will be different than trained data, because there is no real meaning/weight to these binary values.

These features strings comes from a Bag of Words in which if appear on a text, i extract it, to create a well defined set of features to train/predict.

  1. can I, or should I change the values from 1/0 to a more weighted ones? how do i get them?
  2. Is this a right approach assuming i have a bag of words in which i look for in a text and produce features that both in the text and the bag.
Improve this question
$\endgroup$
1
  • $\begingroup$ Can you elaborate how you're getting your inputs and what you want your outputs to do? $\endgroup$
    – Warlax56
    Feb 28, 2022 at 17:09

1 Answer 1

Reset to default
1
$\begingroup$

This looks closely similar to text classification. The main concept in any supervised classification is that the model receives the same features (in the same order) when it is applied as when it was trained.

This is why traditionally the bag of word representation is used: every word in the vocabulary is assigned an index $i$ to be represented as feature $i$. The value of the feature can be boolean (1 if present in the instance, 0 otherwise) or numerical (frequency of the word in the instance, or some more complex value like TFIDF). The meaning of these feature is simple: it tells the model whether a particular word is present or not. The model calculates how often a particular label is associated with a particular word. Thus in a decision tree the model is made of conditions such as: "if the instance contains word A and does not contain word B and contains word C then the label is Y".

Crucially, the vocabulary is fixed at the training stage. This implies that any new word found in the test instances cannot be used at all. This is the problem of out-of-vocabulary (OOV) words. It's also usually recommended to remove the least frequent words, because they likely happen by chance and cause a high risk of overfitting (see previous link). Overfitting is when the model thinks that there's a strong association between a particular word and a label even though it only had one or two examples which happened by chance.

Improve this answer
$\endgroup$
10
  • $\begingroup$ thanks a lot, but there are many things here that doesn't look good to me. First, the order i choose is arbitrary, which means if you basically just pick the closest vector to the input vector, you will be wrong, because i could flip this matrix as i wanted, the columns order had no real meaning hence the word "close" here doesn't really mean Euclidian close. Second, as you said i can not present new words/features, which means we are not really learning, just describing the vector dimension, so only if the input fit the EXACT vector we trained, we have a hit, otherwise, results are noise. $\endgroup$
    – baltiturg
    Feb 28, 2022 at 16:59
  • $\begingroup$ @baltiturg as I said in the answer, the order must not be arbitrary, the vocabulary must be indexed once on the training data and then the indexes must be re-used. Otherwise the columns don't have any meaning indeed, so the classifier cannot do anything. In python you can use CountVectorizer: first you use fit_transform on the training set, then you use only transform on the test set. $\endgroup$
    – Erwan
    Feb 28, 2022 at 17:42
  • $\begingroup$ Second point: the learning is what happens when the model finds the statistically significant patterns in the training data, building conditions like the one I gave as example. The goal of the learning is not for the model to understand what the words mean, but to discover which ones are associated with a particular label so that it can predict labels for fresh instances later on. Note that what you're describing about learning new words corresponds to some extent to what happens in deep learning: using word embeddings, the model can find semantic similarities between different words. ... $\endgroup$
    – Erwan
    Feb 28, 2022 at 17:48
  • $\begingroup$ ... This can give better results but it's a more advanced level, I'd suggest you understand and try the traditional approach before attempting it. $\endgroup$
    – Erwan
    Feb 28, 2022 at 17:48
  • $\begingroup$ thanks again, when you say "the order must not be arbitrary", what you mean? those features are words, they become columns, how do you organize the columns initially? you have no way to organize the columns order other than just randomly becaues the order have no meaning. You mean probably that i index them once randomly then reuse them. But hey if I expect a vector like [1100110] and i get instead [0100110] it might seems they are closed, but they are not! because the first scalar could as well be in a different location. You can't measure vectors distance without order, no? $\endgroup$
    – baltiturg
    Feb 28, 2022 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.