# Tag Info

7

My suggestion would be to attempt to build some kind of clustering on your unlabeled data that somewhat approximates a labelled dataset. The rationale is more or less as follows: You have some feature vector for representing your documents Based on that feature vector, you can come up with a number of different clusterings, with either fuzzy, rough, or ...

4

Your problem belongs to the framework of PU learning (only positives, a lot of unlabelled). It is also close to the more common frameworks of Semi supervised learning (few positives and negatives, a lot of unlabeled). There are many survey papers that you can look up on the field. A classical method in the field, that was also tested on spam as in your ...

3

If the user inputs a title, then you could construct a system which finds the most similar titles in the corpus, and outputs the ingredients in the retrieved recipes. Some ideas below: 1) Represent the titles in a common vector space using a vocabulary-based vectorization and use Jaccard or Cosine similarity to find the most similar titles. examples of ...

3

I see two approaches: either you do all your process using only the positive class, so based on one class classification approaches such as isolation forest, one class SVM, recontruction error of autoencoder (trained only by your positive class), and so on... All those classifiers learn from one class. or you can do it in 2 steps. First try unsupervised ...

3

Supervised Learning In general, supervised learning refers to a situation in which you have some $X$ that is related to $y$, so that you can model how both are related (on average). In this case you can build a statistical model: $$y = \beta X + u.$$ The model learns $\beta$, so that you are able to predict $y$ by: $\hat{y}=\hat{\beta}X$. Examples are ...

3

It's a binary semi-supervised classification problem. First, establish a base-line for the supervised case. Then try if the unlabeled data helps Supervised From your labeled data: create a training, validation and test set. Don't touch the test set until the very end. Try something simple, e.g. a multilayer Perceptron (MLP) with 350 input nodes and 1 ...

3

A couple of points I have since found myself: I was right in suspecting that self-training could be used for PU learning. In fact, I found the original paper on PU Learning, and indeed the paper is a variation on self-training. (Oddly enough, the original authors had Positive, Unlabeled and Negative examples!) The authors of this survey identify three ...

2

The framework you cope with is semi supervised. You have mostly unlabelled data and you can have some labeled data by manual labelling. Active learning is one method to cope with the situation, by focusing your labelling efforts in the most beneficial areas. You can read a survey on these techniques at Settles, Burr (2010), "Active Learning Literature ...

2

Train 2 generative models, one for each dataset (spam only, spam plus ham), that will give you the probability that a datapoint is drawn from the same probability distribution of the training data. Assign emails as spam or ham based on which model gives you the highest probability of the document arising from the training data used to train it. Example ...

2

Why can they not just use a binary classifier where the negative class=unlabeled? E.g., when there are only a small portion of data are labeled as positive samples. It happens in reality where you don't have enough resources to label all the data. If you trained your model assuming all unlabeled data are negative samples, your model will have to find a ...

2

Given your time budget and the potential challenges associated with class imbalance, I'd throw away the unlabelled data and use supervised learning on the labelled data. Try a simple classifier, e.g., logistic regression or random forests or xgBoost, and use cross-validation to see how well they perform. In advance put aside a held-out test data and don't ...

1

Semi-supervised learning is having label for a fraction of data, but in self-supervised there is no label available. Imagine a huge question/answer dataset. No one labels that data but you can learn question answering right? Because you are able to retrieve relation between question and answer from data. Or in modeling documents you need sentences which are ...

1

There are a variety of techniques that you could use, depending on what you would like to do. If your goal is to gain insight into the phrases that are being used in each group, then I'd recommend looking for the most frequent N-grams of different lengths that appear in each class. Here is a related stackoverflow question that shows how you can use nltk and ...

1

I don't think it's possible to know for sure if PU learning would work in your setting or not. It's certainly relevant to cases like the one you describe, so it would be worth trying. But there are other valid options, and even within PU learning there are different approaches to choose from (you might be interested in this question). In my opinion the ...

1

I'm not sure I understand your problem very well but let's see. First let me try to formalize the task as a ML problem: Identifying the SMS of interest is a binary classification task. Your "completeness" score seems to correspond to the standard recall measure. It is usually a good idea to also look at precision, i.e. out of the SMS identified as ...

1

In constrained clustering you can provide examples of objects that should, or that must not, be in the same cluster. This can be used, e.g., for model selection: run several times and return the result with the fewest violations. Or to guide cluster extraction from a hierarchical result.

1

You should be able to solve this with currying. Make a function that takes the label as input and returns a function which takes y_true and y_pred as input. Note that the label needs to be a constant or a tensor for this to work. def conditional_loss_function(l): def loss(y_true, y_pred): if l == 0: return loss_funtion1(y_true, ...

1

Taking the specific example of Collaborative-Filtering recommender systems, an initial dataset containing a large percentage of positive examples and a small percentage of unlabelled examples for one class, is often tackled by imputing a negative value to the unlabelled examples based on the argument of popularity. This means that for a class which has ...

1

It's classification, isn't it? You have labeled training data. You want to label your test set accordingly. Use a classifier...

1

What you are looking for is called KNN algorithm, also knows as k-nearest neighbours. It’s a supervised algorithm where you have points and their clusters given and you use these to learn a pattern for test points.

1

Best solution would be to use the inception pertained model in your case as it's very good with over 1000 classes. Also you can label some of the data and make a CNN in Tensforflow, and then use this data to classify other data.

1

Most of the semi-supervised methods are heuristics and more or less are modifications of the standard supervised learning algorithms, where you are trying to take into account unlabeled data considering a small dataset of labeled data. If the data from these two datasets do not follow the same distribution, then you have to look for transfer learning methods....

1

I think those are one of the most cited papers: https://arxiv.org/pdf/1409.7495.pdf http://www.jmlr.org/papers/volume17/15-239/15-239.pdf http://openaccess.thecvf.com/content_cvpr_2017/paper/Tzeng_Adversarial_Discriminative_Domain_CVPR_2017_paper.pdf http://openaccess.thecvf.com/content_cvpr_2017/papers/Bousmalis_Unsupervised_Pixel-...

1

If you do not have data then I think the problem is more of research than Machine Learning. Ask your research team to gather primary data about the product by conducting surveys, polls,etc. Depending upon the data you get from research, you can use some statistical methods like hypothesis testing to solve the problem.

1

One strategy that seems good here is instance-level constrained clustering. These methods are semi-supervised algorithm that have "must-link" and "cannot-link" constraints between instances of known labels. So in your example, you would bind the 4 pairs (red, blue), (red, yellow), (blue, yellow), and (man, woman) as "must-link", and the 6 pairs (red, man), (...

1

From your description of the data, this is not a time-series problem. Time is not a factor here, for each user you have a set of variables after you choose a time threshold. Although you seek the "when", that's probably out of your reach, based on your explanation. If I understand correctly you're doing classification but you're unsure about your "data ...

1

Whenever you have skewed data-set, it means that you know a typical class better than the others. In such cases it means that the data is your knowledge and is not in a way that finds the minimum of the Bayes error because you don't know the distribution of other available classes and consequently you won't be able to find out whether the distribution of ...

1

I do not know any papers, I would be greatly appreciated if someone would link some. In my case, I always test by un-labelling my already known labelled Data by using a "traditional" 67% - 33% (train - test) split and checking how the labelling performs in various metrics (Accuracy, Logloss, etc.). Moreover, there are various categories of Semi-Supervised ...

1

I just want to chime in and highlight that using clustering to segment your data before modelling might be a bad idea. At a high level, clustering is a two/three dimensional visualisation of similarity. If you have two/three dimensions to your dataset you're probably OK using clustering to segment your data as all the possible relationships are being ...

1

You could also try tSNE, also available in scikit-learn. It is a probabilistic dimensionality reduction technique, specifically designed for plotting data in two or three dimensions, while preserving the original distances between the samples as much as possible. You should be able to see how well your data is clustered.

Only top voted, non community-wiki answers of a minimum length are eligible