New answers tagged

0

This is a standard problem with distance/similarity measures between texts of different length. I'm not aware of any standard way to solve it, but in your case I would simply try to remove any email shorter than a certain length from the training set (you can experiment with different thresholds). This would hopefully force the centroids to be more specific, ...


1

There can be no objective answer to this question. Obviously the more one understands the better, but the field of ML is vast, quite specialized and ranges from very theoretical to very applied research, so it's perfectly reasonable to publish in ML without a strong background in maths. A better way to estimate your own ability to publish papers in a ...


0

A common approach for this is LDA (Latent Dirichlet Allocation), which not only gives you the groups, but also a way to identify the topics of the groups by giving you the most common or distinctive words for each topic.


1

Lasso stands for ´least absolute shrinkage and selection operator´. It has a penalty that is the absolute value and makes a lot of variables converge to cero. There is a ton of blogs that explain really well Lasso on the internet, have a look! Elastic Net is a combination of Ridge and Lasso. So it will also reduce the variables a lot. Ridge is a quadratic ...


1

It doesn't seem that non_negative is an argument in some versions. Try using decode_error = 'ignore'. If you're working with a large dataset, this error could also be resulting from hash collisions, which can be solved by increasing the number of features: vect = HashingVectorizer(decode_error = 'ignore', n_features = 2**21, ...


0

You're probably seeing those artifacts because your model doesn't see those pixels immediately outside your tile and so can't know how to "blend" things. (I'm assuming your tiles have a stride equal to the input size) A typical approach I've seen used (and used myself) is, at inference time, to keep only a central portion of each tile and then to have ...


2

Edit: oh, now I think I see why @CarlosMougan said no. You said ...start the same GridsearchCV with the same parameter and just change... If you mean use the optimal values for all hyperparameters except n_estimators and now search only on that one hyperparameter, then Carlos is right, and for the right reason. Below, I interpreted your suggestion as ...


0

How would you know you have to do cluster analysis before looking at your data ? Setting aside data quality questions (which you should never do), a bare minimum of EDA will help you : Know if it's relevant to do a clustering analysis (rarely imo) Know if K-means is the best clustering tool (rarely imo) Get an idea of the number of the clusters Then you ...


0

you can develop simple predictive model like Linear regression to predict price of house given other features value, also analyse the features weight/coefficient and optimize your linear regression model.


0

1) It seems that your data are unbalanced, you should look into that. Common techniques include oversampling the minority class, but you might have a bigger problem here. 2) It is unclear that you have enough information for what you are trying to achieve (type of device and location doesn't seems to be enough). 3) Based on the two preceding remarks, you ...


1

Start with some natural thresholds : >3* average, >4* good, >4,5*: excellent, 4,9+* perfect. Then you can correct your rating based on some averages, other metrics or even text (but that's hard). Honestly I am not sure it will work as ratings should be viewable and giving different status to customers with same average rating will get noticed. Leave them ...


2

There cannot be a unique answer to your question. There is a discrepancy in your question though - I am aware that this is a classification problem on which I am working on. Could you please help me with the right step by step guide that I should follow in order to achieve an efficient clustering at the end? However, I am assuming that you are ...


2

First my understanding of your problem. You want to find the best hyperparameters for a Random Forest. For that, you want to first adjust n_estimators parameter and then the rest of parameters in different runs. Before answering to your question, you will only want to do a thorough search of hyperparameters when you want to have an improvement of around 1%...


0

If you are using a neural network for classification, here are a few things you can do on the data even if you don't have the labels for them. If the data points are real-valued vectors, you can normalize them by calculating the (featurewise) mean and standard deviation. You can train an autoencoder on this data (by reconstructing the original input), and ...


1

Check the values of train_new. You'll see that the columns mentionned are not of the expected types. Another suggestion, i'm not sure of xgboost's handling of nulls. Might be that in those columns aswell.


0

One solution to make the recall more important whitout getting to the problem mentionned by @Jurgy is to use $F_\beta$, a modified version of $F_1$ where recall is considered $\beta$ time more important. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated both in terms of recall/precision, and in term of type I /type II error. ...


0

For a given class, you may usually treat the problem as a binary classification (the given class versus all others). You can do the same for your feature importance calculations. However we won't be able to help you much outside of a given problem and a given model, as available feature importance solution depends on the model used.


0

I did something similar a while ago. We wanted to classify several types of pdf. We first extracted the text of the documents. We created NLP features with the text Then added pdf metadata: size of the file, number of pages, name of the document... We then built a classification model with a few samples and did Active Learning I guess that you could also ...


1

Most modern implementations do both, at least optionally. sklearn has max_features and bootstrap. ranger has mtry and replace/sample.fraction. xgboost's random forest has colsample_bynode and subsample. h2o has mtries/col_sample_rate_per_tree and sample_rate (and a couple modifiers).


0

Sure. Just treat the range as a prior on the latent variables. Typically we use a boring prior (e.g., a normal distribution, a uniform distribution), but in your case, if $X_7$ is unknown and in the range $[0, 7.3]$, then your prior for $X_7$ could be the uniform distribution on that range. Then apply the machinery of the EM algorithm, and it should all ...


0

A few observations/questions/considerations: Wikipedia: Unsupervised learning is a type of self-organized Hebbian learning that helps find previously unknown patterns in data set without pre-existing labels. Being an anomaly is a label. Why not just Z-score it? https://en.wikipedia.org/wiki/Standard_score#Z-test . Calculate distribution: $\mu$ and $\sigma$, ...


4

Just to clarify (and I think you've got this right, but I'm just being careful), it is best practice to: 1: Split your data into train and test 2: Split train into train and eval 3: Grid search over hyperparameters, for each combination, train on train, evaluate on eval. Select the hyperparameters which allow you to get the best score on the eval set ...


0

As you can see from the error, it's that you're trying to predict using a subset of the features. There are a couple of duplicates (down to the code?), but perhaps without satisfactory answers: Sklearn ValueError: X has 2 features per sample; expecting 11 https://stackoverflow.com/q/57380948/10495893 Fixing this leads to a fundamental question of data ...


0

You can train an RNN with character embeddings. This can be done by splitting the name into sequences of chars and vectorize them numerically. If you are working with Keras, you can feed them into an Embedding() layer that will learn how to represent characters. RNN layers will then process their sequence. At the output node, your Network will perform a ...


3

You need to set bootstrap=False in the random forest to disable the subsampling. (I originally commented because I expected there to be more impediments [in addition to your already-coded random_states and max_features=None], but I guess there aren't any!) You probably don't want to do this in general; by stripping out all the randomness so that the first ...


0

I have a similar use-case and a working product based on tensorflow object-detection api and pytesseract for OCR. On top of the extracted text, I perform regex for validation of the extracted information and cleaning it to meet requirements of other processes. Steps: 1. Annotate images with some tool like labelimg. I annotated a set of 1K images, similar ...


2

The main problem I see here is that OHE is almost never a good idea with that many categories. With neural networks you will usually get better performance by using embeddings. So instead of X1 -{OHE}-> 10,000 -> {..} -> 1,000 you could go straight to X1 -{embedding}-> 50, where the embedding dimension should probably a lot lower than 1,000. ...


3

If you expect that all zeros is a result of error in the measuring of the features (i.e. the observations should not be all 0s but they are), then I would say: Keep all the data, but increase k (from k-means) by 1. This extra one will hopefully become the class of all these wrong observations. If you expect that all zeros is correct (i.e. these observations ...


3

It's a matter of data quality so it depends how the dataset was built: Either these instances are meaningful, i.e. it makes sense that an observation would have zeros for all the features and that it would happen that often. Or these are the result of an error, typically the complete absence of measurement for these observations. Naturally one wants to ...


0

I will go through your question one by one: How to use time series for this data You can train an RNN multivariate regressor, by feeding time series of your variables. Your first layer would be recurrent (LSTM or GRU), and provided with the following input_shape: ( batch size , input size , Number of variables ) I have to only two dimension bike ...


0

This project does have Machine Learning , Kmeans to be exact. It's in the third tutorial. Thanks to @Simon Larsson for pointing that out. I missed it. Kmeans is an unsupervised learning algorithm that does what is called Clustering.


0

Apart from the great answers mentioned here, one should also think about shift-invariance of softmax (or for that matter any exponential functions). Consider logits output from a classifier network (3 classes) [a, b, c]. Then the probability distribution will remain invariant even if it had been [a+x, b+x, c+x]. For example, if we consider e^e^x - e for ...


1

Statistical programs, such as R, typicall use Least Squares estimation. It's a deterministic algorithm that makes a linear model find its optimal tuple of parameters. Because of this, you don't have to worry about the choice of a loss function. In case you wanted to train your linear regression with a gradient descent algorithm, instead, you'd have to ...


1

When I undersample and train my classifier on a balanced dataset and test on a balanced dataset, the results are pretty ok It's not surprising that the results are good since the job is easier in this case. It's actually a mistake to test on the artificially balanced dataset, since it's not a fair evaluation of how the system will perform with real data. ...


0

It is therefore a problem of detecting the importance of the variables. You have many solutions based on the analysis of a regression model, Stepwise regression algorithms (backwards, forwards), random forest... However, the simplest solution is probably to use a decision tree: the results obtained are presented under a graphical form that is simple to ...


1

The purpose of the test split is normally to evaluate the performance of your model in data it has not seen before. While the available performance measures for GAN generators have their problems, they do exist. For images, you have Inception Score and Frechet Inception Distance. For text, you have quality vs. diversity plots. The evaluation measures ...


0

Keep in mind how matrix multiplication works with regards to the dimensions: Multiplying a matrix with dimensions $n,m$ by a matrix with dimensions $m,k$ results in a matrix of size $n,k$. Therefore, you can add as many rows as you like to the second matrix with no change to the shape of the result of the matrix multiplication. But of course the first ...


1

o(t) is not the result of concatenation of h(t-1) and x(t), but a simple matrix multiplication. See wikipedia for further details: https://en.wikipedia.org/wiki/Long_short-term_memory


0

Have a look at a few networks: -Siamese Network -One-Shot Learning Model Once going through these models you will understand how they can very well with very limited amounts of data.


0

It depends on what you want to achieve. If you want to visualize the results of your SVM classification, then you should do it after. If you want to reduce noise, speed up training ... or whatever reason you want to reduce the dimensionality of your problem. You should do it before. One idea here that can be useful, is to do LDA inside a pipeline and ...


0

But can I assign the cluster labels from the pca reduced data to the original data ? would it be a right approach ? I guess not. Yes, that is totally the right approach. Principal components are just the linear combinations of your original features that explain the most variance, so you can definitely use them for clustering. Moreover, since you only kept ...


1

There is plenty of material on the internet in the form of blog posts. This, together with the philosophy of learning by doing, leads me to the following advice: google "python XXX tutorial" where XXX stands for a basic machine learning algorithm and, for a few of the first results, use Google Colab to mimic what the tutorial explains. Whatever you don't ...


0

Uncertainty is fairly easy, if you have a probabilistic output. Just apply the model to unlabeled data sets and pick the one with highest average uncertainty. In the binary classification case, that's just lowest mean(abs(p - 0.5)). modAL (https://github.com/modAL-python/modAL) has some utilities that could be useful in the multi-class case where there are ...


0

A completely different answer: I am currently following a course Computer Vision and Image Analysis: https://courses.edx.org/courses/course-v1:Microsoft+DEV290x+1T2020/course/ With your problem in mind you could follow along. Depending on previous knowledge you could skip a few sections. (I skipped immediately to Beyond Classification/Object Detection) ...


0

I would suggest that you should use a pre-trained OCR model and train your own custom model which only outputs required data. Training method: Just use a pre-trained OCR model like this, and remove the tail of the model and add your custom output layer with the required number of fields (in your it's case invoice and date). After this, freeze the head of ...


1

Sorting data won't affect the training of your model, it is similar to changing the random seed. It can affect the validation that you are doing. In case you do time series you can do sliding window or roll-out-window, that they need the data to be sorted before the splitting. It seems that you want to do time series regression with supervised learning so ...


3

However Root Mean Square Error seems similar to MSE and is the root of it, gradient of RMSE with respect to $i^{th}$ prediction differes from that of MSE. $$\frac{\sigma{RMSE}}{\sigma{y_i}} =\frac{1}{2}\frac{1}{\sqrt{MSE}}\frac{\sigma MSE}{\sigma y_i}$$ Gradient of RMSE equals to the gradient of MSE multiplied by this $\frac{1}{2}\frac{1}{\sqrt{MSE}}$ ...


0

It is very hard to beat gradient boosted trees with native support for categorical features such as https://catboost.ai/ on a tabular data set. Assuming you data has not temporal/spatial/order structure (like in speech/image/text) I am very doubtful you will get a better results with deep learning. Having said that, the no-free-lunch theorem states that ...


Top 50 recent answers are included