3
Overfitting is "The production of an analysis which corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably." (Oxford dictionary)
When you fit a ML model, you use a dataset that you assume is a sample of the real statistical distribution you want to model. ...
3
The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take the left branch, and so your model has (slightly) less capacity. That may be a good or a bad thing, depending on where you land on the bias-variance curve.
So, ...
3
When you split the data , you are training your model on 80 percent of the dataset , but you are finding the accuracy of the remaining 20 percent of the data.
And when you are not splitting the data your model is learning and calculating accuracy from a similar dataset , hence the accuracy is pretty high.
You should:
1.Always split the data and try to ...
2
The two pictures you show illustrate how to interprete one perceptron and a MLP consisting of 3 layers.
Let us discuss the geometry behind one perceptron first, before explaining the image.
We consider a perceptron with $n$ inputs. Thus let $\mathbf{x} \in \mathbb{R}^{n}$ be the input vector, $\mathbf{w} \in \mathbb{R}^{n}$ be the weights, and let $b \in \...
2
This seems intuitive.
When we have a single-label response (Multi-Class setting), our response prior to encoding looks like:
[1, 4, 2, 3, 5]
After encoding,it becomes
[[1,0,0,0,0], [0,0,0,1,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,0,1]]
Indicating the presence of ith Class. But only one.
Similarly, in a multi-label setting-
Input should be - [[1,2,5], [4,1], [...
2
For data, you can search on IEEE Dataport , Kaggle.
For detecting medicine names and other info trained deep learning models like CNN or you can also perform fine-tuning from the existing model.
2
In the first code you split the data X randomly with this line:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)
Then after training the model you correctly apply it to the test set, which is 20% of the instances:
y_pred = xg.predict(X_test)
Whereas in the second code you apply the model to the full data X ...
2
Yes, you can recognize these cards. For easy implementation, you can check here and here. Also, you can build your custom neural network model with tensorflow, keras, pytorch etc. Recognizing visually "52 card deck" is already solved problem. Because, cards have good features & landmarks. You either can use neural networks or "old school&...
2
The correlation does not effect your model using decision trees in a classification problem.
In the theory of decision tree models, you don`t need correlation or check of multicollinearity. Because the split in decision trees is made of entropy/information gain.
The correlation does only check linear dependencies. The same is, when the dataset is highly ...
2
What you need to remember is :
Disk space : slow to read/write but cheap, any computer has > 100Go of hard disk
Memory : fast to read/write but expensive, usually a few Go
When you run an algorithm you usually use only RAM. In deep learning your dataset is often too big so you store it in the disk space and load pieces of it into RAM when needed for ...
2
You are thinking about this correctly. If data doesn't vary between your outcomes then it doesn't need to be included.
That being said, if you are using time series techniques such as trend decomposition to feature engineer, then changing the structure of your data could complicate interpretation (ie: what is a moving average if you've removed data points?).
...
2
I'll try and provide some intuition for you here, instead of focusing on the mechanics of the math behind the methods.
Imagine you are evaluating whether a coin is fair or not, so you collect a sequence of heads and tails as your data set. In MLE, we simply look at the data we collected and find the maximum likelihood... this works well when we have no prior ...
2
Missing values doesn't necessarily mean missing information. Sometime missing value represent an information in itself. For example: we have a data set which have features such as pool area, no. Of rooms and area. Now pool area have 90% of its value missing. You can create a new column called is_pool, which tells if the house has pool or not, from pool area ...
2
Very likely no!
Machine learning algorithms aren't magic, they cannot see or find stuff that is not there.
We know for a fact that some trends and hints exist that link a companies exterior communication to it's industry e.g. social media companies like blue logos (think Twitter, Facebook, linkedin, etc.).
However for the most part logos, brand names, etc. ...
1
This is regular text classification, but with very little text (only the job title). You could start with a simple one-hot encoding over the words in the job title, then apply your favourite algorithm (e.g. Naive Bayes, Decision Trees, etc). It will probably work better with some form of normalization of the words (at least using the lemma in order to mach ...
1
In Sutton & Barto, vectors are considered column vectors by default. So if you have this kind of product:
$$\mathbf{a}\mathbf{b}^T$$
where $\mathbf{a}$ and $\mathbf{b}$ are $d$ dimensional vectors, it does not calculate the scalar product. Instead it treats both vectors as matrices and calculates a matrix product, which will be a $d \times d$ matrix ...
1
Following discussion with Erwan: one of his previous answer partially has answered my question. However I would like to understand the following. One needs to have a corpus, then label news/tweets in fake/not fake, then run the model. But how the algorithm works on texts and takes relevant words or features for detecting fake news?
First, let me emphasize ...
1
Under/overfitting depends on two things: the amount of data in your dataset and the complexity of your model.
To identify when each of these is happening, you will have to split the data you have into two parts: training data and test data. You then train your model only on the training data, and then evaluate its performance (e.g. calculate its accuracy or ...
1
Please find beautiful, explanation about KDE, In your graph on X Coordinateif the tail is stretching long towards right side then its positively skewed, it means most of your data points were distributed to left side and vise versa for negative skewness.
Always we needs to ensure that data points on the graph needs to be equally distributed to form ...
1
You may find this resource helpful : https://xang1234.github.io/multi-label/
One possibility is to cluster similar labels together so that they are
processed together by the multilabel classification algorithms.
Community detection methods such as the Louvain algorithm allow us to
cluster the label graph. This is implemented in the
...
1
Editing my answer -
def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
"Train model using sklearn pipeline"
train_df = self.read_data(train_file, lower_case)
dev_df = self.read_data(dev_file, lower_case)
learner = self.pipeline.fit(train_df['text'], train_df['truth'])
# Fit the learner to ...
1
I understand that with "learning curve" you are referring to the plot of the loss function over the training data (or subsets of it) when optimizing with iterative methods, like gradient descent. This is sometimes referred to as the training loss plot.
You use a training loss plot during training, to evaluate the convergence of your training ...
1
So the question asks why you are seeing a decrease in the loss function (for both training and validation?), but you are also observing decreasing generalisation performance over iterations.
One first thought could be due to the loss function that you have chosen might not be appropriate for your task.
1
First thing that comes to my mind is to do different encodings. There are some ways to deal with high cardinality categorical data such as: Label Encoding or the famous target encoding. Before anything else I will recommend changing the encoding type.
But, since your question about which predictor use with small and space data. I will go still with logistic ...
1
The Python Library
Python-tesseract is an optical character recognition (OCR) tool for python. That is, it will recognize and “read” the text embedded in images. Python-tesseract is a wrapper for Google's Tesseract-OCR Engin
and please tell me where you find the medicine wrapper ( Packaging ) of different medicines.
1
The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same score of 1, and then it wouldn't make much sense to apply the nDCG penalty discounts.
A similar measure often used with binary relevance scores is the mean ...
1
A feature may be a predictor but it doesn't have to be. In Y = f(X), where Y is a predicted outcome, features are all available variables in X. So features are essentially input variables. Let's say you have a case where you have 10 features. It could be that only half of those have any predictive value. Throwing out the other features would not change the ...
1
This is a simplified explanation : Ground truth is a term used in statistics and machine learning that means checking the results of machine learning for accuracy against the real world. The term is borrowed from meteorology, where "ground truth" refers to information obtained on site.
How you get that ground truth : Many options but usually humans ...
Only top voted, non community-wiki answers of a minimum length are eligible
