Questions tagged [machine-learning]
Machine Learning is a subfield of computer science that draws on elements from algorithmic analysis, computational statistics, mathematics, optimization, etc. It is mainly concerned with the use of data to construct models that have high predictive/forecasting ability. Topics include modeling building, applications, theory, etc.
11,162
questions
0
votes
0
answers
6
views
How can i understand multiple lstm cells by unrolling?
I do not Unterstand the concept of multiple units in lstm.
If i have an lstm layer with 64 cells, how would be the cells applied to each time step by unrolling.
My understanding is that each time step ...
1
vote
0
answers
9
views
How to properly visualize high-dimensional embeddings along with the decision boundary in 2-D?
I have a number of embeddings (300-dimensional FastText vectors for each instance of each class) that I apply a classifier to (Logistic Regression for now). I want to visualize the embeddings as well ...
- 11
0
votes
0
answers
6
views
Exploring the Concept of Gradient Flow
Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
- 1
0
votes
2
answers
11
views
How to rank relatedness of two feature in dataset by their distribution?
Let's say we are given a dataset and want to rank them by similarity of distributions. I don't want to use visualization. Is there any sufficient way that you can share with me?
I have an idea like, ...
0
votes
0
answers
6
views
pytorch: implementing logistic regression: input dimension of torch.nn.Linear is input.flatten(start_dim=1)
I tried to implement a logistic regression class using pytorch. The following implementation worked.
...
- 1
2
votes
0
answers
70
views
+50
How to remove the hotspots from given image by using Python and opencv?
In the picture below there are some regions which are very bright (i.e. more white). Some bright regions are wide and some are narrow or thin. The red box covers one such wide bright spot, and blue ...
- 29
0
votes
1
answer
16
views
What exactly is Gradient norm?
I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
- 1
1
vote
1
answer
20
views
Different validation sets give very different results. What can be the reason?
I have ~78k microscopy images of single cells, where the task is to classify for cancer (binary classifier). The images are labeled according to which patient the data came from. I do the train-val ...
2
votes
2
answers
183
views
Some simple questions about confusion matrix and metrics in general
I will first tell you about the context then ask my questions.
The model detects hate speech and the training and testing datasets are imbalanced (NLP).
My questions:
Is this considered a good model?
...
- 89
0
votes
0
answers
13
views
Confusion regarding what constitutes a feature in a LSTM?
I have a Time Series problem, where I am trying to predict a single output at time $t$, $y_t$, given the $2$ previous time steps; $X_{t-2}, X_{t-1}$.
Let's just look at one observation for simplicity.
...
- 101
1
vote
1
answer
18
views
How do I exploit partial labels for classification?
How does one learn a classifier from data that isn't always fully labelled? For example, say one has corrupted data from the CIFAR-10 dataset (which has labels like bird/automobile/ship/truck). Now ...
- 11
0
votes
0
answers
22
views
What statistical test should I use to compare two random forest models, where each model has a different set of variables available? [closed]
What statistical test or methodology should I use for comparing two random forest models, where a different set of variables is made available for each model? I need a test where a power analysis can ...
- 111
0
votes
1
answer
19
views
Struggling with understanding RandomForest model with SMOTE
From what I understand my code is telling me that my base model is performing at 96% on it's training data, 55% on it's test data.
And my SMOTE model is performing at ~96% on both.
From my ...
0
votes
0
answers
14
views
Model returns near perfect PR-AUC score but other metrics seem fine. Is my model overfitting?
I am currently working on a very imbalanced dataset:
24 million transactions (rows of data)
30,000 fraudulent transactions (0.1% of total transactions)
The dataset is split via Year, into three sets ...
-2
votes
0
answers
17
views
how do to divide the datasets into 80/20 per cent of training and test sets and 10 foldcross-validation during model training to minimize bias [closed]
Classification: Diabetes Expert System
Sediba MedResearch is a research company that focuses on the study of diseases aiming to provide accurate expert systems that can help medical practitioners ...
0
votes
1
answer
21
views
Anomaly Detection: Large number of categories
Looking for some advice.
I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes.
...
- 1
1
vote
1
answer
20
views
Extract phrases/keywords that are SIMILAR to a python list of keyword/phrases, from a document
EDIT : If I had to match single worded phrases, I could first tokenize the text from the document and then calculate the cosine similarity of all the tokens with all the keywords from the ...
- 1,566
0
votes
0
answers
12
views
In recommendation systems, for methods that use backproporgation to get user feature, do they need to retrain the whole model when a new user is added
I'm tying to learn about recommendation systems recently. I have some deeplearning background so I focused more on machine learning based methods for recommendation systems. I see that a lot of paper ...
- 101
0
votes
1
answer
25
views
What are the approaches for extracting an injury and its description from a paragraph?
Suppose I have a paragraph which explains the injuries and its descriptions. I want to extract the injuries and its corresponding descriptions from the text. How can I do that?
For example, the ...
- 197
1
vote
0
answers
40
views
Questions about time series and overfitting and underfitting
When we want to compare between loss and val_loss, do we only care about the last epochs? Do the following charts show that the model is not overfitted nor underfitted? Can you please give me comments ...
- 89
1
vote
2
answers
37
views
High accuracy on test and validation data but still can not predict on real data
Hello i am having a classification between two classes A and B and i have trained CNN model. I have high accuracy on all three set of data i.e training (98.7%) validation (99.3%) and test(98%) but ...
- 11
0
votes
1
answer
47
views
How to intrepret low F1 score and high AUC on training set?
I am currently working on a very imbalanced dataset:
24 million transactions (rows of data)
30,000 fraudulent transactions
(0.1% of total transactions)
and I am using XGBoost as the model to predict ...
0
votes
1
answer
20
views
What is the minimum ratio to consider the data set as balanced for the classification algorithm?
Can I consider 20% / 80% as a balanced dataset?
My target variable ratio is 62% and 37%. I hope It is a balanced dataset. Please let me know if I am wrong.
However, I would like to know, what is the ...
1
vote
0
answers
29
views
Can you do a power analysis to determine the sample size for a virtual species simulation which is modeled using random forests?
I am simulating virtual species on a 100x100 grid (the size for now). Each grid layer represents one environmental variable. The "suitability function" defines the probability of a presence ...
- 111
1
vote
2
answers
32
views
1
vote
0
answers
9
views
confusion with Xavier Initiliazation definition
When researching online, I keep finding that Xavier/Glorot initialization is:
however, the original paper by Glorot said that this was a common initialization strategy that they soon found did not ...
- 11
1
vote
0
answers
32
views
Convolution neural network loss increasing instead of decreasing
I am working on a binary image classification task in which I have greyscale images of size (1, 224, 224) (all normalized between 0 and 1) and a set of labels (0 or 1). I have around 2.6k images with ...
0
votes
0
answers
9
views
Validation Loss not decreasing for RESNet model
I have been trying to train a Resnet model to classify Diabetic Retinopathy images into binary classes. The dataset consists of around 35k images. The val loss and accuracy does seem to behave weirdly ...
1
vote
2
answers
56
views
Looking for a couple of ideas please
I’ve got some data by postal zone that includes:
Postal zone code
Average rent value per square foot
Brand affinity 1
Brand affinity 2
Brand affinity 3
Brand affinity 4
…and so on
The brand affinity ...
- 11
0
votes
1
answer
32
views
corr() is giving an error. please help out of this problem and tell me what is this error about
when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
0
votes
0
answers
11
views
what happens when weghts in perceptron algorithm is first initialized with some random values which is very distant from the correct values?
take this example of a small dataset
so here there was a question that instead of initializing weight vector as zeroes what if we initialize to [1000 , -1000] (there is no offset i.e classifiers ...
0
votes
1
answer
21
views
How complex can I make a classifier's loss function in Scikit-Learn?
I want to customize the loss vanilla loss function being used by scikit-learn classifiers like the Logistic Regression classifier, etc.
For example, if the vanilla empirical risk minimization ...
1
vote
1
answer
34
views
Best classifier for ordinal categorical variables
I'm getting different results between R and Python for a classification problem using ordinal categorical features.
I would like to ask you what do you think is the best classification algorithm in ...
- 111
1
vote
1
answer
46
views
What should I Improve from my Neural Network Model (Logistic Regression)
Initial Information
I built a Neural Network Model (Logistic Regression) to classify Lung Cancer based on the patient's (user) symptoms
My dataset is kind of small (only about 276 data)
Here is the ...
- 11
0
votes
0
answers
19
views
LSTM can accept inputs of different shapes in some cases
I thought that with an LSTM you could use sequences of any length as input, but with shape fixed for each time step, but I encountered an anomalous behavior.
The following code gives the error that I ...
- 111
0
votes
2
answers
37
views
Where can I find the applied data science research papers?
I'm trying to find conferences that have applied data science papers published. I'm only interested in top ranked conferences. And I notice quite a number of them are quite theoretical, e.g. IJAI, ...
- 399
2
votes
2
answers
71
views
Random Forest Classification model performing much better with 70:30 TEST:TRAIN rather than the opposite
I'm working on a Classification problem as a side project and I'm receiving results contrary to what I'd expect.
With 100,000 records, each with 7 components for X, the model is performing much better ...
1
vote
0
answers
25
views
MobileNet validation loss not decreasing over time
I am trying to train a MobileNetV2 on a custom dataset, to image Classification task.
Cardinality is 864 images, split in 70%/20%/10%, balanced between the 3 different classes.
Weights are pre-loaded ...
- 11
0
votes
1
answer
31
views
Does a classifier based on optimal bayes classifier equation classify every new instance the same way?
I'm trying to understand how optimal bayes classifier works and I was wondering if, given that the function we try to maximize when making a new prediction does not depend on the instance we are ...
0
votes
0
answers
13
views
Embedding vector of MaskRCNN (Resnet with FPN)
I have a MaskRCNN model for instance segmentation with Resnet 50 - FPN backbone trained in detectron2. And I want to extract the embedding/feature vectors for visualizing input and hopefully detecting ...
0
votes
0
answers
10
views
Can the addition of a non-support vector change the SVM solution?
If I understand the math behind the classic SVM for non-separable data correctly, the addition of a non-support vector (non-SV) should theoretically not alter the solution. My reasoning is that since ...
0
votes
1
answer
22
views
Do I need to standardize time series data in change point detection?
I have process data in time series data(0min, 1min, ... 999min). I don't know what does the variables mean. They are just written in X1, X2, ... X52. Each row means the data at the time. At certain ...
- 1
0
votes
0
answers
23
views
dolly 2.3b machine laerning using trainer.train
first time poster here so please forgive me and correct me on my posting mistakes...
Im trying to teach the databricks/dolly-v2-3b llm, some data, which is just one sentence.
In the future I would ...
- 1
0
votes
1
answer
23
views
How to make an RNN model in PyTorch that has a custom hidden layer(s) and that is compatible with PackedSequence
I want to make an RNN that has for example more hidden layers or layer normalization.
I know that is it possible to make a custom RNN by subclassing nn.module, but with this approach is it not ...
1
vote
2
answers
104
views
what qualifies as a data leakage?
I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
- 2,507
0
votes
1
answer
14
views
How do I use a column with data of different layers for AI?
I am working with real estate data for an ML/DL project. In the csv file there is a column in which each cell contains data like the examples below:
...
0
votes
0
answers
15
views
FTT Features to use after time-domain is transformed to frequency-domain
Please forgive the question if it sounds trivial/naive, I am from computer science background, not electrical/computer engineering.
I work with GPS trajectory dataset for classification. Data was ...
- 101
0
votes
2
answers
38
views
can training for too long lead to overfitting? I am not sure about the specifics of this
does training for a large number of epochs lead to overfitting? I am concerned about this as I am getting an accuracy of nearly 1 on val and training dataset when I am training for 50 epochs
4
votes
1
answer
92
views
Are imbalanced data problems solvable? [closed]
I am working as a data scientist for the past 2 years where I have worked on problems related to binary classification, revenue prediction etc.
In the past two years, I have had 2 problems that ...
- 2,507
0
votes
0
answers
7
views
Image classification approach for float outputs
I have an image input and the model should be able to predict its 15 feature values as output. I am being told that i should use an image classification model to solve this. can somebody suggest me a ...
- 1