Questions tagged [data-science-model]

The tag has no usage guidance.

175 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
2answers
81 views

What to do when seed has a big impact on model performance?

I have a training procedure set up for an image recognition task. Each time I train a model, I record training loss, validation loss, validation precision and validation recall. Recently I switched ...
4
votes
1answer
596 views

Text classification based on n-grams and similarity

I have tried to cluster hundred texts using k-means clustering. I would like to consider other algorithms to group text based on their content and try to spot news not related to other news (topic ...
4
votes
1answer
115 views

How to deal with annotation errors?

I know my annotators are not perfect, sometimes making mistakes. What would be the best way to deal with the annotation errors for my training data?
4
votes
1answer
108 views

In a residuals vs fitted plot, how do I interpret a homoscedastic variance that is not randomly distributed above/below the line?

I'm learning linear regression, and I ran a step function for linear regression and checked out the residuals vs fitted plot for the final equation. The residual looks homoscedastic but it's not ...
4
votes
2answers
117 views

What's a reasonable distribution to model views over time of... this question?

Sorry for the self-referential title ;) I'd be curious to know what's a curve that could be used to model - in the scale of (say) weeks* - the "engagement lifecycle" of a single news page of an ...
3
votes
1answer
249 views

Two steps optimization of a credit card limit

I have a problem similar to what is on the title but not the same. The problem on the title allows me to explain the dynamics of my need. I have to determine what the optimal value is for a variable ...
3
votes
1answer
46 views

How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
3
votes
2answers
143 views

How do you use KS-test in a data science report?

I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result , but when I use KS-test to test the model, I got a low p-value,1.2e-4, ...
2
votes
1answer
17 views

My own model trained on the full data is better than the best_estimator I get from GridSearchCV with refit=True?

I am using an XGBoost model to classify some data. I have cv splits (train, val) and a separate test set that I never use until the end. I have used GridSearchCV to determine the best parameters and ...
2
votes
1answer
46 views

Highly correlated features

What are the conditions to remove highly correlated features in a correlation map? Given the correlation map below, is it OK to remove diagnosis feature and should we remove highly correlated features ...
2
votes
1answer
39 views

How can I reduce the number of dimensions using a Clustering algorithm in a mixed dataset?

I am working with a mixed data set, corresponding to TV consumption data, with the aim of reducing the number of features to only those relevant to detect TV consumption patterns (or consumption ...
2
votes
0answers
46 views

Creating a new feature from an existing one using decision trees

Is it possible to create a new feature out of two, or more than two existing features using a decision tree? If so, how, and can it produce features with good information value that can better help ...
2
votes
0answers
91 views

flexibility vs complexity vs number of predictors in machine learning

I'm new to machine learning so am quite confused with the above concepts. It seems to me both flexibility and complexity measures how well the model fit the data (in terms of the curvy-ness), so what'...
2
votes
0answers
45 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
2
votes
0answers
619 views

Advantages and disadvantages of using softmax/sigmoid and categorical_crossentropy/binary crossentropy for a binary classification with a CNN

I'm doing a deep learning model using tensorflow and keras. I have a question about the output architecture. I want to classify between two classes, images with defects and images without defects, I'...
2
votes
1answer
50 views

what are the next step after ML prediction and how to proceed?

I have trained an ML model with a good accuracy but what next? I am facing difficulty in answering this question, how will you present your model. Which framework do you use How do you make sure ...
2
votes
0answers
23 views

Step extraction from a paragraph

Came across an interesting problem: Given a paragraph describing how to do a process, need to break it down to various steps. Basically, need to determine for each sentence in the paragraph, if this ...
2
votes
1answer
39 views

How to cluster over time?

I have a problem where I have a set of objects (say smartphones for ease of explanation). For each phone, we collected several characteristic features (such as duration of calls, how many calls made, ...
2
votes
0answers
24 views

Best practices for scaling data science / engineering teams

I am trying to find best practices for scaling data science teams, i.e find an efficient workflow/methodology to divide work between Software Engineers and Researchers working on a same product. I’...
2
votes
0answers
26 views

Natural language Generator using Data from table

I am working on some natural language generator part. eg.1 Input to it will be Col1 Col2 Col3 A B 13 X Y 14 Output should be two sentence, one ...
2
votes
0answers
90 views

detecting anomaly from cdr data

I need to find anomalies from this. The term anomaly in my case is in which square id and in which time the total activity goes higher, how can I detect anomalies with the help of machine learning ...
2
votes
0answers
139 views

Own Implementation of Neural Networks heavily under fitting the data

I tried to implement a Basic Deep Neural Network Algorithm for a classification problem on my own. I have tried on the iris data set for this test but, my implementation has been giving me very poor ...
2
votes
2answers
146 views

Predicting missing data. Looking for good data predicting technique

I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data ...
1
vote
1answer
11 views

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Logistic regression is supposed to work well only on data that is linearly separable. As we can see in the pair plot, the data points heavily overlap. The logistic regression model is in fact showing ...
1
vote
0answers
8 views

Identifying the best group that are most productive

can you please give some idea to solve this problem, Below is my data, I want to identify the best inventory based on the CPM(target value) .. For example: I want to be able to say - Business section, ...
1
vote
1answer
24 views

Normalization of production data

When training a model we split the dataset into training set and test set. In case a normalization/standardization is needed on any column then this process is done separately for training set and ...
1
vote
1answer
24 views

Which Data Science book gives complete guideline to data science?

There are many books in the stores on data science. Is there any book that gives complete guideline to data science? Like separately talks about python (not R), statistics, math, data Visualization ...
1
vote
0answers
26 views

How to compare two one hot encoded data frames based on column names?

I have two datasets with shapes (329, 159) and (26,24). Both of them are one-hot encoded. The columns in the smaller dataset are present in the larger dataset. The smaller dataset has scores that I ...
1
vote
0answers
16 views

Iteratively improving ML model on a small dataset

I have a spam classification model which I created using a very small dataset.I have exported it as shown ...
1
vote
0answers
14 views

Assigning Weight to MultiClass Variable

I am working on an academic project where I go through the bug database and classify the bug (a Multiclass Classification). After this classification, I would like to assign a weight to assign a ...
1
vote
2answers
46 views

Keras very low accuracy, saturate after few epochs while training

I am very new to the data science domain and directly jumped to TensorFlow models. I've worked on examples provided on the website before. My first time doing any project using it. I am building a ...
1
vote
0answers
40 views

why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies

Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only ...
1
vote
1answer
45 views

which Model to apply on panel data where unique id has 6-8 records and total records are 2,000,000?

I am new to such panel data where I have multiple observation for same ID in different Quarter and I am not sure what kind of machine learning algorithm I can apply. I have data from Q1-18 till Q4-...
1
vote
0answers
20 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
1
vote
0answers
40 views

How does stacking help Bias and Variance?

How does stacking help in terms of bias and variance? I have a hunch that stacking can help reduce bias but i am not sure, could someone refer to a paper?
1
vote
0answers
26 views

Calculating the lower and upper bounds forVC-dimension of a decision tree

I have a problem finding the lower and upper bounds of the decision tree. Suppose there is a decision tree with a hypothesis space of depth 2 and an input space with 10 variables (the variables take ...
1
vote
0answers
12 views

Parameter estimation using convolutional for exponential functions

I was inspired by this paper where the authors fed an 1-dimenisonal input waveform to the simple convolutional neural network and got an estimate of four parameters of the analytic model. This ...
1
vote
0answers
16 views

Rank advertisement and Score keywords used in advertisement based on its performance

I am trying to classify my advertisements and trying to understand which advertisement is performing better than others and why. Step - 1: compare two ads based on CTR. The problem here is I am ...
1
vote
1answer
141 views

Model retraining

I trained my model with RandomForestRegressor(), but now my training data is updated continuously. So I have to train my model with all the train data set i.e past ...
1
vote
0answers
38 views

How can I make a better unsupervised text classifier model? Is POS tagging part of Machine Learning and Data science?

I have got complaints data, which is not good, and often contains less than 3 words in every complaint (sometimes so short that only one word of them makes sense). The objective is to find what's ...
1
vote
0answers
18 views

Correcting high AR(1) coefficients in dynamic Gordon model

I have just finished my thesis on a heterogeneous dividend expectations model applied to the COVID-19 crisis! However after receiving some feedback there is one last issue I want to resolve. I'm using ...
1
vote
1answer
179 views

How to extract embeddings of categorical variables

I am little bit confused about encoding categorical variables. There are other posts/blogposts on this issue but none is talking about the problem I am facing. I have a dataset with mixed variables (i....
1
vote
0answers
40 views

How to train a recommender system to improve the customer class?

Considering the definition of a recommending system A recommender system, is a subclass of information filtering system that seeks to predict the "rating" or "preference" a user ...
1
vote
0answers
14 views

Space partitioning to estimate product weight

Looking for some vocabulary to help me refine my research so I can tackle this problem. Here's an overview of the problem statement I'm working on. At my company we manufacture various products, ...
1
vote
0answers
61 views

How to group various similar search keywords and find top 100 keywords from big dataset

I have search keywords in one of my database table. These are the keywords searched by users on a website. My requirement is to find the top 100 search keywords after consolidating various similar ...
1
vote
0answers
37 views

Different Decision Tree pruning method

I am trying to learn different pruning methods for decision trees. I have put together a list of methods below. Reduced Error Pruning Cost Complexity pruning Minimum error pruning Pessimistic Error ...
1
vote
1answer
25 views

I Have Issues Installing Basemap

I tried to install Basemap and it gives me this: preparing transaction: done verifying transaction: done executing transaction: failed ERROR conda.core.link:_execute(507): An error occurred while ...
1
vote
0answers
29 views

Is this the case of data leakage?

Let's say, I have the dataframe with numerical features A, B, C. I do not have the target variable but I extract the target variable Y from the features A, B, and C. E.g. ...
1
vote
0answers
21 views

KS-Score Methods

I came across 2 methods to calculate KS-Score and select best probability threshold. Decible Method TPR - FPR Is there any specific scenario on which it depends which method to select. or we cna ...