Questions tagged [data-science-model]

The tag has no usage guidance.

157 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
2answers
106 views

What's a reasonable distribution to model views over time of… this question?

Sorry for the self-referential title ;) I'd be curious to know what's a curve that could be used to model - in the scale of (say) weeks* - the "engagement lifecycle" of a single news page of an ...
3
votes
1answer
63 views

Text classification based on n-grams and similarity

I have tried to cluster hundred texts using k-means clustering. I would like to consider other algorithms to group text based on their content and try to spot news not related to other news (topic ...
3
votes
1answer
41 views

How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
3
votes
1answer
155 views

How to correctly set a target for a time series based model?

I need help determining the best way to go about creating the target variable for a machine learning model that is trading a financial instrument (stocks, foreign currencies, crpyto, etc). Below is ...
2
votes
0answers
22 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
2
votes
3answers
61 views

Continuous VS Categorical variable

In the dataset I have a continuous variable AGE and categorical variable AGE_CATEGORY as well. They are both highly correlated. ...
2
votes
1answer
39 views

Negative loss, 100% accuracy

I have a very small dataset with 567 images. I've used pretrained model resnet50 in transfer learning. But whenever i fit my model it's giving me 100% accuracy , 100% validation accuracy and in per ...
2
votes
1answer
59 views

How to deal with annotation errors?

I know my annotators are not perfect, sometimes making mistakes. What would be the best way to deal with the annotation errors for my training data? Thanks!
2
votes
0answers
22 views

Step extraction from a paragraph

Came across an interesting problem: Given a paragraph describing how to do a process, need to break it down to various steps. Basically, need to determine for each sentence in the paragraph, if this ...
2
votes
0answers
31 views

How to cluster a set of objects each with its' own set of data?

I have a problem where I have a set of objects (say smartphones for ease of explanation). For each phone, we collected several characteristic features (such as duration of calls, how many calls made, ...
2
votes
0answers
21 views

Best practices for scaling data science / engineering teams

I am trying to find best practices for scaling data science teams, i.e find an efficient workflow/methodology to divide work between Software Engineers and Researchers working on a same product. I’...
2
votes
0answers
24 views

Natural language Generator using Data from table

I am working on some natural language generator part. eg.1 Input to it will be Col1 Col2 Col3 A B 13 X Y 14 Output should be two sentence, one ...
2
votes
1answer
69 views

In a residuals vs fitted plot, how do I interpret a homoscedastic variance that is not randomly distributed above/below the line?

I'm learning linear regression, and I ran a step function for linear regression and checked out the residuals vs fitted plot for the final equation. The residual looks homoscedastic but it's not ...
2
votes
0answers
81 views

detecting anomaly from cdr data

I need to find anomalies from this. The term anomaly in my case is in which square id and in which time the total activity goes higher, how can I detect anomalies with the help of machine learning ...
2
votes
0answers
118 views

Own Implementation of Neural Networks heavily under fitting the data

I tried to implement a Basic Deep Neural Network Algorithm for a classification problem on my own. I have tried on the iris data set for this test but, my implementation has been giving me very poor ...
2
votes
2answers
133 views

Predicting missing data. Looking for good data predicting technique

I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data ...
1
vote
0answers
11 views

Are there different types of confidence values?

Query related to apriori association in ML. I am learning apriori association from a teacher. I have a simple dataset of five transactions as follows. T1: {Milk, bread} T2: {Milk, bread} T3: {Milk, ...
1
vote
2answers
36 views

What is the purpose of the 'train model' step in data mining?

My understanding is that training a model is something done in machine learning using training data so that the model can predict values when new data is given to it. Data mining is the process to ...
1
vote
0answers
34 views

Advantages and disadvantages of using softmax/sigmoid and categorical_crossentropy/binary crossentropy for a binary classification with a CNN

I'm doing a deep learning model using tensorflow and keras. I have a question about the output architecture. I want to classify between two classes, images with defects and images without defects, I'...
1
vote
0answers
16 views

How to calculate cov(e|X) in Generalised linear regression?

This image taken from wiki https://en.wikipedia.org/wiki/Generalized_least_squares. I am not able to calculate Cov(e|X)? Can someone tell me that.And also give me dimesion of cov(E|X)= sigma^2 * rho
1
vote
0answers
42 views

Find periodicity of a signal using python

I have a dataset that contains occurrences of the Kettle in a single-occupancy house for the duration of a month. In this dataset, ‘ts’ column indicates the unix-timestamp (this can be converted to ...
1
vote
0answers
13 views

Correcting high AR(1) coefficients in dynamic Gordon model

I have just finished my thesis on a heterogeneous dividend expectations model applied to the COVID-19 crisis! However after receiving some feedback there is one last issue I want to resolve. I'm using ...
1
vote
0answers
27 views

How to extract embeddings of categorical variables

I am little bit confused about encoding categorical variables. There are other posts/blogposts on this issue but none is talking about the problem I am facing. I have a dataset with mixed variables (i....
1
vote
1answer
77 views

How do I predict a set of frequently bought items?

I have a dataset of retail transactions wherein different users buy certain items together. For example, a user A buys a toothpaste, a toothbrush and a floss at the same time, and a user B buys a ...
1
vote
1answer
31 views

what are the next step after ML prediction and how to proceed?

I have trained an ML model with a good accuracy but what next? I am facing difficulty in answering this question, how will you present your model. Which framework do you use How do you make sure ...
1
vote
2answers
60 views

Extract information using NLP and store it in csv file

I have a text file that stores the pickup, drops, and time. SMS text is a dummy file that is used to train a cab service model. The text is like in this format: ...
1
vote
0answers
55 views

How to group various similar search keywords and find top 100 keywords from big dataset

I have search keywords in one of my database table. These are the keywords searched by users on a website. My requirement is to find the top 100 search keywords after consolidating various similar ...
1
vote
0answers
22 views

Different Decision Tree pruning method

I am trying to learn different pruning methods for decision trees. I have put together a list of methods below. Reduced Error Pruning Cost Complexity pruning Minimum error pruning Pessimistic Error ...
1
vote
1answer
16 views

I Have Issues Installing Basemap

I tried to install Basemap and it gives me this: preparing transaction: done verifying transaction: done executing transaction: failed ERROR conda.core.link:_execute(507): An error occurred while ...
1
vote
1answer
22 views

Why gaussian assumption in GMM-HMM ASR?

I am reading a book entitled Speech and Language Processing by Daniel Jurafsky and James. For Acoustic vector, I would like to know why Gaussian assumption is made? I searched over the internet and ...
1
vote
1answer
48 views

Forecasting with a Machine Learning Algorithm

Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all ...
1
vote
0answers
33 views

K in the Naming of Models and Techniques

Why is k chosen for the names k-nearest neighbors and k-fold cross validation? Is it arbitrary or just another mysterious naming?
1
vote
0answers
19 views

Interpreting Adaboost model results

I'm trying to get a better grasp of model interpretability using many different kinds of models for a binary classification problem. Quick note: By interpretability in this case, what I mean is ...
1
vote
0answers
18 views

RESOURCE EXHAUSTED error with Kfold on a facial landmarks data

i am currently working with helen dataset to predict facial landmarks. Currently running the program on a Geforce 1070. The highest batch size i can give is 230 and i am getting a loss of 3 and ...
1
vote
1answer
32 views

How to use weights from our data set properly? Should we use them at all?

The census data set I'm using: https://archive.ics.uci.edu/ml/datasets/Adult So, I'm currently using this census data to make observations and predict whether someone is married or not. However, when ...
1
vote
0answers
38 views

Hypothesis Testing and Independent variables Importance

I am learning Data Science and I have a confusion in one topic. I would like to describe my approach. I have understood the problem statement. I have used snowflake schema to join tables that have ...
1
vote
0answers
29 views

types of metrics to evaluate the performance of regression models?

I have a conceptual question related to types of performance analysis of regression models. In general we have - R Square, Root Mean Squared Error and Mean squared Error to evaluate Regression ...
1
vote
1answer
4k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
1
vote
0answers
194 views

ValueError: Error when checking input: expected input_5 to have 4 dimensions, but got array with shape ()

I am trying to load a densenet-121 model pretrained on imagenet weights and train on my dataset. I have two files namely train.csv and validation.csv and I have splitted the train.csv file into 80-20 ...
1
vote
0answers
11 views

Soft Margin SVM kernels

Kernels are used to map datasets into higher dimensions so that they could be linearly separable. However, if we introduce the slack variable in the soft margin SVM, we are allowing some mistakes, and ...
1
vote
1answer
61 views

If the entire production dataset is known is advance, should I use it when fitting normalization parameters?

I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage. I presume this ...
1
vote
0answers
17 views

Prediction that a Employee can take a leave or not based on his previous track record

which prediction algorithm should I use to find the probability that an employee takes leave and come back on the specified period(i.e.does does not extend his leave). Thank You.
1
vote
0answers
23 views

What approach should I take to model forecasting problem in machine learning?

I have a dataset which contains 4000k rows and 6 columns. The goal is to predict travel time demand of a taxi. I have read many articles regarding how to approach the problem. So, every writer tell ...
1
vote
0answers
140 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
1
vote
0answers
55 views

Best model for Antimicrobial Resistance (AMR) prediction?

Some classes of problem are best solved by a specific class of machine learning model, due to the structure of the data (e.g. Deep Learning for computer vision). Prediction of bacterial resistance/...
1
vote
1answer
243 views

Column header in XTS and Data frame in R

I used quandl function extract stock data with object type as xts. ...
1
vote
0answers
115 views

Anomaly detection on text data using one Class SVM

I am working on an Anomaly detection problem that is first of its kind for me. I am able to do data cleansing and train the model however I have no idea on predicting results, as it is new for me and ...
1
vote
0answers
725 views

Passing variable length sentences to Tensorflow LSTM

I have a tensorflow LSTM model for predicting the sentiment. I build the model with the maximum sequence length 150. (Maximum number of words) While making predictions, i have written the code as ...
1
vote
1answer
316 views

checking model stability - Performance for different class

I tried to do multi-class classification problem. The goal is to predict whether the match will be won by HomeTeam, AwayTeam or Draw. I did feature engineering from the attributes and finally came up ...