Questions tagged [model-selection]

Model selection is the process of comparing several models and their respective results to choose the model is best according to some evaluation metric.

Filter by
Sorted by
Tagged with
0
votes
1answer
12 views

svm.LinearSVC: larger max_iter number doesn't always increase the accuracy/precision/recall

Background: Supervised machine learning Data shape 10+ features, target = 1 or 0 only, 100,000+ samples (so should be no issue of over-sampling) 80% training, 20% testing train_test_split(X_train, ...
0
votes
0answers
12 views

Model or algorithm for iterative optimization

Here is my problem : At every loop, I have new data that depends on the previous outputs. I need to approximate the function that optimizes (minimizes / maximizes) this new data on every iteration. ...
2
votes
2answers
68 views

What regression model can handle tiny amounts of data?

I'm trying to use machine learning to predict properties of a material during a crash test, but each data point requires physically crashing an expensive toy car, so I can only gather a few hundred ...
1
vote
1answer
20 views

Modeling strategy for predicting a day/hour based on my dataset

This is my first time posting here. I'm usually on SO. So I'm not sure if these kind of questions fit into DS stackexchange. I genuinely need opinions on this. What data do I have - ...
4
votes
1answer
27 views

help understanding nested cross validation

From what I read online, nested CV works as follows: I divide my whole data in k folds, where k-1 folds are the train set and one fold is the test set. There is the inner CV loop, where we may ...
0
votes
0answers
8 views

Do pseudo r squared metrics make sense for classifiers that aren't logistic regression?

I'm working with some domain scientists that are used to using logistic regression to predict a binary value. One of the ways they evaluate their logistic regression model is through the Nagelkerke $...
-1
votes
2answers
25 views

How to approach a problem when the information is in the relationship between the points, and not the points itself?

I am trying to analyze vehicular mobility models, where I am trying to learn how a particular vehicle moves and then detect similar patterns from the testing data. Here's what I have done for now: I ...
0
votes
0answers
18 views

Predicting house pricing given a dataset consisting of [ location: date of transaction: price ]

What would be the right way to tackle the problem of predicting median house pricing, given that the data I have for training consists in a big list of entries that have the following values: ...
2
votes
0answers
24 views

How to measure the stability of hyperparameter selection in a model-building procedure?

For my project I run several model-building-procedures. I use the mean and standard deviation of the test scores in the outer folds as an estimator for the generalizability of the model-building ...
3
votes
1answer
61 views

How to handle associated features in machine learning

I am working on a classification project in which some features are linked and I'm not sure how to handle them. I will simplify my project like that : There are different jobs, and multiple ...
0
votes
1answer
18 views

What model is recommended: I am using text features in a regression and want to interpret coefficients

I am using the text of comments on a forum to predict how many upvotes it will get. I want to be able to say, "Reviews with X, Y, Z words are more upvoted". So to do this, I want to use text features ...
0
votes
1answer
59 views

Building document classifier based on keywords, what would be the steps?

I have a requirement of classifying documents(.doc files) based on the profiles. I have a csv file with data: ...
0
votes
1answer
24 views

I have hourly data of a metric for 15 days, Can i predict the outcome values for same metric for the next 15 days?

I have tried a linear regression model for the same data, Since the regression line is continuous i'm not sure if it works to predict the outcome values for next 15 days, or for a given period of time!...
1
vote
2answers
74 views

What arguments should I pass to input_shape parameter of LSTM function in Keras?

My dataset has 2944424 rows and 6 columns. I am using an LSTM in Keras to forecast taxi demand. I am having problem with the input_shape parameter of the LSTM. It ...
1
vote
0answers
29 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
5
votes
3answers
161 views

Why my network needs so many epochs to learn?

I'm working on a relation classification task for natural language processing and I have some questions about the learning process. I implemented a convolutional neural network using PyTorch, and I'm ...
0
votes
0answers
25 views

Model Selection

I have employed a variety of ML algorithms using 10-fold cross validation using the caret package in R on my data set. Can I employ ANOVA test on their f measures or Auc's to see if there is any ...
0
votes
0answers
16 views

Choosing predictive model for dataset

have a question about the type of model which I should use for a dataset I have. I have use 2 data-sets for my project. After hypothesis testing, I me Out of the 7 input variables, 6 of them are ...
0
votes
1answer
16 views

About 1M rows of data. Should I restrict myself to few columns as well?

I'm trying to build a predictive model from about 1 million rows of data. My goal is to predict a certain numerical value. I have the intuition that I should use very few numerical binary columns so ...
1
vote
2answers
26 views

What should I check if model accuracy is no better than baseline level(random guess)

I have a data with only 8 columns: id created_time employee_id rank position hourly price num_work_completed work_category hired Hired is the target variable with 1 representing hired and 0 ...
2
votes
2answers
95 views

Interpretation of ROC AUC score

i tried to evaluate 6 models and after plotting , this what i get : So i'm wondering , if those results are "Right" ? Thank's in advance.
0
votes
0answers
19 views

Cross validation for periodic signal in time series

I have a dataset of 90 periodic signals (current signals). These signals are divided into two big areas: Fault-free area and Fault area. I sliced the signals using a window of 80ms with an overlap of ...
0
votes
0answers
12 views

Many smaller city models or one country model?

I have a model selection question. I have models that predicts house prices. At Country level (with all the datapoints) the winner is a RandomForest with rsme 0.22 For a city level with many ...
0
votes
0answers
16 views

Predicting U.S. suicide count based on a set of inputs

I'm trying to design a model (or multiple) that can predict the number of U.S. suicides for a future year, based on a few inputs--"age", "sex", "population" (of the age/sex), and "gdp_per_year". I'm ...
0
votes
0answers
13 views

How do business policy rules/overrides work in large, production ML systems (esp. credit scoring)

so my startup has gotten to the stage where we are doing couple of 100k dollars per month. However our ML based credit scoring has become a jumble of a few hundred business policy rules with a ML ...
0
votes
0answers
67 views

Places365 for pytorch

I'm trying to use Places365 (the Vgg implementation) in PyTorch. I downloaded the model and the weights from the repo. The Vgg16 version of Places365 found in the official Github repo contains a ...
0
votes
1answer
22 views

How can I get an objective measure of song similarity?

I was browsing through ML project ideas and found an interesting one (just the problem statement ) which was: detecting if two songs are similar using lyrics. I found it to be an interesting idea but ...
0
votes
0answers
5 views

Will the combination of Chaid/CART tree modelling improve the accuracy of the Decision Tree Regression Model?

Will the performance of the decision tree regression model significanlty improve if we consider CHAID modelling first by identifying the key continous/categorical dependent variables and then builidng ...
0
votes
0answers
11 views

Is it possible to calculate precision/recall for a bag-of-words model?

Suppose I have a list of keywords given to a document: {keyword, extract, graph, represent, text, weight, number, document} and then I have the keywords ...
0
votes
0answers
16 views

Optimal architecture of neural networks for classification of samples with both text features and other features

Question: What is (from your experience) the most optimal architecture for a neural network for binary classification when the feature space is a mix of text and contextual features? Background: The ...
0
votes
1answer
31 views

Having averaged trials which are less than the number of features

Suppose I have an experiment where I have 70 features and 48 samples. The target variable is binary (0,1) and the 48 samples are divided such that 24 of them correspond to outcome 1 and the other 24 ...
2
votes
0answers
40 views

Test RMSE of polynomial regression drops when using more variables?

I am testing polynomial regression for a data set of 50 variables and a sample size of 5000. I ordered the coefficients of the linear model from high to low and then made different models using the p ...
0
votes
0answers
24 views

Best model for Antimicrobial Resistance (AMR) prediction?

Some classes of problem are best solved by a specific class of machine learning model, due to the structure of the data (e.g. Deep Learning for computer vision). Prediction of bacterial resistance/...
1
vote
1answer
45 views

Probabilistic Machine Learning model to match spatial data

I have spatial data from multiple sources. This data consists of ID, lat, long, and time. My goal is that given a new lat-long, the model needs to return (preferably with a probability) the data ...
1
vote
0answers
55 views

I am getting p value 0 for my population I teven tried zscore for normalization

a1 is the columncolumn a1 g=scipy.stats.mstats.zscore(a1) g stat, p = normaltest(g) print "%.3f, %.9f" %(stat ,p) Result 8.570, 0.00000000 p1=scipy.stats.kstest(g,'norm') p1 KstestResult(...
2
votes
1answer
45 views

How to do vocabulary estimation based on observed writings?

Below is a scatter plot of the data set I am dealing with. The X axis is the total number of words per essay for a particular individual, and they Y axis is the number of unique words. In principle, ...
0
votes
0answers
100 views

Using logLoss as metric function for highly unbalanced dataset

ihave an highly unbalanced dataset and the caret pacjage only allows me to select accuracy or kappa as performance metric. Is it correct to use a mlogloss function to compute model performance? Do you ...
2
votes
1answer
42 views

Model selection: large mean and variance vs small mean and variance

This question was always in my mind. Imagine you are doing 5-10 fold cross validation and one model gives you mean accuracy of 0.8, but with 0.2 standard deviation and the other one gives 0.7 with 0....
2
votes
2answers
104 views

How to Work with Imbalanced Data

I am building a binary classifier from a set of feature vectors some of which are categorical like Yes or No (two options). I am replacing them with 1 and 0 and since there is strong imbalance between ...
2
votes
1answer
43 views

What metrics determine the quality of the model?

Working on this Kaggle competition, and have some questions. Using this code: ...
2
votes
2answers
138 views

Where does the “deep learning needs big data” rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
0
votes
0answers
57 views

Hyperparameter optimization when calculating learning curves

I'm selecting a model for a regression problem and want to calculate learning curves. My dataset consists of ~20,000 x-y pairs. I'm using kernel ridge regression with different kernels, different ...
2
votes
1answer
47 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
0
votes
1answer
20 views

Tests to find which theory agrees better with observation

I have three curves ( 1.> observation: yobs , 2.> theory-1: yth1 , 3.> theory-2: yth2 ). All of these curves are functions of a single variable (say variable x.) From a computational perspective, all ...
1
vote
1answer
60 views

checking model stability - Performance for different class

I tried to do multi-class classification problem. The goal is to predict whether the match will be won by HomeTeam, AwayTeam or Draw. I did feature engineering from the attributes and finally came up ...
0
votes
1answer
40 views

Why some ML models can't take advantage of text ordering information?

On this google tutorial (https://developers.google.com/machine-learning/guides/text-classification/step-4) it is said: > Build n-gram model [Option A] We refer to models that process the tokens ...
3
votes
2answers
465 views

LSTM vs ARIMA for demand prediction

I'm new to the field of time series prediction. I'm looking for a demand prediction model to predict when the product will be sold out from the online supermarket (when the supply is known in advance)...
1
vote
2answers
141 views

How do you pronounce ROC?

In regards to ROC curves, how do you pronounce ROC? I have always spelled out the letters like R-O-C but I sat through a sales demo today where they guy pronounced it as a word like "rock" as in "the ...
0
votes
0answers
16 views

How to attribute variance to an input parameter?

Some data Maybe this is easiest to explain by going straight with the data. Here is how much money Bob has at the end of each day. ...
6
votes
4answers
540 views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...