Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

Methods and principles of building "computer systems that automatically improve with experience."

0
votes
0answers
4 views

Is it possible that I conduct a supervised machine learning like random forest after creating a target variable through clustering?

I create 4 clusters by using k-mean cluster analysis. After that, I try to classify this target variable with random forest in order to confirm if the clusters are properly grouped. Is it possible?
0
votes
1answer
7 views

Both train and test scores high. Could my model be overfitting?

all. Currently I am working in a regressor to predict the median house value using 1990 census as a dataset (the same on Aurélien Géron used on his book). My best result so far has been a score of ~0....
0
votes
1answer
12 views

About 1M rows of data. Should I restrict myself to few columns as well?

I'm trying to build a predictive model from about 1 million rows of data. My goal is to predict a certain numerical value. I have the intuition that I should use very few numerical binary columns so ...
0
votes
1answer
14 views

Reducing MAE or RMSE of linear regression

I'm trying to guess a home price, at final I intend to figure out a formula by using linear regression. As you can see over the url, I have 1480 data with 45 features in which ...
0
votes
1answer
11 views
2
votes
3answers
65 views

Cross validation Vs. Train Validate Test

I have a doubt regarding the cross validation approach and train-validation-test approach. I was told that I can split a dataset into 3 parts: Train: we train the model. Validation: we validate and ...
1
vote
2answers
12 views
0
votes
1answer
9 views

Feature Normalisation for Grayscale Matrix [on hold]

Is it required to do feature normalization in the grayscale matrix values from 0 to 256?
0
votes
1answer
16 views

Keras MLP Classifer Model not converging

I'm trying to make MLP based classifier based on numerical and categorical data The train_X (Input) data that I'm working with is look like that each data type is 20 numerical and 1 categorical ...
0
votes
1answer
22 views

K-Fold and Random Subsampling (RSS) Dataset generation?

Let say if I have a large dataset of 300k instances with 200 features, I want to reduce its size. Can I apply K-Fold technique to the 200 features then the trimmed dataset are applied with RSS to trim ...
2
votes
1answer
24 views

Transfer learning for a regression problem

if my understanding is correct, in case of image classification and NLP, if I have a pre-trained model, to train on new data, I can reshape the data according to the pre-trained model. So there is no ...
0
votes
0answers
16 views

Recommendation on deep learning hardware for research [on hold]

I want to build a budget PC for deep learning research for trying new architectures and benchmarking for both CNNs and LSTMs. I have a few options on the components and I would like to hear your ...
0
votes
1answer
17 views

Why are parameter updates downscaled by uncentered variance (instead of centered variance) in Adam optimizer?

In Adam optimizer algorithm, parameter updates are computed as follows: $\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$ Where $\hat{m}_t$ is a bias-corrected ...
1
vote
1answer
13 views

Automatically Remove highly correlated features using Variance Inflation Factor?

I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated. Multiple features can be ...
1
vote
2answers
19 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
0
votes
0answers
9 views

What research exists to help design training data sets where contamination due to privacy is an issue?

Suppose one is building a classifier that: takes as input the e-mail body text returns true or false if person X should be included in the address list To build this classifier we have historical ...
0
votes
0answers
15 views

Comparing word frequencies between two corpora to find keywords

I have a set of tweets that are either labelled as ironic or non-ironic, and have separated them into two different corpora: ironic and non-ironic. I want to compare these two to see which words are ...
0
votes
0answers
10 views

best approach for CNN training with multiple subcategories and one category

I need to classify pictures into 2 categories: approved and rejected. Rejected category has different type of images which are not allowed (subcategories), for example nude or gore or anime etc. What ...
0
votes
0answers
7 views

Prediction Algorithm for Lithium ions batteries whether battery need maintenance or not also need to calculate health percentage [on hold]

I am new in ML, I have datasets for a particular device like battCapacity: Capacity of battery, battVolDc: Battery Voltage, battSoc: State of Charge, battRunHrs:Battery running hours, ...
1
vote
0answers
24 views

Comparing multi-class results with binary classification results

We used machine learning to discriminate the following five disease classes: Normal (N), Myocardial Infarction (MI), Coronary Artery Disease (CAD), and Congestive Heart Failure (CHF). In the past, ...
0
votes
0answers
10 views

Random Forest Classifier: Find the decision path for a single data point

I am working with RandomForestClassifier and I would like to be able to analyse the decision path which each decision tree takes for a single data point. What I understand is that the final prediction ...
0
votes
0answers
16 views

How best to find patterns in data that exhibit one-to-many and many-to-one relations?

I have a lot of log data related to alerts on a computer platform. Each log entry represents an alert, but several of the same alert can be fired for a single event over a span of time (either because ...
1
vote
3answers
43 views

K means visualisation after reducing dimensionality with PCA

In clustering ($K$ means, for example) when I have $N$ features and after creating the model (with this $N$ features) to visualize this model I need to reduce this $N$ dimensions into $2$ or $3$ ...
1
vote
1answer
27 views

How to create a system to detect text structure of a file?

Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
1answer
31 views

Binary Encoding of Ordinal Categories

I have a data frame in which one of my columns is the target value and there are lots of ordinal categories in columns of data frame. I want to encode these ordinal categories in columns with this ...
0
votes
0answers
14 views

Predict vs. Impute: Filling missing data using Random Forest

I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals ...
1
vote
1answer
20 views

How to Use Shap Kernal Explainer with Pipeline models?

I've a pandas DataFrame X. I needed to find prediction explanation of a a particular model. My model is given below: ...
0
votes
0answers
23 views

Effect Size in comparison of overall accuracy from Random Forest

I would like to compare two overall accuracy statistics (of a Random Forest classifier). My Data: two samples with each containing 25 features and one categorical class variables (9 different ...
0
votes
1answer
23 views

Random Forest application with 40+ Predictor Variables

I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals ...
0
votes
0answers
11 views

How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)

I came through this questions and I failed to find the right answer for it. How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)?...
1
vote
1answer
37 views

Is there a way (like planning algorithms) to draw a successive polyline to fill a specific shape (like triangle)?

there is a specific shape (like triangle) area, i am trying to develop an program to draw a successive polyline inside the triangle to fill the triangle, one line 3 layers this post demonstrates some ...
2
votes
0answers
25 views

Neural networks (keras): predicting a periodic output array

I have a non-linear multiple regression problem where my target arrays have a length of 256 (for a single sample). These arrays have a periodic structure, since it's actually composed of 16 semi-...
1
vote
0answers
17 views

Custom keras metric error

I'm trying to create a new metric for my model and this is my function ...
1
vote
0answers
19 views

Email clustering using numpy and pandas

Im trying to cluster my raw email data using numpy and pandas only. Hence Im Trying to create an email dataframe from raw emails that I have. However the issue that I have is that I keep getting an ...
0
votes
1answer
19 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
1
vote
2answers
15 views

What should I check if model accuracy is no better than baseline level(random guess)

I have a data with only 8 columns: id created_time employee_id rank position hourly price num_work_completed work_category hired Hired is the target variable with 1 representing hired and 0 ...
0
votes
1answer
19 views

Using sigmoid in binary DNN output layer instead of softmax?

For a binary DNN, the output is $y_0 + y_1 = 1$ since they are the probability distribution, hence the sum must equate to 1. However, I've been told that $y_1$ is sufficient to represent the output of ...
0
votes
1answer
10 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier which I would like to evaluate the performance of. It's both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while it's ...
1
vote
1answer
45 views

Predicting house price using linear regression

I'm trying to predict a house price using linear regression method. I gather the real data from a real estate website. I have some features and two numerical value in which the price is the target ...
0
votes
0answers
18 views

Input of the decoder of a LSTM seq2seq neural network?

I have been trying to setup a seq2seq network with LSTM layers where the output of the network should be the seven day ahead forecast, given the previous fourteen days as input. I started with a ...
0
votes
1answer
26 views

Understanding softmax layer in Deep Neural Network

In a DNN, once the logits vector is produced say $[y_0,y_1]$ where the number of neuron in the logits layer is 2, the condition holds where $y_0 >= 0$ and $y_1<= 1$. This vector is then ...
1
vote
2answers
21 views

What are tokens and tokenizations?

I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What ...
0
votes
0answers
5 views

Analyzing multispectral images data from a drone

We are planning to develop an application/software for monitoring of rice field. After researching, I was curious on how to analyze the data from drone camera. Reading the data and then showing the ...
0
votes
0answers
11 views

Bayesian Statistics — Classification Using Probabilistic Inputs

would like to hear people's opinion on the problem. I am working on a project and would like to know if on right path. Essentially, I have a segment of 100k people from a population i'm trying to ...
0
votes
1answer
20 views

k means clustering when k = n

I want to find the minimum value of the objective function if we set K equal to the number of samples. I know the objective function is $J=\Sigma_{n=1}^{N}\Sigma_{k=1}^{K}r_{nk}||x_n-\mu_k||^2$ And ...
0
votes
0answers
15 views

Best technique for testing if an event causes a trend?

Example - A major holiday is coming up and the frequency of a product being purchased starts increasing significantly as it gets closer to the day of the holiday. Then after the holiday, the frequency ...
1
vote
0answers
19 views

Proportion of positive/negative label in Supervised Learning

I'm working on a Supervised Machine Learning problem and I have a question about the proportion of positive/negative label. I would like to categorize some batch as OK or NOK. But actually my batchs ...
0
votes
0answers
18 views

How to resolve IndexError: too many indices for array

I am attempting to load my dataset of images using the following code. ...
0
votes
1answer
29 views

Neural network or other algorithms?

I have a regression problem, with a million rows or so, around 10-15 features. What should work better on that particular setting? Neural network or regular regressors?
0
votes
0answers
8 views

NLP Model to predict HTML missing attributes based on Word Embedding

I am trying to build a model "doc2vec" which predicts the missed attributes (id, name) of the tag in HTML depending on the text or code around. Is that possible in doc2vec? If so, how to do it ...