Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

0
votes
2answers
33 views

Loops in R programming,

I want to update this Remaining column in this table below: How do I do this in R programming? I tried while (n.Remaining>0) { n.remaining <- n.total-n.expense} Desired Output:
0
votes
1answer
15 views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
0
votes
0answers
23 views

Filtering unique row values in SQL, Advanced

My data looks like this: Number(String), Number2(String), TransactionType(String), Cost(Integer) For number 1, Cost 10 and -10 cancel out so the remaining cost is 100 For number 2, Cost 50 and -...
0
votes
2answers
20 views
0
votes
0answers
25 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
1
vote
0answers
14 views

R Combine Multiple Rows of DataFrame by creating new columns and union values

I have a dataframe in R that looks like this ...
0
votes
2answers
38 views

Dataset Merging

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
0answers
16 views

Text Mining/Printing with Python

I have a list of users based on frequency of tweets[Here], and a dataset of tweets that looks like [This].I need to write a python script/program that reads my list of users line by line, then for ...
0
votes
0answers
29 views

Working with Stanford twitter datasets in R

I intend to use frequency data and sentiment analysis on the Stanford twitter data. I have a couple issues though. One is the size of the data, which I want to reduce, but how do i do this? A ...
0
votes
2answers
31 views
0
votes
0answers
17 views

XGBoost feature significance and feature importance

In a regression model it is possible to judge at a specified significance level (often alpha = 5%) whether a variable has a significant influence on the target attribute. With XGBoost, you can use ...
0
votes
1answer
21 views

Normalization and Outlier on Target variable which is continuous

I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?
0
votes
0answers
9 views

How to impute opt out data

Consider the following problem: You have been given some survey data on job satisfaction and have to create a model for the employees with the highest risk of leaving the company. The questions have ...
2
votes
1answer
20 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
1
vote
2answers
30 views

Dealing with a dataset with a mix of continuous and categorical variables

How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the ...
4
votes
1answer
78 views

How do I find the relevant features out of 11,000+ possibilities?

While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many? I know that we need to find out which features are relevant, but ...
2
votes
1answer
38 views
6
votes
2answers
125 views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does as per the Sci-Kit documentation. Standard Scaler - https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Normalizer....
1
vote
0answers
11 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
0answers
17 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
3
votes
0answers
33 views

How to deal with count data in random forest

I am working on a classification model where my target class is a biased class with the class shape as 0 1 20694 101 Most of my features are the ...
1
vote
0answers
25 views

preparing time series data for building a rnn

I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a ...
3
votes
0answers
32 views

How to filter this signal to get a heartbeat signal?

I am giving my first steps in data analysis, gathering/cleaning. To learn, I am trying to create a simple code that can detect heartbeats from color variations from the image coming from the camera ...
2
votes
1answer
44 views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
1
vote
0answers
5 views

How to a object type data which contains a string of unique id for feeding in my model?

I am having a dataset in which data type of unique id of a user is in object form. I need to convert it into Int for feeding this data into my model. here is first rows of my dataset. ...
2
votes
2answers
94 views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
1
vote
1answer
12 views

How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

I currently have a sparse matrix object of TfidfVectorizer which is of 1000 length. Right now it is displayed like this : ...
1
vote
1answer
12 views

How can I extract the top words from a string in my dataframe column?

I have a column in my dataframe which is in a string providing description of a product. For example : This is a shirt. It is blue in color. The sizes available are large, small. The shirt is tight ...
1
vote
1answer
15 views

How can I see a long string in my dataframe?

I have a column in my dataframe in which there are sentences which are too long. I want to see them as a whole but every time I perform even a simple iloc operation i get output like 'i am going to...'...
0
votes
1answer
27 views

Am I justified in dropping this independent variable?

I'm currently doing churn prediction in R and during EDA, I discovered that a variable, say gender, has 1720 males who don't churn, and 280 males who do. Also, it has 864 females who don't churn, and ...
1
vote
1answer
88 views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
1
vote
2answers
34 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
-1
votes
0answers
7 views

How should I create attributes in my dataset if any permutation of the values yield the same result?

I am trying to make a movie rating predicting system. My approach is that the team (actors, directors, writers, producers, etc) is the determining factor. There can be more than one of each of those ...
0
votes
1answer
31 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
61 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
-1
votes
0answers
14 views

Will mean-value imputation have the same effect in these two cases before normalization?

Will mean-value imputation have the same effect, if performed before normalization, on the distribution of the normalized values for those values that were originally not missing, for min-max ...
-1
votes
0answers
16 views

Which is the best way to generate synthetic data for data science development

I have a data pipeline in place for streaming and batch data, which collects data from various sources and land it into hadoop's hdfs storage. I want to generate synthetic data with the same schema of ...
-1
votes
0answers
8 views

operations on multiple entries in one column based on conditions meet from multiple column entries

I have a data set where I need to find the difference between the first time entry and last time entry based on the following conditions: Day column entries are equal BusOne[0] = BusOne1 ... BusOne[n]...
3
votes
3answers
624 views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
0
votes
1answer
12 views

Converting string_id to number_id

I have column with movie ids like this: tt0984332 tt0984332 tt0847742 ttnanana1 I need to convert in to numbers that can be inserted into neural network as features, like this: 0 0 1 2 How can I do ...
-1
votes
0answers
10 views

How to smoothen array of samples?

I am implementing a Genetic Algorithm library. However, on Roulette Wheel Selection algorithm sometimes one or two chromosomes dominate he roulette wheel with their relatively large fitness scores. I ...
0
votes
1answer
117 views

Regression in Python with many NaN values spread across all columns

I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank ...
1
vote
1answer
55 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
1
vote
0answers
157 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
0
votes
1answer
25 views

Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension. For example: ...
1
vote
1answer
26 views

Wave form analysis ML algorithm [closed]

I am trying to determine a person's emotions from their speech. This immediately rings Machine Learning bells and the first step in any ML problem is getting and processing data. My first question is, ...
0
votes
2answers
40 views

Building predictive model with low correlated data [closed]

I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
0
votes
1answer
16 views

How to do multivariate survival analysis on dataset having only categorical variables

I have a dataset where I have around 50 independent variables used to run survival analysis on the target variables. But out of these 50 variables, 46 variables are categorical variables i.e. having ...
1
vote
1answer
57 views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
0
votes
1answer
24 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...