Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
0answers
110 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
4
votes
3answers
188 views

How to deal with count data in random forest

I am working on a classification model where my target class is a biased class with the class shape as 0 1 20694 101 Most of my features are the ...
1
vote
0answers
62 views

preparing time series data for building a rnn

I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a ...
4
votes
0answers
44 views

How to filter this signal to get a heartbeat signal?

I am giving my first steps in data analysis, gathering/cleaning. To learn, I am trying to create a simple code that can detect heartbeats from color variations from the image coming from the camera ...
2
votes
1answer
3k views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
1
vote
0answers
11 views

How to a object type data which contains a string of unique id for feeding in my model?

I am having a dataset in which data type of unique id of a user is in object form. I need to convert it into Int for feeding this data into my model. here is first rows of my dataset. ...
2
votes
2answers
2k views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
0
votes
1answer
38 views

How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

I currently have a sparse matrix object of TfidfVectorizer which is of 1000 length. Right now it is displayed like this : ...
1
vote
1answer
18 views

How can I extract the top words from a string in my dataframe column?

I have a column in my dataframe which is in a string providing description of a product. For example : This is a shirt. It is blue in color. The sizes available are large, small. The shirt is tight ...
1
vote
2answers
54 views

How can I see a long string in my dataframe?

I have a column in my dataframe in which there are sentences which are too long. I want to see them as a whole but every time I perform even a simple iloc operation i get output like 'i am going to...'...
0
votes
1answer
36 views

Am I justified in dropping this independent variable?

I'm currently doing churn prediction in R and during EDA, I discovered that a variable, say gender, has 1720 males who don't churn, and 280 males who do. Also, it has 864 females who don't churn, and ...
1
vote
2answers
3k views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
2
votes
2answers
223 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
0
votes
1answer
116 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
346 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
3
votes
3answers
2k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
0
votes
2answers
25 views

Converting string_id to number_id

I have column with movie ids like this: tt0984332 tt0984332 tt0847742 ttnanana1 I would like to convert them into numbers that can be inserted into neural ...
0
votes
1answer
1k views

Regression in Python with many NaN values spread across all columns

I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank ...
1
vote
3answers
224 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
1
vote
0answers
641 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
0
votes
1answer
46 views

Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension. For example: ...
1
vote
1answer
183 views

Wave form analysis ML algorithm [closed]

I am trying to determine a person's emotions from their speech. This immediately rings Machine Learning bells and the first step in any ML problem is getting and processing data. My first question is, ...
0
votes
2answers
71 views

Building predictive model with low correlated data [closed]

I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
0
votes
1answer
74 views

How to do multivariate survival analysis on dataset having only categorical variables

I have a dataset where I have around 50 independent variables used to run survival analysis on the target variables. But out of these 50 variables, 46 variables are categorical variables i.e. having ...
1
vote
1answer
1k views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
0
votes
1answer
62 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...
1
vote
0answers
345 views

Converting HTML tables to a data.frame in R

I am trying to convert an HTML table of daily data into a R data.frame for trend analysis. Using xml2 and rvest I am able to download the table into RStudio, but am struggling to A) create the ...
1
vote
0answers
43 views

How do I prepare data in which each output row depends on multiple input rows?

My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y. In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on ...
1
vote
3answers
2k views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
2
votes
2answers
4k views

Pandas Conditional Fill NaN Forward/Backward

Updated 22 Oct. 2018: I have the following dataset: ...
0
votes
2answers
1k views

Delete rows of a column using another column statistics in R [closed]

I have a dataframe with the following columns: If a year (see "year" column) has less than 10 weeks reported, I would like to delete it from the dataframe. The column "weeks_reporting" gives the ...
0
votes
3answers
12k views

How to access substrings in pandas column and store it into new columns?

I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like ...
0
votes
3answers
16k views

How to replace a part string value of a column using another column

How to replace a part string value of a column using another column. My DataSet here is : ...
0
votes
1answer
49 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
1
vote
0answers
37 views

Extracting the time duration for events from an event log

I have an event file (very similar to a log file) that logs event information that includes user data and time stamps. The events have an ordering for each user. So event E1 occurs before E2 that ...
1
vote
1answer
4k views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
2
votes
2answers
3k views

How to get a count the number of observations for each year with a Pandas datetime column?

I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year. This is the column: ...
2
votes
2answers
52 views

What methods can be used to detect duplicacy in image dataset?

I want to remove duplicate images from a dataset of 50Million images. What is the best method to detect all the duplicates? Do you think one shot learning is good for this?
3
votes
2answers
2k views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
1
vote
0answers
40 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
1
vote
3answers
6k views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
4
votes
1answer
176 views

Can someone please explain what this sample function is upto?

So there is a function in Dino_Name_Generator at Deeplearning.ai notebook ...
1
vote
6answers
147 views

How to statistically prove that a column in a dataframe is not needed

I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
1
vote
3answers
107 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
-1
votes
1answer
5k views

How do I add a column to a Pandas dataframe based on other rows and columns in the dataframe? [closed]

I've tried a lot of different methods, but I can't seem to find the right way to do this. I want to create a new column based on the time and ...
1
vote
0answers
21 views

How to prepare future data for training

Assume I have a large data of ecommerce website sessions with user id key (a user can have multiple sessions with random time between them). The data is on S3 in json gzipped format. On some sessions ...
0
votes
1answer
33 views

data analysis EDA issues, indent or type

I am a newbie in DS world, right now i am working on some EDA practice, and run into an issue here. Here is my code: Convert some columns to numeric values ...
1
vote
0answers
23 views

calculate days from a column to today in r

I have data set with a Date column, now i am trying to calculate duration in days from today to dates in date column with a new column named duration. I tried ...
1
vote
2answers
4k views

LOOKUP using 2 dataframes in Python

I have 2 dataframes: df1 Id CategoryId 1 A 1 B 2 A 2 E 2 F df2: ...

1
3 4
5
6 7
10