Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
-1
votes
2answers
5k views

How to deal with nan values after merging / joining two dataframes? [closed]

A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
-1
votes
1answer
416 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
-1
votes
0answers
11 views

Saved sklearn encoder does not work properly?

Used Sklearn's one hot enoder on datframe (shape 100000*10) which gave me 120 columns. Saved this encoder in .pkl file Later loaded this encoder for encoding same <...
-1
votes
1answer
26 views

Increase accuracy in binary classification with ambiguous data

I am fairly new to Datascience and currently working on an assignment that requires me to do a binary classification on a set with about 9 parameters for X. I tried working on it using different ...
-1
votes
1answer
21k views
-1
votes
2answers
4k views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
-1
votes
3answers
5k views

Comparing a column value to list of values and if it contains the value then assigning that list value to new Column

Column and wanted to check with this list ...
-1
votes
2answers
56 views

Loops in R programming, [closed]

I want to update this Remaining column in this table below: How do I do this in R programming? I tried while (n.Remaining>0) { n.remaining <- n.total-n.expense} Desired Output:
-1
votes
1answer
161 views
-1
votes
1answer
78 views

Text standardisation for manually entered data

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies ...
-1
votes
1answer
122 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
-1
votes
2answers
72 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-2
votes
1answer
441 views

How should I analyze this data from reddit (sample text included) [closed]

I have downloaded some data to learn about machine learning and distributed computing. I used WinZip to uncompress a .bz2. Now I have text (opened in Notepad) which looks like this (this file is long, ...
-2
votes
1answer
6k views

How do I add a column to a Pandas dataframe based on other rows and columns in the dataframe? [closed]

I've tried a lot of different methods, but I can't seem to find the right way to do this. I want to create a new column based on the time and ...
-2
votes
1answer
220 views

clean the data with sample selection?

I am using Python to do weather forecasting. Here is the original data. Thanks! The input has 180 features and the meanings are: Suppose we are doing forecast for hour k. We will use the historic ...
-3
votes
3answers
246 views

Median versus Average, how to choose?

I want to test how long it takes to run an algorithm. So here is what I am doing: close all the other un-needed applications, run my algorithms alone considering some unstable computer system ...
-3
votes
1answer
40 views

How to do feature selection for classification problem? Which technique will work? [closed]

I have 200 variables with 200000 records. How to find best features from this variables? I have tried correlation technique via Heatmap but all the variables have near to same correlation score < 0....
-3
votes
1answer
74 views

Unable to figure out the groupby question

I am not able to understand what is asked in the question. Can somebody please explain it to me? The given dataset is this, with 15 countries, 5 continent and countries population, renewable (...
-3
votes
1answer
36 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...

1
9 10 11 12
13