We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
1answer
17 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking of ...
2
votes
2answers
30 views

Pandas datetime error when reading from excel file

I am trying to read an excel file that has two columns using pandas. This is how the data looks in excel file: ...
1
vote
2answers
23 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
1
vote
3answers
28 views

How to extract insights from the given data?

Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in ...
0
votes
0answers
17 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
0
votes
1answer
37 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
2
votes
1answer
33 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
0
votes
2answers
48 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
6
votes
2answers
130 views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
1
vote
0answers
8 views

Recognizing emerging topic within ongoing topic

I have been collecting a large amount of tweets from Twitter for a few weeks related to a series of specific keywords, and would like to address a specific problem. Say I collected all tweets ...
0
votes
0answers
62 views

Suggestion for handling specific missing data

I have data, that describes distance from given location to nearest object (e.g. school, shop etc). Because of performance reasons I couldn't scrape the data about objects, that are futher away than 2....
0
votes
3answers
32 views

What is the better to pre-process data, MS-Excel or Python/R [closed]

While handling data, many anomalies occur that might change the results. For example, outliers, missing values, etc. So in this context, what is the better tool to pre process data? A visual tool like ...
0
votes
0answers
11 views

How to get a sample/summarised data from a table for data analysis?

I am currently working on EHR data analysis. I have a measurement table which has information lab test information of the patients. However the record count is huge. These are table count in my EHR ...
0
votes
0answers
14 views

Optimal practices to group data by Customer ID for churn prediction

Here's a quite common problem and I read a couple of questions/answers on it, however I still having my doubts about what are the best practices for grouping data by Customer ID for churn prediction. ...
0
votes
0answers
12 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
1answer
41 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
18 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
18 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
0answers
16 views

Data Transformation Tips for xgboost's XGBClassifier

I have this X_train and test distribution for the 4 features 'X', 'Y', 'TX' and 'TY'. I realize the range of the distribution is widely varying .. Can you suggest a good way to clean/ transform that ...
8
votes
4answers
228 views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
0
votes
0answers
12 views

How do I clean the data for cluster analysis using sas miner

Hi I am looking to run some associative and clustering analyses using this dataset on kaggle. All the methods on Kaggle is in R and Python. Any ideas on how to clean and deal with the missing values? ...
1
vote
1answer
31 views

How to handle out of scope value?

I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set ...
1
vote
1answer
69 views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
0
votes
0answers
15 views

Averaging over Irregular Intervals

Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation? Specifically, suppose we are ...
0
votes
0answers
9 views

Where can I get an untokenized version of GLUE's SST-2 dataset?

On the GLUE faq, they say: Similarly, for SST, the data provided is already tokenized. We're working on obtaining a version that is not tokenized. Feel free to train on other distributions of ...
2
votes
0answers
49 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
0
votes
0answers
24 views

How to handle a data set with large number (about 75%) of binary variables?

I am doing a research right now and want to classify (predict) churns of costumers using machine learning. My data set consists of about 500,000 observations with 20 variables: 15 are binary, 2 ...
1
vote
1answer
24 views

Remove rows that are too much alike not to be duplicates

I have a dataset of real estate advertisements. Several of the lines are about the same real estate property so it's full of duplicates that aren't exactly the same. What would be the best methods to ...
1
vote
3answers
33 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
0answers
8 views

Transforming target from object array to integer array to use sparse_categorical_crossentropy for class prediction

I want to do a neural network to predict to which loan class does a borower pertains. There are 6 classes [ A, B, C, D, E, F]. I tried to get rid of the NAs and ...
0
votes
1answer
22 views

Grouping similar rows to detect duplicates

I have a dataset containing all the real estate ads for sale in the process of publishing a city: ...
1
vote
2answers
51 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
1
vote
1answer
30 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
0
votes
0answers
15 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
20 views

Model Checking in Multiple Imputation

Trying to understand how to incorporate Multivariate Imputation by Chained Equation (MICE) for handling missing data in python. I know there are a few libraries that can implement MICE on a dataset ...
1
vote
0answers
28 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
3
votes
2answers
43 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
26 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
0
votes
1answer
22 views

What are the limits of capping ratios to [0-1]?

I work with some financial ratios. By construction those variables should be between 0 and 1. However, for some reasons I end up with values <0 and >1. These reasons include : reporting problems (...
0
votes
1answer
17 views

Should inputs be shuffled for training a model with expected time dependency?

Consider a prediction model with numerical inputs and outputs. Suppose data is inserted tick by tick, i.e., when new data is available it is inserted asynchronously. Current output depends on current ...
0
votes
1answer
26 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
1
vote
1answer
11 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
2
votes
1answer
53 views

Filtering a panda dataframe in one line

I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process) Given the following ...
0
votes
0answers
8 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
6
votes
2answers
32 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
0
votes
3answers
72 views

Select samples from a dataframe in python [closed]

I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most. thanks
0
votes
0answers
23 views

Real-time Time-Series Error Correction

I have some sensors, each of which generates data points at mostly-regular intervals. So for each sensor I have a time series. I have to correct measurement errors from the sensors, replacing them ...
0
votes
1answer
65 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
25 views

Handling unwanted negative numbers

With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a ...
2
votes
0answers
40 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...