Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
0answers
6 views

Group by 2 variables in R to calculate sum, count, percentages, means, modes based on value of 3rd variable

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Would be 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all w/ POS Direction ...
0
votes
1answer
46 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
2
votes
0answers
31 views

Fixed-Width vs Adaptive Binning

I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
-1
votes
0answers
22 views

Renaming Column Elements [closed]

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
2
votes
1answer
38 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
1
vote
0answers
15 views

Replace values before and after a certain value [closed]

I have a variable like this: V1: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this: V1: 1 1 1 1 1 1 1 0 0 0 0 ...
2
votes
2answers
31 views

Data Quality Assessment in Image Data

I'm working on a CNN model that classifies images. After scraping image files from the Internet, I found that many of them didn't look this way as described by the searching keyword (for example ...
1
vote
1answer
15 views

Replacing missing values with mean of feature calculated from previously replaced values

I am not sure how to ask this, but I will try my best. I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all ...
1
vote
0answers
14 views

Need some advice on approach to select only the informative emojis from the data set?

I have a giant data set from a local elections, which contains hashtags, emojis, and comments. I wanted to make a network analysis using only emojis. So far I have a network analysis graph made in R ...
2
votes
0answers
13 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
0
votes
2answers
32 views

Skewed Data doesnt follow normal distribution

My data (continuous) is highly skewed and it doesn't follow the normal distribution. Using sns.distplot I found out that exponweib fits the data better. How to ...
2
votes
1answer
36 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
1
vote
0answers
34 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
3
votes
0answers
16 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
0
votes
1answer
14 views

How to order the data with respect to data type

I am having large data set (82 variables) Is there any way to arrange data such a way that I have to get all numerical variables firstly then categorical variables so that I can run hypothesis ...
1
vote
3answers
62 views

Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
1
vote
1answer
31 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
2
votes
2answers
55 views

Recommended data cleaning techniques for multivariate time series prediction?

I have to predict the next step(s) in a multivariate time series with about 30 features and 50.000 samples. I am thinking of using LSTM. Which techniques are usually recommended for cleaning the data ...
0
votes
2answers
25 views

Dealing with data with no classes

I am new for data science subject and I am trying to learn on my own. Please bear with me if this is a stupid question. I have a dataset that has no classes. The data set comprises people's activity ...
0
votes
2answers
29 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
1
vote
2answers
53 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
0
votes
0answers
31 views

Linear regression plotting abline()

I have a ufc data set which i got and started cleaning for my own practice This the link to data:-https://www.kaggle.com/rajeevw/ufcdata#raw_fighter_details.csv and using ...
1
vote
0answers
13 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
2answers
59 views

How to handle addresses of the restaurants to feed the data-set in the ML model?

I have data from different restaurants which have also address of the restaurants now I want to predict the food delivery timing based on the given data, now the restaurant address is one of the ...
0
votes
1answer
15 views

How much of a disadvantage is a small sample size?

I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my ...
0
votes
0answers
21 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
25 views

How to make a heatmap using keypoints from MPII data set

So I am tiring to figure this out but keep running into a wall. So what I want to do is create a heatmap from a 2D image based on the key points generated from the MPII dataset. I have found a CSV ...
0
votes
1answer
32 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking of ...
2
votes
2answers
45 views

Pandas datetime error when reading from excel file

I am trying to read an excel file that has two columns using pandas. This is how the data looks in excel file: ...
1
vote
2answers
28 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
1
vote
3answers
30 views

How to extract insights from the given data?

Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in ...
0
votes
0answers
18 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
0
votes
1answer
42 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
2
votes
1answer
43 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
-1
votes
2answers
521 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
6
votes
2answers
354 views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
1
vote
0answers
10 views

Recognizing emerging topic within ongoing topic

I have been collecting a large amount of tweets from Twitter for a few weeks related to a series of specific keywords, and would like to address a specific problem. Say I collected all tweets ...
0
votes
0answers
73 views

Suggestion for handling specific missing data

I have data, that describes distance from given location to nearest object (e.g. school, shop etc). Because of performance reasons I couldn't scrape the data about objects, that are futher away than 2....
0
votes
3answers
39 views

What is the better to pre-process data, MS-Excel or Python/R [closed]

While handling data, many anomalies occur that might change the results. For example, outliers, missing values, etc. So in this context, what is the better tool to pre process data? A visual tool like ...
0
votes
0answers
15 views

Optimal practices to group data by Customer ID for churn prediction

Here's a quite common problem and I read a couple of questions/answers on it, however I still having my doubts about what are the best practices for grouping data by Customer ID for churn prediction. ...
0
votes
0answers
13 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
1answer
317 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
30 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
23 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
0answers
18 views

Data Transformation Tips for xgboost's XGBClassifier

I have this X_train and test distribution for the 4 features 'X', 'Y', 'TX' and 'TY'. I realize the range of the distribution is widely varying .. Can you suggest a good way to clean/ transform that ...
8
votes
4answers
293 views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
0
votes
0answers
14 views

How do I clean the data for cluster analysis using sas miner

Hi I am looking to run some associative and clustering analyses using this dataset on kaggle. All the methods on Kaggle is in R and Python. Any ideas on how to clean and deal with the missing values? ...
1
vote
1answer
32 views

How to handle out of scope value?

I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set ...
1
vote
1answer
393 views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
0
votes
0answers
17 views

Averaging over Irregular Intervals

Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation? Specifically, suppose we are ...