Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

144 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
234 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
3
votes
1answer
19 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
3
votes
1answer
33 views

Data-preprocessing for Machine Learning model

I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the ...
3
votes
0answers
17 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
3
votes
1answer
140 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
3
votes
0answers
28 views

Method to create master product database to validate entries, and enrich data set

We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the ...
2
votes
0answers
16 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
2
votes
0answers
107 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
2
votes
0answers
90 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
2
votes
1answer
1k views

Tips on how to preprocess data and outliers for churn analysis

I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions, 1) When I skewness and kurtosis for the normality test, two ...
2
votes
2answers
246 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
2
votes
1answer
33 views

What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.
2
votes
1answer
45 views

Convert from many sub-tables to a single tidy dataframe

I have data in a bad format that I want to make tidy using an R script. I lack the skills to convert it into the format I want, so I look for an answer that (a) outlines a method, or (b) hands me the ...
2
votes
0answers
68 views

PCA huge parts of missing data filling

I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors. The issue I’m facing is that some of the factors come in and out of the time ...
2
votes
0answers
65 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
2
votes
0answers
49 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
2
votes
0answers
107 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0answers
136 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
84 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
2
votes
1answer
25 views

How to make smaller categories with factor character variables

I have this data set with consist of ISO3166 Alpha-2 codes for countries. Example: DE, AD, AE etc They are coded as factor variables in R and there are about 173 observations. Now because there are ...
1
vote
0answers
16 views

Is there a pandoc for data manipulation?

You know how pandocs converts markdown to HTML and pdf etc. Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?
1
vote
0answers
22 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
1
vote
0answers
18 views

Split-Half Reliability Python

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
1
vote
0answers
19 views

Excluding “mislabeled” examples in training set based on out of time data

This question is specifically regarding imperfect labels. I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained ...
1
vote
0answers
13 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
1
vote
0answers
47 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
1
vote
0answers
114 views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
1
vote
1answer
24 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
1
vote
1answer
43 views

Group_by 2 variables and pivot_wider distribution based on 2 others

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
1
vote
1answer
735 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
1
vote
0answers
15 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
1
vote
1answer
46 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
1
vote
0answers
13 views

Recognizing emerging topic within ongoing topic

I have been collecting a large amount of tweets from Twitter for a few weeks related to a series of specific keywords, and would like to address a specific problem. Say I collected all tweets ...
1
vote
1answer
3k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
1
vote
0answers
38 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
1
vote
2answers
58 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
1
vote
1answer
34 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
1
vote
0answers
147 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
3answers
179 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
1
vote
0answers
33 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
1
vote
1answer
99 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
1
vote
1answer
81 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
71 views

Cleaning Excel Data in Merged Cells

I need to use an Excel file that have many merged cells with no specific order. I guess that the file creator thought that it would be more readable to merge the cells that have the same value. ...
1
vote
0answers
14 views

Term for an identifier that has been superseded

Is there a 'proper' term for an ID (or IDs) that have been superseded by (or merged into) another ID? My use case: rsIDs are used by geneticists to refer to a SNP. These take the form of a string '...
1
vote
0answers
49 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
1
vote
0answers
52 views

How to concatenate many .psv files in google collaboratory?

I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files section. I unzipped it with the ...
1
vote
0answers
35 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
1
vote
0answers
20 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
0answers
113 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
1
vote
0answers
62 views

preparing time series data for building a rnn

I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a ...