Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

127 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
1answer
269 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
3
votes
0answers
20 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
3
votes
1answer
145 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
3
votes
1answer
51 views

Convert from many sub-tables to a single tidy dataframe

I have data in a bad format that I want to make tidy using an R script. I lack the skills to convert it into the format I want, so I look for an answer that (a) outlines a method, or (b) hands me the ...
3
votes
0answers
31 views

Method to create master product database to validate entries, and enrich data set

We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the ...
2
votes
1answer
53 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
2
votes
0answers
17 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
2
votes
0answers
123 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
2
votes
0answers
101 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
2
votes
2answers
302 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
2
votes
1answer
37 views

What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.
2
votes
0answers
72 views

PCA huge parts of missing data filling

I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors. The issue I’m facing is that some of the factors come in and out of the time ...
2
votes
0answers
70 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
2
votes
0answers
50 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
2
votes
0answers
107 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0answers
140 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
84 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
2
votes
1answer
27 views

How to make smaller categories with factor character variables

I have this data set with consist of ISO3166 Alpha-2 codes for countries. Example: DE, AD, AE etc They are coded as factor variables in R and there are about 173 observations. Now because there are ...
1
vote
1answer
33 views

Hierarchical dirichlet process results

I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution. The ...
1
vote
0answers
18 views

NLP Classification labels have many similarirites, converge to and replace to only have one

I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really ...
1
vote
1answer
14 views

Grouping profiles strings having the same words, but occurring out of order

I have a dataframe containing a column of profile types, which looks like this: ...
1
vote
0answers
18 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
1
vote
0answers
17 views

Splitting Temporal Data

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in ...
1
vote
0answers
20 views

how to deal with columns that has different value in only 1 or 2 rows?

I have very high dimensional data. Almost 20% of the columns has different value in less than 1% of rows. All of these are binary columns and many columns has 0s filled in more than almost 98% of rows....
1
vote
1answer
78 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
0answers
20 views

Is there a pandoc for data manipulation?

You know how pandocs converts markdown to HTML and pdf etc. Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?
1
vote
0answers
15 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
1
vote
0answers
27 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
1
vote
0answers
19 views

Excluding “mislabeled” examples in training set based on out of time data

This question is specifically regarding imperfect labels. I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained ...
1
vote
0answers
20 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
1
vote
0answers
30 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...
1
vote
0answers
171 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
1
vote
1answer
49 views

Group_by 2 variables and pivot_wider distribution based on 2 others

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
1
vote
1answer
1k views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
1
vote
0answers
19 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
1
vote
1answer
50 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
1
vote
0answers
13 views

Recognizing emerging topic within ongoing topic

I have been collecting a large amount of tweets from Twitter for a few weeks related to a series of specific keywords, and would like to address a specific problem. Say I collected all tweets ...
1
vote
1answer
4k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
1
vote
0answers
39 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
1
vote
2answers
62 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
1
vote
1answer
36 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
1
vote
0answers
234 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
3answers
275 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
1
vote
0answers
35 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
1
vote
1answer
114 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
1
vote
1answer
97 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
14 views

Term for an identifier that has been superseded

Is there a 'proper' term for an ID (or IDs) that have been superseded by (or merged into) another ID? My use case: rsIDs are used by geneticists to refer to a SNP. These take the form of a string '...
1
vote
0answers
55 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
1
vote
0answers
66 views

How to concatenate many .psv files in google collaboratory?

I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files section. I unzipped it with the ...