Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
1answer
14 views

A question on pipeline

I made a pipeline with standard scaler and k means .When I fit the pipeline to the training data, Does the standard scaler just fits or fits and transforms the training data?
1
vote
1answer
25 views

Does mini-batch gradient descent nullify the effect of stratification on the training data set?

In data pre-processing, stratified shuffle is used to ensure that the distribution of the original dataset is reflected in the training, test and validation dataset. Mini-batch gradient descent uses ...
0
votes
1answer
23 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
0
votes
2answers
90 views

Dataset Merging [closed]

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
3
votes
1answer
142 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
28
votes
3answers
31k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
1
vote
2answers
1k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
0
votes
1answer
16 views

How to use SimpleImputer class to multiple numeric columns in a datasets? [closed]

impter=imputer.fit(df[["Age","Salary"]]) Error Input contains NaN, infinity or a value too large for dtype('float64').
0
votes
1answer
53 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
1answer
47 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0
votes
0answers
8 views

When I'm saving a string into CSV file, all the new line characters are also visible in the csv file. I want each sentence to be on new line [migrated]

When I'm saving a string into CSV file, all the new line characters are also visible in the csv file. New line characters should put all sentences on different line but all '\n' are visible in the ...
0
votes
1answer
18 views

Combining Two CSV's in Jupyter Notebook

I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
0
votes
1answer
16 views

How would you encode missing pixels in image data?

I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it. Since the values are always positive, and ...
0
votes
1answer
108 views

Comparing data sets with different measurements

I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. One piece of data (https://www.pandasecurity....
4
votes
1answer
115 views

Looking for smallest set of rows that form a natural key in a data set

I have several sets of text files on hdfs that are exports from relations. Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row ...
4
votes
1answer
237 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
0
votes
1answer
383 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
21
votes
3answers
14k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
1
vote
1answer
47 views

How to access the data of the column on which some groupby operation has been carried out? [closed]

Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries ...
1
vote
1answer
100 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
1
vote
0answers
17 views

Is there a pandoc for data manipulation?

You know how pandocs converts markdown to HTML and pdf etc. Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?
1
vote
0answers
23 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
3
votes
1answer
19 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
6
votes
3answers
655 views

Does label encoding an entire dataset cause data leakage?

I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the ...
0
votes
0answers
8 views

Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it?

I have Timeseries data to clean and I got the tip to use a kalman Filter. My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
0
votes
0answers
10 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
1
vote
2answers
183 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
1
vote
1answer
42 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
0answers
13 views

How does one decide on the right image quality filters for a training and testing set?

What image quality metrics do people use to choose appropriate quality (e.g. focus, orientation, obstruction, etc.) images for important for image classification models? I would imagine anything that ...
4
votes
1answer
46k views

TypeError: float() argument must be a string or a number, not 'function'

I am trying to clean the data. But I don't know how to remove a function from a column in data frame. At row number 473 it show column N has a function . How it should be filtered out ?
2
votes
2answers
252 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
0
votes
1answer
137 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
1
vote
1answer
17 views

Cleaning a certain feature to predict salary using Machine Learning

Info: I am working on a dataset, and i would like to create a model that would predict salary. Columns are as follows: ...
1
vote
1answer
17 views

How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand. Here is my code (...
1
vote
1answer
26 views

Predicting equipment failure with time series alarm data

I am trying to predict machine failures based on alarm data. The situation: There is approximately 4000 machine failures per year. These are labelled poorly (it is entered manually and can have ...
1
vote
1answer
3k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
1answer
16 views

Logistic Regression Multi-level Independent variables

im trying to study logistic regression, when i did the target variable with all features, i had the summary showing the p-values as usual, but one for the features has 60 level, another feature has 13 ...
0
votes
0answers
147 views

What are some standard or preferred battery of tests to perform to test for quality of my data before feeding to a ML algorithm?

we are designing a rules based engine to check the quality of the data before training our ML models. The data we have is time series data. We have about 3-4 features using which we have to make a ...
2
votes
1answer
18 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
0
votes
1answer
37 views

how does xgboost handle inf or -inf values?

all, i am using xgboost for binary classfication. I have infs and -infs in my data due to the fact i am calcaulting ratios from one col and and another e.g. df[col1]/df[col2] , since i have zeros ...
1
vote
1answer
15 views

Reformatting data- Giving each value in a list it's own row while retaining the list's ID

I am looking to reformat some data. It currently looks like this: Using this as an example, the below is the format i'm trying to achieve: So each element in the list gets it's own row, but the ...
1
vote
1answer
35 views
0
votes
1answer
2k views

How to find the ranges in Equal frequency/depth binning?

I have been looking into the site http://www.saedsayad.com/unsupervised_binning.htm and there it shows range values to the right under equal frequency binning .... I have so much looked to find how ...
3
votes
3answers
537 views

Are there libraries or techniques for 'noisifying' text data?

Data augmentation techniques for image data and audio data (eg speech recognition) have proven successful and are now common. Are there libraries or techniques for augmenting text data? For example: ...
0
votes
0answers
12 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
0
votes
1answer
15 views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
20
votes
5answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
0
votes
2answers
46 views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
1
vote
1answer
24 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...

1
2 3 4 5
11