Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

142 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0
votes
1answer
43 views

How do I iterate over my images in dataset?

I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like ...
0
votes
0answers
17 views

How to use data provided in EEM20 forecasting competition?

I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...
0
votes
0answers
21 views

Flow System - Analysing Measurement Errors

I have a closed system with multiple flow measurement points as shown; where in theory (A1 + A2 + A3) = (B1 + B2 + B3) = C = (D1 + D2) and D1 = (E1 + E2 + E3 + E4) D2 = (F1 + F2 + F3 + F4) Some of ...
0
votes
0answers
8 views

Manipulating the relative weight of tokens inside CountVectorizer

This is transparently a classic IMDB data recommender question. I'm trying to build a recommender system that suggests movies that a user is likely to enjoy. If I have my terminology correct I am ...
0
votes
0answers
23 views

How could you predict customer churn using transactions date?

I am new to machine learning and I would like to predict churn using dates of transactions. I tried to prepare my data and I couldn't obtain good results. I would like to predict unique Customers, ...
0
votes
1answer
15 views

Is it acceptable to make new independent variables out of old ones?

I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
0
votes
1answer
30 views

Standardizing features by one specific feature

I am working on a project with a dataset that looks something like the following: ...
0
votes
0answers
7 views

Identifying data distribution for my data and model building strategy

i am a beginner for panel data econometrics. Need help of experts of the field deciding if it is panel or pooled data or should i use any other methodology. I have data from wind mill and we use wind ...
0
votes
0answers
27 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
0answers
72 views

Since is not possible test all the possible combination, what is the correct procedure to follow on building Machine Learning?

Sorry, I'm a little confused and this is a general question. How can I be sure that the procedure that I am following is the correct one? Following the steps for building a machine learning model, we ...
0
votes
0answers
32 views

One Hot Encoding where all sequences don't have all values

Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder ...
0
votes
0answers
16 views

Trouble understanding how I could use multivariate time series to predict when an error will occur?

First off, I have very limited knowledge statistics-wise and am more of a coder. I was thrown into a large scale project and could use some guidance. I have a large multivariate time series dataset ...
0
votes
0answers
23 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
125 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
0
votes
1answer
20 views

How to group a unique column and find the mean value in a separate column in R

My table of data looks like this image (but with many more lines): I want to be able to find the mean age of each region, so the mean age for region SSC20790 and so on. I believe I can use the <...
0
votes
0answers
35 views

Linear regression plotting abline()

I have a ufc data set which i got and started cleaning for my own practice This the link to data:-https://www.kaggle.com/rajeevw/ufcdata#raw_fighter_details.csv and using ...
0
votes
0answers
23 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
41 views

How to make a heatmap using keypoints from MPII data set

So I am tiring to figure this out but keep running into a wall. So what I want to do is create a heatmap from a 2D image based on the key points generated from the MPII dataset. I have found a CSV ...
0
votes
0answers
19 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
0
votes
0answers
77 views

Suggestion for handling specific missing data

I have data, that describes distance from given location to nearest object (e.g. school, shop etc). Because of performance reasons I couldn't scrape the data about objects, that are futher away than 2....
0
votes
0answers
22 views

Optimal practices to group data by Customer ID for churn prediction

Here's a quite common problem and I read a couple of questions/answers on it, however I still having my doubts about what are the best practices for grouping data by Customer ID for churn prediction. ...
0
votes
0answers
14 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
0answers
85 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
0answers
25 views

Data Transformation Tips for xgboost's XGBClassifier

I have this X_train and test distribution for the 4 features 'X', 'Y', 'TX' and 'TY'. I realize the range of the distribution is widely varying .. Can you suggest a good way to clean/ transform that ...
0
votes
0answers
17 views

How do I clean the data for cluster analysis using sas miner

Hi I am looking to run some associative and clustering analyses using this dataset on kaggle. All the methods on Kaggle is in R and Python. Any ideas on how to clean and deal with the missing values? ...
0
votes
1answer
29 views

Grouping similar rows to detect duplicates

I have a dataset containing all the real estate ads for sale in the process of publishing a city: ...
0
votes
0answers
23 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
0
votes
2answers
225 views
0
votes
1answer
671 views

Normalization and Outlier on Target variable which is continuous

I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?
0
votes
1answer
40 views

How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

I currently have a sparse matrix object of TfidfVectorizer which is of 1000 length. Right now it is displayed like this : ...
0
votes
1answer
145 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
398 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
1answer
110 views

Comparing data sets with different measurements

I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. One piece of data (https://www.pandasecurity....
0
votes
1answer
316 views

Downsampling audio files for use in Machine Learning

I'm trying to use the work (Neural Networks) done in this repo: https://github.com/jtkim-kaist/VAD It says this: Note: To apply this toolkit to other speech data, the speech data should be ...
0
votes
1answer
283 views

Tool vs Python Script for Transforming Data in Mongo

We have a bunch of Mongo collections (data collected from APIs, web scraping, etc) that we need to transform to a cleaner data structure (standardized schema) on a monthly basis. Are there any good ...
0
votes
1answer
65 views

What are appropriate labels for age categorical labels?

I am converting some age data to categorical variables. What are some appropriate labels? Some people might take offense to using "Young", "Old" or "Millenial", etc. Is there a "standard" list of ...
-1
votes
2answers
2k views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
-1
votes
1answer
129 views
-1
votes
1answer
76 views

Text standardisation for manually entered data

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies ...
-1
votes
1answer
110 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
-1
votes
2answers
69 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-2
votes
1answer
212 views

clean the data with sample selection?

I am using Python to do weather forecasting. Here is the original data: X.txt is the input, and y.txt is the output. https://drive.google.com/drive/folders/0BxxxnsxpOeLNUHRPdHdYUW9NYkk?usp=sharing ...

1 2
3