Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
0answers
36 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
1
vote
0answers
23 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
0answers
152 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
1
vote
0answers
65 views

preparing time series data for building a rnn

I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a ...
1
vote
0answers
13 views

How to a object type data which contains a string of unique id for feeding in my model?

I am having a dataset in which data type of unique id of a user is in object form. I need to convert it into Int for feeding this data into my model. here is first rows of my dataset. ...
1
vote
0answers
932 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using: ...
1
vote
1answer
66 views

How do I prepare data in which each output row depends on multiple input rows?

My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y. In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on ...
1
vote
3answers
4k views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
1
vote
0answers
63 views

Extracting the time duration for events from an event log

I have an event file (very similar to a log file) that logs event information that includes user data and time stamps. The events have an ordering for each user. So event E1 occurs before E2 that ...
1
vote
0answers
48 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
1
vote
3answers
195 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
1
vote
0answers
76 views

calculate days from a column to today in r [closed]

I have data set with a Date column, now i am trying to calculate duration in days from today to dates in date column with a new column named duration. I tried ...
1
vote
1answer
340 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
1
vote
0answers
77 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
1
vote
0answers
41 views

How to preprocess raw data in the form RDF triples to perform Information Gain and Principal Component Analysis? [closed]

I have my dataset in the form RDF triples in various domains such as Movie, music etc. Data is in the form of RDF triples (Subject, Property,Object (all uris). A sample input in the movie domain is ...
1
vote
0answers
292 views

Delete strings with a specific last character in tibco spotfire [closed]

I am using tibco spotfire and I have columns with data like this "0001V", "0002V", "0003N". I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I ...
1
vote
1answer
31 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
1
vote
1answer
568 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
1
vote
1answer
438 views

Downsampling audio files for use in Machine Learning

I'm trying to use the work (Neural Networks) done in this repo: https://github.com/jtkim-kaist/VAD It says this: Note: To apply this toolkit to other speech data, the speech data should be ...
1
vote
2answers
2k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
1answer
563 views

Am I supposed to scale binary features alongside other numerical features?

When I am cleaning my data, I have some features which contain large numbers and some features that are binary. Should I scale the large features and then add the binary columns or just scale them all ...
1
vote
0answers
37 views

Find irregular bounding shape of 3D particle distribution

I have some 3D particle data e.g. (x0,y0,z0) (x1,y1,z1) (x2,y2,z2) ... I want to find the irregular bounding shape of the distribution. The image below shows an ...
1
vote
0answers
27 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
1
vote
0answers
85 views

Dimension reduction : binary description and regression

Let assume that I own a model agency (that is a far-fetched example), and I would like to create a tool for my HR department, that gives an estimate salary for any given profile. I have a huge ...
1
vote
1answer
374 views

R data frame create a new variable which corresponds to one of the existing one

Currently this is what I get. "community_area_clean" is the new variable that I added by extracting a variable that lists all the communities from a data frame. "CommArea_Name" is the original "...
1
vote
2answers
323 views

Multivariate linear regression accounting for threshold / data cleaning

I am trying to make a linear regression model for the sale price of a house based on many variables (based on the data from this Kaggle challenge https://www.kaggle.com/c/house-prices-advanced-...
1
vote
0answers
65 views

How to model this variable?

I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, ...
1
vote
0answers
340 views

Scoring consistency within dataset

Suppose I am given a set of structured data. The data is known to be problematic, and I need to somehow "score" them on consistency. For example, I have the data as shown below: ...
1
vote
0answers
101 views

What's the best way to rank aggregate imdb rating data?

I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: 1) multiply each average rating by the total ...
1
vote
0answers
70 views

What tools are available for semi-automated matching of dirty columnar data

Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets? The data I'm working with was collected (and handled) by different organizations. ...
0
votes
2answers
161 views

Deep learning - aesthetics data modelling

I want to train neural network on aesthetics. I am getting confused on how to go about for training data. Assume, I have large data set of landscapes, portraits, wildlife etc which are aesthetic ...
0
votes
2answers
52 views

Dealing with data with no classes

I am new for data science subject and I am trying to learn on my own. Please bear with me if this is a stupid question. I have a dataset that has no classes. The data set comprises people's activity ...
0
votes
3answers
132 views

What is the better to pre-process data, MS-Excel or Python/R [closed]

While handling data, many anomalies occur that might change the results. For example, outliers, missing values, etc. So in this context, what is the better tool to pre process data? A visual tool like ...
0
votes
2answers
124 views

Building predictive model with low correlated data [closed]

I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
0
votes
2answers
96 views

Data Cleaning, refining etc

I have data as below: Carpanter Carepnter Carpentor Labourer Labor Labour Housewife House Wife housewife. I want to clean data and rectify the spelling mistakes ...
0
votes
2answers
226 views

Filling in missing values

Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do fillna.(alldata.mean()) or will that cause ...
0
votes
1answer
6k views

How to read Several JSON files to a dataframe in R?

I have a folder with 30,000 plus JSON file. A sample file with contents is posted below. ...
0
votes
1answer
23 views

Cleaning NaNs with averages pre or post split? [duplicate]

I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode. Technically, the validation/ test data has never been seen before - so how could I include it in the ...
0
votes
2answers
442 views

Does resizing the image size, reduce the quality of the images in CNN?

I am doing a project for cancer recognition. My data set has hunderds of images but not of equal size. I wanna resize them to the size of the smallest image. But I am wondering do you think using the <...
0
votes
1answer
5k views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
0
votes
1answer
38 views

CSVs: how to output missing data to make processing easier?

If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing? Some non-empty sentinel, like NA ...
0
votes
1answer
28 views

Is it acceptable to make new independent variables out of old ones?

I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
0
votes
1answer
345 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
30 views

CASE sentence for SQL query

I have a question abut MySQL I would like to handle a column with characters without changing the original DB. Below is the levels of the column named 'Maker'. 기아, 현대, BMW, VolksWagen, Porsche So ...
0
votes
2answers
377 views

Url string processing: what is the best way?

I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls: 1) https://...
0
votes
1answer
18 views

how to export the data edited to a new excel file

The provided code imports the excel file and removes the not NaN values and store it in df2. Now, How do i save the changes to an another file. ...
0
votes
2answers
2k views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
0
votes
1answer
1k views

Script to convert non-numeric elements that are numbers to numeric

I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward. I downloaded some Google Play Store ...
0
votes
1answer
37 views

Am I justified in dropping this independent variable?

I'm currently doing churn prediction in R and during EDA, I discovered that a variable, say gender, has 1720 males who don't churn, and 280 males who do. Also, it has 864 females who don't churn, and ...
0
votes
2answers
2k views

Delete rows of a column using another column statistics in R [closed]

I have a dataframe with the following columns: If a year (see "year" column) has less than 10 weeks reported, I would like to delete it from the dataframe. The column "weeks_reporting" gives the ...

1
7 8
9
10 11
13