Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

1
vote
0answers
23 views
0
votes
0answers
8 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
1
vote
1answer
10 views

Can someone please explain what this sample function is upto?

So there is a function in Dino_Name_Generator at Deeplearning.ai notebook ...
0
votes
3answers
44 views

How to statistically prove that a column in a dataframe is not needed

I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
0
votes
3answers
24 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
0
votes
0answers
8 views

Join Dataframes in Pandas and Sum Columns [migrated]

Lets say I have 2 dataframes in pandas. I want to perform a left join on these dataframes in a very specific way as follows, and the easiest way to explain is probably via an example. First dataframe:...
-1
votes
1answer
21 views

How do I add a column to a Pandas dataframe based on other rows and columns in the dataframe? [on hold]

I've tried a lot of different methods, but I can't seem to find the right way to do this. I want to create a new column based on the time and ...
1
vote
0answers
11 views

How to prepare future data for training

Assume I have a large data of ecommerce website sessions with user id key (a user can have multiple sessions with random time between them). The data is on S3 in json gzipped format. On some sessions ...
0
votes
1answer
17 views

data analysis EDA issues, indent or type

I am a newbie in DS world, right now i am working on some EDA practice, and run into an issue here. Here is my code: Convert some columns to numeric values ...
1
vote
0answers
10 views

calculate days from a column to today in r

I have data set with a Date column, now i am trying to calculate duration in days from today to dates in date column with a new column named duration. I tried ...
0
votes
2answers
46 views

LOOKUP using 2 dataframes in Python

I have 2 dataframes: df1 Id CategoryId 1 A 1 B 2 A 2 E 2 F df2: ...
-1
votes
0answers
18 views

Better clean and tidy data string categorical variable

I noticed there are some ways to clean data set for text categorical variables to convert it into numerical variables by using getting dummies (convert it into 1 or 0 binary variable) or maybe ...
0
votes
0answers
14 views

Fetch nested structure from pdf data

I get a server scan report after every patch. And every report has some hierarchical structure. e.g. Page_1 will have Server name. Page_2 will have Error categories (e.g. compilation_error, ...
1
vote
1answer
29 views

Dataset with disproportionately more of a single label than any other

I'm using the data analysis software Orange to analyze rows of data with labels $\{H, T,L\}$. $T$ is the neutral state of the system I'm trying to model so data is almost always labeled with $T$. ...
0
votes
2answers
40 views

How to handle negative integer values in a data set?

Suppose we have a data set of fields with negative integer values. So can we consider that fields with negative values for further proceedings or do we need to ignore that fields? If we can consider ...
4
votes
3answers
94 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
1
vote
0answers
8 views

What is the difference between 'if the data if of good quality' and 'if the data is tidy'?

I'm doing Data Analyst nanodegree from udacity. I'm confused between the difference even after going through the lecture a few times.
1
vote
1answer
12 views

Symlink or Rename files with spaces?

Premise: I often get files from colleagues that I need to work on. Often times, these files have spaces in the names. Working with these files at the command line or in scripts can be tedious. ...
8
votes
1answer
66 views

Under what circumstance is lemmatization not an advisble step when working with text data?

Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data? For example, would lemmatization be ...
1
vote
0answers
37 views

Create a new row when a character exists in Python Dataframe

I have an excel sheet which look like this: ColumnA ColumnB ColumnC a 1|2|3 b r 4|5 d b 6|7|8|9|10 z When columnB ...
1
vote
0answers
9 views

How do i create the same categories across two or more variables(columns) when converting integers to Factors in R?

I did merge the columns, arranged them as rows and then converted them to categories. I just wanted to know if there was a better way in doing this.
1
vote
0answers
22 views

What are the best free reporting tools?

What are the best free Business Intelligence (Reporting) tools, that allows live updates on the reports and also sharing them to other users? Actually now I’m using the community edition of Pentaho ...
2
votes
2answers
92 views

data science python data cleaning

I am preparing a dataset for a model, but somehow the code just doesn't run well. The major error is: ...
2
votes
1answer
70 views

How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost

How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost... Please can you help me, This is a bit urgent, I am not sure how to do ...
0
votes
1answer
17 views

Sparse Data Interpolation

I have a price data set where on some days there are up to five data points and some days none at all. For example: ...
0
votes
0answers
23 views

fill missing values

I have the following imdb dataset https://github.com/RUG-IDS/Course-2017/blob/master/Assignments/Assignment%201/movievalue.csv I want to clean the dataset and fill the missing values, but I noticed ...
1
vote
0answers
13 views

What are the ways to merge data from different sources? Is there any software framework?

Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and ...
0
votes
1answer
23 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
1answer
19 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished the race in 45 seconds,35 ...
1
vote
0answers
18 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
-1
votes
2answers
62 views

Data Cleansing - Handling CSV files

I would like to hear some views on a problem I have with my dataset (which I presume to be a common one). Let's say I have the following dataset ...
3
votes
0answers
50 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
0
votes
1answer
18 views

Low number of inputs compared to outputs (per row) in neural network

I have data that one row has input of size 1x2 (two values) and output is matrix of size 15x3 (fifteen rows with three values) like that ...
3
votes
4answers
70 views

R: Checklist for data checking

I am new to data science and new to R. I see a lot of resources on data cleaning (i.e., dplyr, tidyr, tidyverse), but not so much on raw value verification. What are the "best practices" I should ...
0
votes
0answers
9 views

Expanding entity extraction to certain types of named-noun phrases (Bi-grams or tri-grams)

I have a corpus of articles from business publications. I'm interesting in extraction information relating to specific types of technologies and processes, as well as just interesting phrases. ...
0
votes
1answer
19 views

Do we need the Preprocessing step on both Test and Train data sets?

I have seen many people handle the missing or inconsistent data in both their test and train data sets. Sometimes they handle only the train data sets and sometimes they merge the train and test data ...
0
votes
0answers
11 views

Remedy feature distribution mismatch with pre-processing

I'm quite new to machine learning and I'm working on a conditional VAE. I'm experiencing a problem where the distribution for one of the generated variables does not match the true distribution at all....
0
votes
1answer
26 views

What are appropriate labels for age categorical labels?

I am converting some age data to categorical variables. What are some appropriate labels? Some people might take offense to using "Young", "Old" or "Millenial", etc. Is there a "standard" list of ...
0
votes
0answers
18 views

How to handle text encoding while text input or cleaning?

Often while data reading, cleaning or performing similar tasks; often I stumble upon error message such as UnicodeEncodeError, or while tokenising these texts, I get various errors because of encoding....
0
votes
0answers
164 views

Data Cleaning Framework/Library for Scala Spark?

I have crawled data from websites and Data is in Json format having different columns. I selected a column e.g (raw_text) but this column has very irrelevent things,I want to clean it using Apache ...
1
vote
0answers
205 views

Audio Analysis : Segment audio based on speaker recognition

I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) &...
1
vote
1answer
285 views

How can I perform stratified sampling for multi-label multi-class classification?

I am asking this question for few reasons: The dataset in hand is imbalanced I used below code ...
0
votes
1answer
62 views

How to train model on specific classes from dataset for object detection?

I want to use darkflow to perform object detection. I have 5 classes of objects I want to detect, say , ...
0
votes
2answers
20 views

if for same input values we have different class labels, Should we have to remove those training examples before building any model on it?

Ex: A1 A2 A3 Class attribute --------------------------- X Y Z A X Y Z B P Q R C Should we remove first and second training ...
2
votes
2answers
182 views

Tidy data in panda dataframe

There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he ...
0
votes
4answers
62 views

Tools to explore various datasets

TL:DR - Do you know of good automated tools to explore a dataset? Long version: I have a few different datasets to work with from most various areas of business. I wonder if there are good software/...
0
votes
1answer
23 views

Storing engineered features in a database

I have some data in raw csv files which I would like to store in a MySQL database. The problem is there are constant feature engineering done on this dataset so coming up with one schema to fit all ...
1
vote
0answers
12 views

How to preprocess raw data in the form RDF triples to perform Information Gain and Principal Component Analysis? [closed]

I have my dataset in the form RDF triples in various domains such as Movie, music etc. Data is in the form of RDF triples (Subject, Property,Object (all uris). A sample input in the movie domain is ...
3
votes
1answer
54 views

Find most important features which differentiate two sets

Assume we have two small sets of feature vectors(each feature vector representing an item), what is a good way of finding which features have the biggest difference (in distribution) between the two (...
3
votes
1answer
321 views

how to remove unwanted characters from data

I need to write a function which identifies and removes the "*" character after some numeric values in a vector. I also need that the resultant vector is a numeric vector. ...