Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

0
votes
1answer
8 views

How to do multivariate survival analysis on dataset having only categorical variables

I have a dataset where I have around 50 independent variables used to run survival analysis on the target variables. But out of these 50 variables, 46 variables are categorical variables i.e. having ...
0
votes
0answers
20 views

Kaggle project suggestion which has more scope for EDA, data cleaning and feature selection [on hold]

I am new to data science. I have learnt Linear Regression, Logistic Regression, Decision Tree(Gradient Boosting,Random Forest etc), Neural Networks, KNN and KMeans & Hierarchical Clustering and ...
0
votes
1answer
17 views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
0
votes
1answer
9 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...
0
votes
0answers
6 views

Converting HTML tables to a data.frame in R

I am trying to convert an HTML table of daily data into a R data.frame for trend analysis. Using xml2 and rvest I am able to download the table into RStudio, but am struggling to A) create the ...
1
vote
0answers
16 views

How do I prepare data in which each output row depends on multiple input rows?

My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y. In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on ...
1
vote
3answers
64 views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
0
votes
0answers
16 views

Need help to speed up code for stitching together badly formatted log files

The objective: I needed to stitch together a large number of badly formatted log files. I [painfully] developed a script for it in R, one that successfully dealt with the various flaws in the dataset. ...
1
vote
2answers
62 views

Pandas Conditional Fill NaN Forward/Backward

Updated 22 Oct. 2018: I have the following dataset: ...
0
votes
0answers
17 views

Handle outliers, Losing many data by removing natural outliers

I have 2 skewed features, here is the summary of one of the features ...
0
votes
2answers
27 views

Delete rows of a column using another column statistics in R [closed]

I have a dataframe with the following columns: If a year (see "year" column) has less than 10 weeks reported, I would like to delete it from the dataframe. The column "weeks_reporting" gives the ...
0
votes
3answers
53 views

How to access substrings in pandas column and store it into new columns?

I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like ...
0
votes
0answers
13 views

I want to create a Data cleaning module for English language to clean certain Emails. How do i begin?

Here is an Example of an Email in my .xlsx file. End goal is to perform text summarization and Topic modelling of the various Email's present in the Excel file.
0
votes
3answers
82 views

How to replace a part string value of a column using another column

How to replace a part string value of a column using another column. My DataSet here is : ...
0
votes
1answer
33 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
0answers
7 views

Does integrating data points that fall under a curve make sense?

I have data I've collected from the fanfiction website, Archive of Our Own. I want to know if there's a relationship between the length of a chapter and the average number of hits per chapter. I ...
1
vote
0answers
24 views

Extracting the time duration for events from an event log

I have an event file (very similar to a log file) that logs event information that includes user data and time stamps. The events have an ordering for each user. So event E1 occurs before E2 that ...
0
votes
0answers
25 views

Working with many unique values in a column

I have only learnt basics of data science about 4 months ago. And now I am working on project. This is my code: https://pastebin.com/er1EA7pZ This is my data: https://www.kaggle.com/grosvenpaul/...
1
vote
1answer
75 views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
0
votes
0answers
12 views

What keywords should I be looking for when verifying web-mined data?

I am scraping websites using the scrapy tool. The websites are all for restaurants and similar places. I have the name, latitude, and longitude information for them from OpenStreetMap. Sometimes ...
2
votes
2answers
26 views

How to get a count the number of observations for each year with a Pandas datetime column?

I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year. This is the column: ...
1
vote
1answer
34 views

What methods can be used to detect duplicacy in image dataset?

I want to remove duplicate images from a dataset of 50Million images. What is the best method to detect all the duplicates? Do you think one shot learning is good for this?
2
votes
1answer
17 views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
1
vote
0answers
37 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
2
votes
0answers
29 views
0
votes
1answer
107 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
3
votes
1answer
31 views

Can someone please explain what this sample function is upto?

So there is a function in Dino_Name_Generator at Deeplearning.ai notebook ...
0
votes
3answers
67 views

How to statistically prove that a column in a dataframe is not needed

I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
1
vote
3answers
39 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
-1
votes
1answer
183 views

How do I add a column to a Pandas dataframe based on other rows and columns in the dataframe? [closed]

I've tried a lot of different methods, but I can't seem to find the right way to do this. I want to create a new column based on the time and ...
1
vote
0answers
15 views

How to prepare future data for training

Assume I have a large data of ecommerce website sessions with user id key (a user can have multiple sessions with random time between them). The data is on S3 in json gzipped format. On some sessions ...
0
votes
1answer
21 views

data analysis EDA issues, indent or type

I am a newbie in DS world, right now i am working on some EDA practice, and run into an issue here. Here is my code: Convert some columns to numeric values ...
1
vote
0answers
11 views

calculate days from a column to today in r

I have data set with a Date column, now i am trying to calculate duration in days from today to dates in date column with a new column named duration. I tried ...
1
vote
2answers
171 views

LOOKUP using 2 dataframes in Python

I have 2 dataframes: df1 Id CategoryId 1 A 1 B 2 A 2 E 2 F df2: ...
-1
votes
0answers
15 views

Fetch nested structure from pdf data

I get a server scan report after every patch. And every report has some hierarchical structure. e.g. Page_1 will have Server name. Page_2 will have Error categories (e.g. compilation_error, ...
1
vote
1answer
31 views

Dataset with disproportionately more of a single label than any other

I'm using the data analysis software Orange to analyze rows of data with labels $\{H, T,L\}$. $T$ is the neutral state of the system I'm trying to model so data is almost always labeled with $T$. ...
0
votes
2answers
44 views

How to handle negative integer values in a data set?

Suppose we have a data set of fields with negative integer values. So can we consider that fields with negative values for further proceedings or do we need to ignore that fields? If we can consider ...
4
votes
3answers
98 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
1
vote
0answers
10 views

What is the difference between 'if the data if of good quality' and 'if the data is tidy'?

I'm doing Data Analyst nanodegree from udacity. I'm confused between the difference even after going through the lecture a few times.
1
vote
1answer
13 views

Symlink or Rename files with spaces?

Premise: I often get files from colleagues that I need to work on. Often times, these files have spaces in the names. Working with these files at the command line or in scripts can be tedious. ...
8
votes
1answer
71 views

Under what circumstance is lemmatization not an advisble step when working with text data?

Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data? For example, would lemmatization be ...
1
vote
0answers
40 views

Create a new row when a character exists in Python Dataframe

I have an excel sheet which look like this: ColumnA ColumnB ColumnC a 1|2|3 b r 4|5 d b 6|7|8|9|10 z When columnB ...
1
vote
0answers
9 views

How do i create the same categories across two or more variables(columns) when converting integers to Factors in R?

I did merge the columns, arranged them as rows and then converted them to categories. I just wanted to know if there was a better way in doing this.
1
vote
0answers
24 views

What are the best free reporting tools?

What are the best free Business Intelligence (Reporting) tools, that allows live updates on the reports and also sharing them to other users? Actually now I’m using the community edition of Pentaho ...
2
votes
2answers
116 views

data science python data cleaning

I am preparing a dataset for a model, but somehow the code just doesn't run well. The major error is: ...
2
votes
1answer
170 views

How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost

How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost... Please can you help me, This is a bit urgent, I am not sure how to do ...
0
votes
1answer
23 views

Sparse Data Interpolation

I have a price data set where on some days there are up to five data points and some days none at all. For example: ...
0
votes
0answers
27 views

fill missing values

I have the following imdb dataset https://github.com/RUG-IDS/Course-2017/blob/master/Assignments/Assignment%201/movievalue.csv I want to clean the dataset and fill the missing values, but I noticed ...
1
vote
0answers
13 views

What are the ways to merge data from different sources? Is there any software framework?

Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and ...
1
vote
1answer
40 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....