Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
0answers
236 views

Scoring consistency within dataset

Suppose I am given a set of structured data. The data is known to be problematic, and I need to somehow "score" them on consistency. For example, I have the data as shown below: ...
8
votes
2answers
651 views

Fixing data inconsistencies

I'm trying to analyze some data I have but there is a lot of inconsistencies in my data. I have a SQL table that I'm trying to analyze. The table is a table of universities with the following ...
2
votes
2answers
147 views

Use test data as train: does it make sense?

There is a classification problem(two classes). We have train data, for which we know class labels and we have test data. Imagine, that you have created model that with good accuracy(~95%) make ...
14
votes
4answers
20k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
7
votes
1answer
431 views

Neural Networks: How to prepare real world data to detect low probability events?

I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records ...
0
votes
1answer
54 views

Add new factor across multiple groups

I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for ...
2
votes
1answer
639 views

Encoding feature with tree-path data

What are the common techniques for encoding discrete features with tree-path value (e.g. file/folder path) without expanding tree into multiple features corresponding to each node? Update. Overall, ...
1
vote
0answers
78 views

What's the best way to rank aggregate imdb rating data?

I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: 1) multiply each average rating by the total ...
4
votes
1answer
1k views

Creating training data

My task is to classify free text originated from customer complaints about our product. I have created a Taxonomy and have around 10 different categories. I've realized that these categories include ...
1
vote
1answer
43 views

convert some observations into variables

I have a table formatted like the following table: ...
21
votes
3answers
14k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
1
vote
0answers
69 views

What tools are available for semi-automated matching of dirty columnar data

Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets? The data I'm working with was collected (and handled) by different organizations. ...
2
votes
1answer
8k views

Ignoring symbols and select only numerical values with pandas

In one field I have entries like 'U$ 192,0'. Working on pandas, how I ignore non numerical data and get only the numerical part?
2
votes
1answer
400 views

What metrics must i use in my data(unstructured) preprocessing research?

i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats). i wish to prove preprocessing improves the content quality. are there metrics available to prove ...
2
votes
2answers
228 views

Sort by average votes/ratings

I have a data set that's a dictionary of tuples. Each key represents an ID number and each tuple is (yesvotes, totalvotes). Example: ...
2
votes
1answer
146 views

Is there a tool/ library/ algorithm that can learn from the steps I perform in cleaning text and apply them to similar data?

I have a dataset of organization names that is quite messy. I used all the popular NER tools on it without much success(I assume it's because they lack context). I resolved to using OpenRefine but I ...
2
votes
0answers
136 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
1answer
115 views

What would be the best way to structure and mine this set of data?

http://pastebin.com/K0eq8cyZ I went through each season of "It's Always Sunny in Philadelphia" and determined the character groupings (D=Dennis, F=Frank, C=Charlie, M=Mac, B=Sweet Dee) for each ...
1
vote
0answers
185 views

Natural Language text categorization using RapidMiner

I'm new to data mining, so this might sound like a very simple task to some. So, I work in reliability engineering in aviation & I have a set of data that is generated on a daily basis regarding ...
1
vote
1answer
4k views

Code Vectorization of gsub in R

How can I vectorize this code in R? ...
3
votes
1answer
207 views

Handling a feature with multiple categorical values for the same instance value

I have data in the following form: table 1 id, feature1, predict 1, xyz,yes 2, abc, yes table2 id, feature2 1, class1 1, class2 1, class3 2, class2 I could perform a one many join and train on the ...
-2
votes
1answer
418 views

How should I analyze this data from reddit (sample text included) [closed]

I have downloaded some data to learn about machine learning and distributed computing. I used WinZip to uncompress a .bz2. Now I have text (opened in Notepad) which looks like this (this file is long, ...
10
votes
2answers
82k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
2
votes
3answers
3k views

What are the best ways to tune multiple parameters?

When building a model in Machine Learning, it's more than common to have several "parameters" (I'm thinking of real parameter like the step of gradient descent, or things like features) to tune. We ...
1
vote
1answer
239 views

Detecting boilerplate in text samples

I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example: DESCRIPTION PROVIDED BY AUTHOR: The goal ...
1
vote
1answer
117 views

removing words based on a predefined vector

I have the dataset test_stopword and I want to remove some words from the dataset based on a vector. How can I do that in R? ...
6
votes
1answer
2k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
3
votes
2answers
639 views

Feature Scaling and Mean Normalization

In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this: 1) Can I mix and match these two approaches? e.g. Feature ...
17
votes
1answer
115k views

removing strings after a certain character in a given text

I have a dataset like the one below. I want to remove all characters after the character ©. How can I do that in R? ...
6
votes
2answers
199 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
1
vote
1answer
1k views

R: Revalue multiple special characters in a data.frame

R noob here.. I have the following data frame >data Value Multiplier 1 15 H 2 0 h 3 2 + 4 2 ? 5 2 k where the multiplier is of ...
1
vote
1answer
226 views

How to start analysing and modelling data for an academic project, when not a statistician or data scientist

I have collected data for a PhD thesis, and need help understanding how to build a road map to do analytical and statistical analysis. The PhD is not itself in statistics or machine learning, but I ...
1
vote
2answers
265 views

Looking for language and framework for data munging/wrangling

I have an extensive series of projects that require importing data from many different types of different sources (websites, web apis, sensors, legacy text files, etc). Is there a good framework (...
2
votes
3answers
551 views

Preprocessing in Data mining?

I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the ...
1
vote
1answer
608 views

Data frame mutation in R

I have a data frame of the following format: Symbol Date Time Profit $BANKNIFTY 4/1/2010 9:55:00 -1.18% <br>$BANKNIFTY 4/1/2010 12:30:00 -2.84% $BANKNIFTY 4/1/2010 12:45:00 7.17% &...
4
votes
1answer
38 views

Alignment of square nonorientable images/data

Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better. Say I have a 2D ...
5
votes
1answer
300 views

Anonymizing Datasets

I would like to know what are the best practices for anonymizing datasets? Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be ...
2
votes
1answer
2k views

Data.frame vs Data.table in R?

Data.table package is claimed to be faster than Data.frame. what are the implementation changes that made this possible? How can one leverage the power of this package for data analysis?
1
vote
2answers
777 views

any reason for this project to use hadoop/spark?

I am setting up for a couple self study projects to explore machine learning techniques. 1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I ...
1
vote
3answers
121 views

Clustering uncertain data with independent uncertainty per dimension

I have $n$ objects located in a $d$ dimensional space, however I do not know their exact coordinates. For each object and each dimension, I have a set of noisy measurements of the coordinate. I would ...
2
votes
2answers
140 views

How do you ascertain the quality of your data? [closed]

How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such ...
4
votes
1answer
115 views

Looking for smallest set of rows that form a natural key in a data set

I have several sets of text files on hdfs that are exports from relations. Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row ...
2
votes
1answer
125 views

Data cleaning: Relationships between columns

I have a training data set distributed in two files. File 1: This contains actual classification for each X1. X1 is unique in this file. X1 has one-to-one relationship with X2, i.e. X2 is also unique....
0
votes
1answer
723 views

Automomatic typographical error correction

Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking ...
0
votes
2answers
314 views

Tidying Time Intervals for Plotting a Histogram in R

I'm doing some cluster analysis on the MLTobs from the LifeTables package and have come across a tricky problem plotting ...
28
votes
3answers
31k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
1
vote
2answers
1k views

How do I get Twitter Dataset for Visualization [closed]

I am a newbie. I would like to do visualization on twitter data : top trends based on country (over map) and time variations (for each months in 1 year or for each year) . Can someone tell me where ...
2
votes
1answer
1k views

InterquartileRange takes up most instances in data set

I'm very new to this community, so please overlook my noobness. I have a data set with 2948 instances and I tried to remove outliers using InterquartileRange filter in Weka. The issue is that the ...
2
votes
3answers
952 views

OCR / Text Recognition and Recovery Problem

I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. ...
1
vote
1answer
29 views

Correcting Datasets with artificially low starting values

I am working on a project where we would like to take the ratio of two measurements A/B and subject these ratios to a ranking algorithm. The ratio is normalized prior to ranking (though the ranking/...

1
7 8 9
10
11