Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
1answer
129 views

Web Framework Built for Recommendations

I'm wondering if there is a web framework well suited for placing recommendations on content. In most cases, a data scientist goes through after the fact and builds (or uses) a completely different ...
3
votes
1answer
89 views

Simple Excel Question: VLookup Error

My data looks like this: Why is this error showing up?
2
votes
0answers
84 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
5
votes
1answer
166 views

Data scheduling for recommender

I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns: Users Items Rating The pair in the ...
3
votes
0answers
28 views

Method to create master product database to validate entries, and enrich data set

We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the ...
5
votes
2answers
121 views

Amalgamating multiple datasets with different variables coding

I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables ...
1
vote
2answers
56 views

Modelling on one Population and Evaluating on another Population

I am currently on a project that will build a model (train and test) on Client-side Web data, but evaluate this model on Sever-side Web data. Unfortunately building the model on Server-side data is ...
3
votes
1answer
973 views

Data preparation and machine learning algorithm for click prediction

I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have ...
6
votes
7answers
275 views

Good practices for manual modifications of data

More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed. When a fraction of data needs it, I write a script and ...
41
votes
6answers
5k views

How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?

Motivation I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject ...
19
votes
5answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
35
votes
7answers
4k views

Organized processes to clean data

From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis. Are there any best practices or processes for cleaning ...

1
7 8 9 10
11